{"id":"90430cc6-7d6c-4589-a2c9-c458eb522a47","arxiv_id":"2412.10489","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CognitionCapturer decodes visual stimuli from EEG by aligning brain signals to image, text, and depth embeddings and generating reconstructions with SDXL and IP-Adapters.","lead":"Researchers trained separate EEG decoders for image, text, and depth, then combined them with a diffusion model to reconstruct images people saw. The system beats prior EEG-to-image methods on some metrics, but its headline accuracy uses a generous scoring rule.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline quantitative claim rests on an OR-rule upper bound in Table 2, not a real fused classifier, and reconstruction metrics in Table 3 are mixed; multimodal superiority over single-modality and over ATM is not established.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The paper has real strengths: the image-only variant beats ATM on classification, the code is released, and the Grad-CAM analyses provide useful interpretability. However, the strongest quantitative evidence for the central multimodal claim is the OR-rule upper bound in Table 2, which the footnote itself admits is not a practical accuracy. The reader identified this issue in the rationale but chose the depth/text informativeness as the weakest assumption; I see the OR-rule evaluation as more directly load-bearing because even fully informative depth and text embeddings would not justify the reported 35.64% as a system-level accuracy. The reconstruction metrics in Table 3 are also mixed, with the 'all' condition underperforming the image-only condition on two high-level metrics, and the supplementary comparison to ATM is unfavorable on most high-level metrics. These problems are fixable: report a real fusion accuracy, include ATM in Table 3, and either demonstrate a genuine multimodal advantage or temper the claim to 'the image-only variant achieves SOTA while multimodal experts provide complementary information in an upper-bound sense.' Since the reader already required conditions, my analysis does not move the verdict; it reinforces that the conditions are necessary.","tokens_in":13861,"tokens_out":5154,"duration_ms":632311,"concrete_test":"Re-implement the classifier using a fixed fusion rule, e.g., average the three Modality Expert logits or concatenate normalized embeddings followed by a linear head, and compute 200-way top-1 accuracy on the Things-EEG test set. If the fused accuracy is ≤ the image-only 33.30%, then the headline 35.64% is entirely attributable to the OR-rule upper bound and the multimodal claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'outperforms state-of-the-art' claim depends on Table 2, where 'CognitionCapturer (all)' Top-1 accuracy of 35.64% is computed by counting a sample correct if any one of the image/text/depth Modality Expert Encoders classifies it correctly (see the footnote in the Results section). This is an upper bound, not the accuracy of any actual fused classifier, because a deployable system cannot know which expert is correct. The image-only variant (33.30% vs ATM's 26.1%) is a legitimate SOTA comparison, but the multimodal gain from 33.30% to 35.64% is an artifact of the OR rule unless a real fusion achieves it. Reconstruction evidence is also mixed: in Table 3, 'all' improves PixCorr/SSIM over 'image' (0.150/0.347 vs 0.132/0.321) but is worse on AlexNet(2) and AlexNet(5) (0.754/0.623 vs 0.813/0.671), and the supplementary ATM reconstruction (Table 6) beats 'all' on most high-level metrics (AlexNet5 0.866 vs 0.623). Thus the quantitative case for multimodal superiority over single-modality and over ATM is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CognitionCapturer, a framework for decoding visual stimuli from EEG by training separate Modality Expert Encoders to align EEG with image, text, and depth embeddings, then mapping the EEG embeddings to CLIP space via a diffusion prior and generating images with pretrained SDXL-Turbo and IP-Adapters. Experiments on Thing-EEG report 200-way zero-shot classification and reconstruction metrics, and the paper claims that CognitionCapturer outperforms state-of-the-art methods both qualitatively and quantitatively.","tokens_in":14106,"tokens_out":4442,"duration_ms":38537,"significance":"If the claims were fully supported, this would be a useful contribution to EEG-based visual decoding, showing that auxiliary multimodal embeddings can improve classification and reconstruction. The framework is extensible, and the authors have released code, which are strengths. However, the quantitative evidence as presented is not sufficient to support the headline claims: the classification gain over single-modality systems rests on an OR-rule upper bound, and the reconstruction comparison against the ATM baseline is mixed or unfavorable on several metrics. The paper's significance therefore depends on substantial revision of both the evaluation method and the claims.","major_comments":[{"comment":"The 'CognitionCapturer (all)' top-1 accuracy of 35.64% is computed with an OR rule: a test sample is counted correct if any one of the three Modality Expert Encoders classifies it correctly, as stated in the table footnote. This is an upper bound, not the accuracy of any deployable fused classifier. The abstract and conclusion claim that CognitionCapturer 'outperforms state-of-the-art methods both qualitatively and quantitatively', and the only number in Table 2 that beats ATM's 26.1% is this upper bound. The image-only row (33.30%) is a legitimate comparison, but the multimodal gain from 33.30% to 35.64% is an artifact of the OR rule. The paper should report a real fusion accuracy (e.g., an ensemble or a learned fusion) or clearly reframe the claim.","section":"Results and Discussion, Table 2"},{"comment":"The reconstruction results do not support the claim of outperforming previous work quantitatively. In Table 3, the all-modality system is worse than the image-only system on AlexNet(2) and AlexNet(5) (0.754/0.623 vs 0.813/0.671), directly contradicting the main contribution statement that reconstruction performance surpasses 'that of any single modality'. In addition, the supplementary Table 6 reports ATM values of AlexNet(5)=0.866, Inception=0.734, and CLIP=0.786, all higher than CognitionCapturer-all (0.623, 0.669, 0.715). The main text statement that the method 'outperforms previous work (Li et al. 2024) in the fine-grained alignment of reconstructed visual stimuli' is not supported by these numbers; the authors need to reconcile the comparison or clearly specify in which metrics their method is superior.","section":"Table 3 and Supplementary Table 6"},{"comment":"The paper's central premise is that text and depth modalities provide 'beyond-image-modality' information that improves EEG decoding. However, the paper itself cites Zhang et al. (2022) to state that CLIP image embeddings already contain depth information, and the text and depth embeddings are deterministically derived from the same images via BLIP2 and DepthAnything. The observed OR-rule classification gain could simply reflect that different experts make different errors rather than that they extract complementary EEG information. To justify the claimed contribution, the authors should provide evidence that a single image-only expert of matched capacity cannot achieve the same or better performance, or directly quantify whether the text/depth embeddings add information beyond the CLIP image embedding in the EEG decoding task.","section":"Introduction and Modality Expert Encoder (Section 3.1)"}],"minor_comments":[{"comment":"The caption says 'acc±std' but no standard deviations are shown anywhere in the table; please report the standard deviations or correct the caption.","section":"Table 2 caption"},{"comment":"The ATM baseline is absent from Table 3 and appears only in supplementary Table 6. Since the paper compares against ATM throughout, please include the ATM row in the main table or clearly cross-reference the supplementary values.","section":"Table 3"},{"comment":"The caption states that 'selected reconstruction results from subject-08' are shown, but the selection criteria are not described; please clarify whether these are representative, best, or cherry-picked examples.","section":"Figure 3 caption"},{"comment":"The main text refers to 'STConv' while the supplementary refers to 'TSConv' for the same module; please unify the terminology.","section":"Supplementary, Modality Expert Encoder"},{"comment":"The claim that different Modality Expert Encoders focus on different brain regions is based on visual inspection of Grad-CAM maps; a quantitative comparison (e.g., spatial centroid distance or overlap coefficient) would make the claim more rigorous.","section":"Section 4.4 (Grad-CAM analysis)"}],"recommendation":"major_revision","confidential_remarks":"The code release and use of a public benchmark are strengths, and the paper fits the scope of the venue. However, the quantitative claims in the abstract and conclusion are not supported by the reported evidence: the classification comparison rests on an OR-rule upper bound, and the reconstruction comparison with ATM is unfavorable on high-level metrics in the paper's own supplementary material. The authors will need to substantially revise the evaluation and modify the claims for the paper to be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the image-only branch of this paper is a genuine step forward, but the headline \"outperforms state-of-the-art\" claim for the full multimodal system doesn't hold as stated. The 35.64% top-1 accuracy in Table 2 is an OR-rule upper bound (any of the three expert encoders correct counts), not the accuracy of any deployable fused model. The authors even note this in the footnote, then still use the number in the abstract and conclusion. That's a real mismatch.\n\nWhat's actually new: training three separate EEG encoders against CLIP image, text, and depth embeddings, then fusing their outputs through a diffusion prior and multiple IP-Adapters for generation. That specific architecture isn't in ATM, NICE, or BraVL. They ship code, use held-out test concepts, and report standard metrics. The image-only variant beats ATM on 200-way classification (33.30 vs 26.1), which is a legitimate SOTA comparison. The Grad-CAM analyses tying different modality experts to different brain regions are a nice touch.\n\nThe soft spots are mostly in the evaluation. Table 3 shows the full system is worse than the image-only variant on AlexNet(2) and AlexNet(5), and the supplementary Table 6 shows ATM's reconstruction actually beats 'all' on most high-level metrics (e.g., AlexNet5 0.866 vs 0.623). So the reconstruction evidence for multimodal superiority is mixed at best. The depth-modality redundancy concern is minor; the paper itself cites Zhang et al. showing CLIP embeds depth, so the incremental contribution of the depth expert is questionable but not fatal.\n\nWho should read this: anyone working on EEG-to-image decoding, especially on contrastive alignment and diffusion-based reconstruction. The paper is worth a serious referee because the core idea is solid and the image-only result is strong, but the evaluation needs significant reworking before publication. I'd ask the authors to report a real fusion accuracy (e.g., averaging logits or training a fusion layer) or to revise the abstract and conclusion to match what the OR-rule actually demonstrates.\n\nRecommendation: send to peer review, but with the expectation of major revision focused on the evaluation.\n\nBest.","headline":"Solid image-only EEG decoding with a novel multi-expert framework, but the headline multimodal claim rests on an OR-rule upper bound and reconstruction metrics are mixed.","tokens_in":14647,"tokens_out":2623,"would_cite":true,"duration_ms":23896,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CognitionCapturer reconstructs seen images by aligning EEG to image, text, and depth embeddings, then decoding through a frozen diffusion model.","keywords":["EEG decoding","visual stimulus reconstruction","multimodal contrastive learning","modality expert encoder","diffusion prior","CLIP embedding space","depth estimation","image captioning"],"falsifier":"Re-run the 200-way classification using a single fused decision rule, such as summing the three experts' logits or choosing the expert with the highest confidence, instead of the any-expert-correct rule; if fused top-1 does not beat the image-only expert's 33.30%, the multimodal gain is an artifact of the scoring rule.","tokens_in":13628,"feed_emoji":"🧠","tokens_out":9730,"duration_ms":77988,"temperature":0.7,"pith_summary":"The paper sets out to show that EEG signals carry visual information beyond what direct EEG-image alignment captures, and that this information can be tapped by aligning EEG to multiple derived modalities: captions, depth maps, and the images themselves. It reports that this multimodal alignment reaches 35.64% top-1 accuracy on a 200-way zero-shot classification task under an any-expert-correct rule, compared with 26.1% for a single-modality baseline, and that reconstructed images improve on both low-level and high-level similarity metrics. The authors argue that separate modality experts prevent fine-grained structural information from being overshadowed by coarse semantic information, and that the trained embeddings can drive a frozen generative model without fine-tuning. If the claim holds, EEG-based visual decoding becomes more practical and more extendable than image-pair-only approaches.","feed_headline":"Multimodal EEG decoder hits 35.6% top-1 on 200-way task","feed_subtitle":"Aligning EEG to image, text, and depth beats single-modality baselines and feeds a frozen diffusion model.","key_machinery":"The central mechanism is the Modality Expert Encoder family: for each EEG-modality pair, a dedicated network maps EEG into the embedding space of that modality's pretrained encoder. The alignment is trained with a modified InfoNCE loss that uses the image index as a label, pulling all repetitions of the same image together and avoiding the contradiction of identical pairs being both attracted and repelled. A diffusion prior trained from scratch with a mean-squared-error loss then maps the EEG embeddings into CLIP space, and the generation stage uses SDXL-Turbo with three IP-Adapters, full for image, style-only for text, and layout-only for depth, to combine semantic and structural information without fine-tuning the generative model.","core_discovery":"The central claim is that 'beyond-image-modality' information in EEG can be recovered by building one Modality Expert Encoder for each of three aligned modalities, rather than a single EEG-image contrastive model. Each expert projects EEG into the same embedding space as a pretrained encoder for the image, the BLIP2 caption, or the DepthAnything depth map, using a modified InfoNCE loss in which all EEG repetitions of the same image are pulled together. After training, a diffusion prior maps each embedding into CLIP image space, and SDXL-Turbo with dedicated IP-Adapters for image, style, and layout fuses the three embeddings to generate the reconstruction. The paper reports that the full multimodal system outperforms its own image-only expert and the ATM and NICE baselines in 200-way classification, and that the reconstructed images score higher on pixel-level and semantic metrics than single-modality reconstructions. It also claims that the text and depth experts attend to broader occipital and temporal brain regions than the image expert, supporting the idea that different modalities expose different neural content.","pith_inferences":["Inference: The reported gain of the full system over the image-only expert (35.64% vs 33.30% top-1) should be rechecked under a single fused decision rule; if fused accuracy does not exceed the image expert, the benefit of text and depth may be an artifact of the any-expert-correct scoring.","Inference: Because CLIP image embeddings already encode depth cues, an ablation with shuffled depth maps would isolate whether the depth expert contributes genuinely new signal or merely a second route to the same visual information.","Inference: The Grad-CAM observation that the depth expert favors the right inferior temporal lobe suggests a testable prediction: a depth-trained decoder should be more sensitive than an image-trained decoder to disruption of that region.","Inference: The framework's modality-agnostic design implies that auxiliary modalities generated from the same images could be plugged in without architectural changes, which the paper does not demonstrate."],"forward_implications":["If the central claim is correct, EEG decoders can be trained without requiring exact image-EEG pairs as the only supervision; aligned captions and depth maps provide additional weak labels.","The EEG embeddings produced by the expert encoders support zero-shot classification and retrieval directly, not just generation.","Because the generative stage is frozen, swapping in a newer CLIP-compatible image generator or adapter should improve reconstruction without retraining the EEG side.","Adding further modalities, such as object masks, audio, or scene graphs, should continue to separate semantic from structural information, since each expert is trained independently.","Per-subject training on roughly 30 minutes of GPU time makes the approach practical for personalized decoding settings."],"supporting_citations":[{"why":"Supplies the Thing-EEG dataset with repeated image presentations and the 1654/200 concept split used for training and testing.","marker":"(Gifford et al. 2022)"},{"why":"Defines the NICE contrastive EEG-image baseline and the training protocol that CognitionCapturer extends.","marker":"(Song et al. 2024)"},{"why":"Defines the ATM baseline and contributes the STConv EEG encoder and diffusion prior design that the experts are adapted from.","marker":"(Li et al. 2024)"},{"why":"Provides the MEG image-decoding approach and the seven reconstruction metrics used in the quantitative comparison.","marker":"(Benchetrit, Banville, and King 2024)"},{"why":"Establishes the contrastive-learning-plus-diffusion-prior recipe that maps brain embeddings into CLIP space.","marker":"(Scotti et al. 2024)"},{"why":"Supplies the CLIP image and text encoders that define the shared embedding space and the pretrained features used by the experts.","marker":"(Radford et al. 2021)"},{"why":"BLIP2 generates the captions that form the EEG-text modality pairs used for auxiliary alignment.","marker":"(Li et al. 2023)"},{"why":"DepthAnything produces the depth maps that form the EEG-depth modality pairs used for auxiliary alignment.","marker":"(Yang et al. 2024)"},{"why":"The InfoNCE objective is adapted with image-index labels to train each modality expert encoder.","marker":"(van den Oord, Li, and Vinyals 2019)"},{"why":"Provides the diffusion-prior formulation and classifier-free guidance used to map embeddings to CLIP space.","marker":"(Ramesh et al. 2022)"}],"fun_headline_variants":["EEG decoder fuses image, text, depth for sharper visual recall","Multimodal EEG experts beat single-modality in visual decoding","CognitionCapturer: aligning EEG to text and depth improves decoding","EEG visual decoding strengthened by adding text and depth experts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the text and depth embeddings derived from each image carry information not already present in the CLIP image embedding, so aligning EEG to them adds 'beyond-image' signal rather than duplicating the same visual information.","fun_headline_variants_meta":{"raw":{"variants":["EEG decoder fuses image, text, depth for sharper visual recall","Multimodal EEG experts beat single-modality in visual decoding","CognitionCapturer: aligning EEG to text and depth improves decoding","EEG visual decoding strengthened by adding text and depth experts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000624,"raw_usage":{"total_tokens":2901,"prompt_tokens":972,"completion_tokens":1929,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1856}},"tokens_in":588,"tokens_out":1929,"duration_ms":13174,"temperature":1.0,"reasoning_tokens":1856,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:00:32.053668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 200-way classification using a single fused decision rule, such as summing the three experts' logits or choosing the expert with the highest confidence, instead of the any-expert-correct rule; if fused top-1 does not beat the image-only expert's 33.30%, the multimodal gain is an artifact of the scoring rule.","supporting_citations":[{"cited_title":"T.; Dwivedi, K.; Roig, G.; and Cichy, R","cited_arxiv_id":null,"evidence_quote":"Supplies the Thing-EEG dataset with repeated image presentations and the 1654/200 concept split used for training and testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the NICE contrastive EEG-image baseline and the training protocol that CognitionCapturer extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the contrastive-learning-plus-diffusion-prior recipe that maps brain embeddings into CLIP space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DepthAnything produces the depth maps that form the EEG-depth modality pairs used for auxiliary alignment."}],"review_version":1}