{"id":"825d933b-5b53-4a3e-8c25-631ac31ad79b","arxiv_id":"2502.04658","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning CLIP with human behavioral embeddings and dynamic MEG responses yields models that better predict human similarity judgments and track individual neural dynamics over time.","lead":"The authors fine-tune a CLIP vision model on human similarity judgments and on millisecond-scale MEG brain recordings, producing models that align more closely with individual human perception. A smart generalist might read this because it is a concrete recipe for personalizing AI to a single person's perceptual and neural style, with implications for adaptive interfaces and clinical monitoring.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Personalization claim lacks a group-level or permutation baseline; rho=0.659 may reflect shared representational structure rather than participant-specific neural dynamics.","rationale":"The reader's verdict (CONDITIONAL, moderate confidence) is well aligned with my assessment: the behavioral and group-level neural alignment results are plausible and partially supported by external benchmarks and datasets, but the individual-level personalization claim is under-supported. The reader's formal weakest_assumption concerns the fidelity of MEG decoding RDMs (LDA/SVM) as ground-truth neural representations, which is a real and related risk. However, I view the most load-bearing concern as the absence of a group-level or permutation baseline for the rho=0.659 result. This gap is what would actually settle whether the personalized models capture individual-specific dynamics versus shared structure or participant-specific classifier noise. The reader's rationale mentions this missing baseline, but the official weakest_assumption field does not, so agreement is partial. I also note a secondary concern not raised by the reader: the headline behavioral comparison (rho=0.78 vs 0.32) compares CLIP-HBA-Behavior's 66-dimensional fine-tuned SPoSE embedding against baseline CLIP-ViT-L/14's 768-dimensional last-layer features. A fairer baseline would be CLIP-ViT-L/14's own 66-dimensional SPoSE projection, which the authors compute for the NIGHTS benchmark but do not report for the 48-object behavioral RDM. If that baseline correlation were substantially above 0.32, the 'over doubles' claim would be weakened, though the qualitative improvement might remain. I did not select this as the primary concern because the eponymous personalization claim is more central to the paper's novelty, and the missing group-level baseline is more consequential for the abstract's individualized claims. The paper deserves credit for evaluating on external MEG datasets with different participants and conditions, which partially supports the group-level neural alignment; nevertheless, the personalized models have no such external or control validation. A conditional acceptance requiring the group-level and permutation baselines is appropriate.","tokens_in":18577,"tokens_out":10080,"duration_ms":100058,"concrete_test":"Train a single group-level CLIP-HBA-MEG on the participant-averaged MEG RDMs of the same 15 participants (or use the existing 3-participant group model), then evaluate it on each participant's held-out 18-stimuli neural RDMs using exactly the same global correlation procedure that produced rho=0.659. In parallel, run a permutation control: shuffle the participant-to-model assignment 1000 times, recompute the global Spearman correlation each time, and obtain a null distribution. If the group-level model achieves a correlation comparable to 0.659, or if the permutation null overlaps the observed value, the individualization claim is not supported. If the group model is substantially lower and the permutation null is far below 0.659, the claim would be strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of individualized neural alignment rests on a single global Spearman correlation (rho=0.659, p<1e-14) between pairwise distances among 15 personalized models and pairwise distances among the corresponding participants' MEG RDMs, computed on 18 held-out stimuli. This statistic is presented as evidence that each model captured its participant's unique neural dynamics. However, the design lacks the control needed to support that interpretation. The models were trained on their own participant's MEG RDMs and evaluated on the same participant's held-out neural RDMs, so the correlation conflates at least three sources: (1) group-level object representational structure shared by all participants, (2) participant-specific artifacts of the LDA/SVM decoding RDMs (e.g., classifier noise, sensor-level biases), and (3) genuine individual neural dynamics. The manuscript reports no group-level model trained on the averaged MEG RDMs, no permutation test with shuffled participant-model assignments, and no cross-participant evaluation (train on A, test on B). Without such a baseline, a model trained on any participant's data — or on the group average — could plausibly produce a comparable rho, since human object representations are largely shared. The paper's own framing requires showing that the personalized models do better than a non-personalized alternative at predicting held-out individual neural RDMs; that comparison is absent. This is the most load-bearing gap because the eponymous personalization contribution depends on it, and the abstract explicitly claims that the models 'track the temporal evolution of individual neural responses' and capture 'participant-specific patterns.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CLIP-HBA, a family of CLIP-ViT-L/14 models fine-tuned with human-derived targets. CLIP-HBA-Behavior is trained with an MSE loss to predict 66 SPoSE behavioral embeddings from THINGS images, and is reported to reach Spearman rho=0.78 (95% CI [0.75,0.80]) against a fully sampled behavioral RDM for 48 held-out objects, versus 0.32 for the CLIP baseline, with improved NIGHTS triplet scores (Table 1). CLIP-HBA-MEG extends the method with a dynamic feature-reweighting matrix and a three-part loss (Eq. 10) aligned to MEG decoding RDMs, and is evaluated on THINGS MEG data and three external datasets (Tables 2-3). Finally, 15 participant-specific models are trained on individual MEG RDMs; a global Spearman correlation of rho=0.659 (p<1e-14) between model-pair distances and participant-pair distances on 18 held-out stimuli is presented as evidence of personalized neural alignment. The paper also reports dynamic saliency maps and discusses applications to personalized medicine and human-AI interaction.","tokens_in":18791,"tokens_out":8757,"duration_ms":76774,"significance":"If the main claims hold, the paper constitutes a useful contribution to human-aligned representation learning: the design excludes held-out stimuli from training, the NIGHTS and external MEG validations are appropriate generalization checks, and the SPoSE-based embedding space offers a degree of interpretability. However, the central novelty, personalized neural fine-tuning, is supported by a single correlational statistic without group-level or permutation controls, and the neural targets themselves are decoding RDMs whose fidelity to true neural geometry is not established. The behavioral gains are large relative to the CLIP baseline but lack comparison to competing human-aligned models and to a noise ceiling. The paper also contains no code or data availability statement. The strengths and weaknesses are unbalanced enough that the conclusions should be revised rather than accepted as stated.","major_comments":[{"comment":"The individual-level claim that personalized models capture participant-specific neural dynamics rests on a single global Spearman correlation (rho=0.659, p<1e-14) between pairwise distances among 15 models and pairwise distances among participants' MEG RDMs, computed on 18 held-out stimuli. The analysis lacks a group-level baseline (e.g., one model trained on averaged MEG RDMs), a permutation test with shuffled participant–model assignments, and a cross-participant evaluation (train on A, test on B). Since each model is trained on its own participant's MEG RDMs and evaluated on the same participant's held-out neural RDMs, the correlation could be driven by shared object representational structure or by participant-specific decoding artifacts rather than by genuine individual dynamics. Moreover, the description of this analysis in §4.5 is internally inconsistent: it first describes a per-pair Spearman correlation over time, then a between-pair distance correlation, and the reported value changes from 0.659 to 0.65 without explanation. Please add a group-average model comparison and a shuffled-assignment permutation test, and clarify the exact evaluation protocol.","section":"§2.7, §4.4–4.5"},{"comment":"The neural targets are 'MEG decoding RDMs' produced by LDA/SVM classifiers, but the manuscript does not specify how these RDMs are constructed, how many trials support each RDM, or whether the decoding RDMs are cross-validated. Because the model is optimized to match these RDMs (Eqs. 5–10), the reported neural-alignments could reflect the model reproducing classifier decision boundaries rather than the geometry of neural population responses. At least one central claim, such as the group-level temporal alignment in Figure 2B or the individual alignment in §2.7, should be validated against raw sensor- or source-space RDMs, or the authors should provide evidence that the decoding RDMs are stable across trials, are not dominated by classifier bias, and capture stimulus-specific representational structure.","section":"§4.3.4, Tables 2–3"},{"comment":"The headline behavioral improvement (rho=0.78 versus 0.32) is measured on 48 held-out objects from the THINGS benchmark, and the model is fine-tuned on SPoSE embeddings that are themselves fitted to human similarity judgments for the same object set. The external NIGHTS benchmark in Table 1 is a welcome generalization test, but the reported gains are modest (0.88 vs 0.81 for 768-d features; 0.85/0.84 vs 0.80/0.79 for 66-d features) and are reported without error bars or significance tests. The phrase 'over doubles behavioral performance' relies on comparison to a single baseline CLIP-ViT-L/14 and does not establish where the method stands relative to other human-aligned models (e.g., DreamSim, DINOv2) or to a noise ceiling. Please add such comparisons or temper the generalization claims.","section":"§2.1–2.2"}],"minor_comments":[{"comment":"The number of held-out stimuli is inconsistent: §2.7 and §4.5 state 18 left-out stimuli, while Table 4 lists a Train/Test Split of 80/20 on 100 stimuli, implying 20 validation stimuli; please reconcile these numbers.","section":"Table 4 and §4.5"},{"comment":"The manuscript provides no code or data availability statement, which substantially limits reproducibility of the fine-tuning pipeline and the personalized-model analysis.","section":"General"},{"comment":"The p-value for the behavioral correlation (p<10^-229) is reported without specifying the statistical test or the number of elements in the vectorized RDMs used for the Spearman correlation; please state the test and sample size.","section":"§2.1"},{"comment":"Equation (4) uses the element-wise product symbol \\odot without defining it; please add a definition or explain the notation in the text.","section":"Eq. (4)"},{"comment":"The 'lower-bound noise ceiling' is mentioned in the text and shown in Figure 6A but is never defined or estimated in the Methods; please specify how this noise ceiling was computed.","section":"§2.7, Figure 6A"},{"comment":"References [22] and [49] are the same work (Kucyi et al., Network Neuroscience, 2024) and should be merged or cross-referenced.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially interesting contribution, but the central personalization claim is under-controlled. The authors appear to have the data needed to add a group-average model, a permutation test, and cross-participant evaluations; these are feasible additions. The neural evaluations also need a clearer link between decoding RDMs and actual neural geometry. Given the complexity of the pipeline, code release should be strongly encouraged or required."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the behavioral half of this paper is in decent shape, and the personalized half is not yet supported. The headline result—CLIP fine-tuned on SPoSE embeddings reaching 0.78 Spearman against human behavioral RDMs on 48 held-out objects versus 0.32 for baseline CLIP—looks believable, especially since it transfers to NIGHTS (0.88 vs 0.81 on test). The external MEG validations also help: behaviorally fine-tuned CLIP does better than baseline at predicting neural RDMs across three datasets with different participants and image conditions. The dynamic MEG fine-tuning pipeline—feature reweighting matrix over ViT layers, time-resolved RDM losses, noise injection—is a real methodological addition, not just a linear probe.\n\nWhere the paper gets soft is the personalization claim. The global correlation rho=0.659 between personalized-model distances and participant neural RDMs is presented as evidence that the models captured participant-specific dynamics. But it is never compared against a group-level model trained on averaged MEG, nor against a permutation/shuffled assignment baseline, nor against cross-participant transfer (train on A, test on B). Those controls matter because human object representations are largely shared; a model trained on any participant, or on the group average, could plausibly produce a similar rho just by tracking common object structure. The abstract's claim that the models 'track the temporal evolution of individual neural responses' and 'participant-specific patterns' is stronger than the reported evidence.\n\nTwo smaller issues. The training targets are MEG decoding RDMs from LDA/SVM classifiers; the model can partially fit classifier artifacts, though the external datasets reduce that worry. And the temporal scalars αT and βT are described but not formally defined, and no code or data are released, so the method can't be reproduced as written.\n\nOn the plus side, the paper is honest about its own limitations—it notes weaker generalization on degraded images—and the hyperparameter table is unusually thorough. The behavioral result should survive referee scrutiny; the personalization section needs a controlled comparison before it can carry the weight the paper puts on it.\n\nBottom line: worth sending to peer review. The right referee will focus on the personalization baseline and on defining the scaling terms. A serious revision could make this a useful paper for people working on human-aligned vision models and on individualized neural fine-tuning.","headline":"The behavioral fine-tuning result is credible and worth referee time; the personalization claim needs a group-level or permutation baseline before it can be accepted.","tokens_in":19433,"tokens_out":3189,"would_cite":false,"duration_ms":29046,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a CLIP model on human behavioral embeddings and millisecond-scale MEG recordings more than doubles its agreement with human similarity judgments (0.78 vs 0.32) and lets per-participant models capture individual neural dynamics.","keywords":["personalized AI","CLIP fine-tuning","human perceptual alignment","MEG neural dynamics","representational similarity analysis","SPoSE embeddings","brain-inspired AI","individual differences"],"falsifier":"Train the group-level or individual CLIP-HBA-MEG pipeline on MEG decoding RDMs whose stimulus labels have been randomly permuted, or on RDMs generated from shuffled classifier outputs, then measure the model's temporal alignment with genuine held-out MEG data. If the permuted-target model still shows the reported peak alignment around 300–400 ms or Spearman correlations near the real model's, the objective was fitting classifier artifacts rather than neural structure; if alignment collapses to chance, the dynamic fine-tuning is genuinely reading representational content.","tokens_in":18324,"feed_emoji":"🧠","tokens_out":6814,"duration_ms":61917,"temperature":0.7,"pith_summary":"This paper tries to establish that an off-the-shelf CLIP vision-language model can be reshaped into a human-like perceiver by fine-tuning it on two kinds of human measurements: sparse behavioral similarity dimensions derived from large-scale odd-one-out judgments, and millisecond-resolution magnetoencephalography (MEG) decoding RDMs. The behaviorally tuned model, CLIP-HBA-Behavior, is reported to reach a Spearman correlation of 0.78 with held-out human similarity judgments on 48 objects, more than double the 0.32 of the unmodified CLIP-ViT-L/14 baseline, and to improve scores on the NIGHTS human-similarity triplet benchmark. The neurally tuned variant, CLIP-HBA-MEG, learns a time-varying visual representation whose alignment with MEG data peaks around 300–400 ms after stimulus onset and generalizes across external datasets with different participants and image degradations. The same pipeline, applied participant by participant, is reported to capture individual neural signatures, with a Spearman correlation of 0.659 between personalized model RDMs and participant MEG RDMs on held-out stimuli. If these findings hold, they imply that human inductive biases can be injected into large pretrained models cheaply and adaptively, making them better tools for cognitive neuroscience, personalized medicine, and human-computer interaction.","feed_headline":"Brain-tuned CLIP doubles match to human perception","feed_subtitle":"Fine-tuning with behavioral embeddings and MEG signals lifts similarity-judgment correlation from 0.32 to 0.78.","key_machinery":"The load-bearing mechanism for the behavioral results is the binding of visual features to 66 SPoSE text dimensions via a dot-product projection, trained with mean squared error loss against behavioral SPoSE embeddings using DoRA parameter-efficient fine-tuning on the last text and vision attention layers. For the neural results, the central object is a learned feature-reweighting matrix W in $R^{{T x L}}$ (T MEG timepoints, L = 24 ViT layers), initialized to the last layer and then optimized in two stages; it recombines all ViT layer activations into a time-varying embedding. Temporal scalers alpha_T and beta_T modulate feature magnitude and semantic binding, dimension-wise Gaussian noise mimics neural variability, and a three-term loss combining Pearson correlation, MSE, and time-generalization aligns model RDMs to MEG decoding RDMs. This machinery converts a static CLIP representation into a dynamic one whose similarity geometry can be compared slice-by-slice to the brain's temporal response profile.","core_discovery":"On the paper's own terms, the central discovery is that the representational geometry of a pretrained CLIP model is highly malleable: aligning it to human cognitive measurements, rather than only to image-text statistics, is enough to reorganize its embedding space so it tracks both average human similarity judgments and the temporal evolution of individual neural responses. Concretely, the paper reports that CLIP-HBA-Behavior achieves a Spearman correlation of 0.78 (95% CI [0.75, 0.80]) against fully sampled behavioral RDMs, versus 0.32 for baseline CLIP-ViT-L/14, and that CLIP-HBA-MEG, trained on MEG decoding RDMs, outperforms the static baseline in neural alignment across THINGS and three external datasets while peaking at 300–400 ms after stimulus onset. For personalization, 15 models fine-tuned on single participants' MEG data yield a global Spearman correlation of 0.659 (p < 1e-14) between model-embedding distances and participant-neural distances on 18 held-out stimuli. The paper interprets this as evidence that training methodology, not architecture alone, determines how human-like a network's representations become, and that individual cognitive styles can be encoded in model weights.","pith_inferences":["Editorial extension: because behavioral fine-tuning saturates with roughly 100 training stimuli, the same recipe could be used to align models to small, hard-to-collect behavioral datasets from special populations such as infants, patients, or non-verbal individuals, where large-scale judgments are impractical.","Editorial extension: the dynamic saliency maps are claimed to show where a person attends at each millisecond, but the paper does not compare them to eye-tracking or fixation data; such a comparison would be a direct, testable way to validate whether the model's dynamic attention is perceptual attention or merely a correlate of RDM geometry.","Editorial extension: the loss terms and reweighting mechanism are architecture-agnostic, so a natural next test is to apply the same MEG-targeted fine-tuning to a non-CLIP transformer and check whether the same 300–400 ms semantic peak appears; if it does, the effect is driven by the training objective rather than by CLIP's text-binding component.","Editorial extension: the paper frames personalized models as cognitive digital twins, but a stronger test of that framing would be to see whether a personalized model trained on one recording session can predict that same participant's neural RDMs from a later session, or distinguish them from other participants' data; the current 18-stimulus held-out validation is a single-session check."],"forward_implications":["CLIP-HBA-Behavior's Spearman correlation of 0.78 on held-out similarity judgments (vs 0.32 baseline) means fine-tuning on 66 interpretable behavioral dimensions can make a general vision model approximate human pairwise similarity judgments far better than its original embedding space.","The NIGHTS benchmark gains (validation/test 0.88 vs 0.81 for 768-d features; 0.85/0.84 vs 0.80/0.79 for SPoSE dimensions) show that the behavioral alignment transfers to a large, independently collected triplet-similarity benchmark.","CLIP-HBA-MEG's neural alignment, peaking around 300–400 ms and generalizing to external participants and degraded images, implies the model has learned a time-resolved representation that tracks the late semantic phase of visual processing rather than only early image features.","Personalized models trained on individual MEG data reach a Spearman correlation of 0.659 between model and participant dissimilarity structure on held-out stimuli, implying that stable individual differences in neural dynamics can be encoded in model weights.","Because only DoRA adapter parameters and the feature-reweighting matrix are updated, the personalization pipeline is cheap enough to run participant-by-participant, supporting sequential or on-device adaptation."],"supporting_citations":[{"why":"Supplies the base CLIP-ViT-L/14 architecture and the baseline embeddings that all fine-tuned variants are compared against.","marker":"[17]"},{"why":"Supplies the THINGS object image database used for training and for the fully sampled held-out evaluation stimuli.","marker":"[18]"},{"why":"Supplies the group-level THINGS MEG data and decoding RDMs used as neural targets for CLIP-HBA-MEG training.","marker":"[19]"},{"why":"Supplies the SPoSE method that produces the 66 interpretable behavioral embedding dimensions used as fine-tuning targets.","marker":"[24]"},{"why":"Supplies the behavioral similarity judgments and the 48 fully sampled objects used to measure behavioral alignment.","marker":"[25]"},{"why":"Supplies the NIGHTS triplet benchmark used to test generalization of human perceptual alignment beyond the training distribution.","marker":"[20]"},{"why":"Supplies the representational similarity analysis framework used to construct and compare model and neural RDMs.","marker":"[26]"},{"why":"Supplies the 15-participant MEG dataset used for individual-level neural fine-tuning and personalization evaluation.","marker":"[11]"},{"why":"Supplies the RISE saliency method that is adapted into the model's dynamic attention mapping.","marker":"[31]"},{"why":"Supplies the DoRA parameter-efficient fine-tuning technique used to update attention layers while keeping most weights frozen.","marker":"[51]"}],"fun_headline_variants":["Personalized brain tuning lifts CLIP perception match 2.4x","CLIP plus your brain signals doubles perceptual alignment","MEG-tuned CLIP matches human similarity at 0.78 correlation","Brain signals reshape CLIP to track individual perception"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The neural-alignment claims stand or fall on treating the MEG decoding RDMs, computed by linear-discriminant and support-vector classifiers and averaged over just three participants for group-level training, as faithful, genuinely time-resolved measurements of object representations in the brain; if those RDMs mostly reflect classifier bias or averaging artifacts, the dynamic fine-tuning objective would be fitting noise, and the claim that the model tracks the temporal evolution of individual neural responses would not follow.","fun_headline_variants_meta":{"raw":{"variants":["Personalized brain tuning lifts CLIP perception match 2.4x","CLIP plus your brain signals doubles perceptual alignment","MEG-tuned CLIP matches human similarity at 0.78 correlation","Brain signals reshape CLIP to track individual perception"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000366,"raw_usage":{"total_tokens":1975,"prompt_tokens":956,"completion_tokens":1019,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":949}},"tokens_in":572,"tokens_out":1019,"duration_ms":8013,"temperature":1.0,"reasoning_tokens":949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:57:24.493322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the group-level or individual CLIP-HBA-MEG pipeline on MEG decoding RDMs whose stimulus labels have been randomly permuted, or on RDMs generated from shuffled classifier outputs, then measure the model's temporal alignment with genuine held-out MEG data. If the permuted-target model still shows the reported peak alignment around 300–400 ms or Spearman correlations near the real model's, the objective was fitting classifier artifacts rather than neural structure; if alignment collapses to chance, the dynamic fine-tuning is genuinely reading representational content.","supporting_citations":[],"review_version":1}