{"id":"9714d7aa-2168-44a7-a7a9-2a158945d5cc","arxiv_id":"2505.12746","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Multimodal LLMs capture the coarse category-level structure of human emotional responses to videos, but not fine-grained item-level emotion structure.","lead":"The paper compares human emotion ratings from video watching with estimates from multimodal AI models (Gemini, GPT-4.1, Molmo). It finds that modern AI models broadly match how humans group videos into emotion categories, but they fail to match the precise one-to-one emotional structure of individual videos.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.69% one-to-one matching rate for the 2,184-video Cowen & Keltner dataset rests on GWOT solutions from only 200 random initializations with no entropic regularization; local minima could be mistaken for a genuine item-level model limitation.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the unregularized GWOT solutions on the large dataset may be far from the global optimum, making the low one-to-one matching rate an unreliable basis for the negative half of the central claim. I agree with that assessment and do not find a separate concern that is more decisive. The positive category-level result is supported by large margins over shuffled baselines and is replicated across multiple models and both datasets, so the conditional status is driven primarily by the item-level claim. The proposed test—more initializations or entropic regularization, plus a synthetic control—would settle whether the low one-to-one rate reflects the model or the optimizer. Since the reader already made the verdict CONDITIONAL for this reason, my stress-test does not change the recommended verdict; it reinforces the condition and specifies how to resolve it.","tokens_in":24417,"tokens_out":11457,"duration_ms":138415,"concrete_test":"Rerun the human-vs-Gemini GWOT on the 2,184-video Cowen & Keltner RDMs with the same histogram-matching preprocessing, but (a) increase random initializations to at least 1,000, or (b) use entropic Gromov-Wasserstein (Sinkhorn, small regularization) with multiple restarts, and record both one-to-one and category matching rates. If the one-to-one rate rises substantially above 1.69% (e.g., into the 5-10% range or toward the Koide human-human rate), the reported item-level limitation is partly an optimization artifact; if it remains below about 2%, the concern does not land. As a control, run the same pipeline on a synthetic model RDM built by permuting a known subset of rows/columns of the human RDM, and verify that 200 initializations can recover item-level matches when the ground-truth mapping is present.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative conclusion—that MLLMs capture category-level but not item-level emotion structure—depends on the one-to-one GWOT matching rate being a faithful estimate of item-level structural correspondence. In Methods (\"Gromv-Wasserstein Optimal Transport\"), the authors solve the unregularized Gromov-Wasserstein problem, which is a quadratic assignment problem, and for matrices \"approaching 2,000 videos\" they use 200 random initializations. For n=2,184 this is a vast nonconvex search space; 200 restarts cannot certify the global optimum, and the reported 1.69% rate is the best solution found, i.e., an upper bound on achievable alignment, not a lower bound on the model's actual correspondence. If the optimizer is trapped in a structurally sensible but item-level scrambled local minimum, the low one-to-one rate would be an artifact of the solver, not of the MLLM. The category-level result (54.6%) is less vulnerable to this concern because it only requires cluster-level alignment, which a local minimum can satisfy while getting individual items wrong. A related but secondary issue is that no human-human GWOT baseline exists for the Cowen & Keltner dataset, so the item-level comparison is calibrated against a different, smaller dataset; however, the optimization uncertainty alone is enough to make the item-level negative claim conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares human self-reported emotion ratings for video clips (Koide-Majima et al. 2020 and Cowen & Keltner 2017) with ratings predicted by three multimodal LLMs (Gemini-2.0-flash, GPT-4.1, and Molmo-7B-D). Using representational similarity analysis (RSA) and Gromov-Wasserstein optimal transport (GWOT) matching rates at one-to-one and category levels, it reports moderately high RSA correlations between human and model RDMs (around 0.55 for Gemini on both datasets), low one-to-one GWOT matching rates (around 2% or below), and substantially higher category-level matching rates (around 50-55% for Gemini on all videos). The authors conclude that current MLLMs capture high-dimensional emotion structure at a coarse categorical level but not at a fine item-level scale.","tokens_in":24499,"tokens_out":5150,"duration_ms":58547,"significance":"If the conclusions hold, the paper provides a useful benchmark and a methodological template for evaluating whether MLLMs align with human affective structure, a topic of growing interest in affective computing and cognitive AI. The main strengths are the use of two independent datasets, three models of different provenance, hypothesis-free structural comparisons, explicit shuffled baselines, and qualitative error analyses that connect performance to visual versus contextual video content. The RSA and category-level matching results appear robust: they are far above shuffled-baseline intervals and replicate across datasets and models. However, the central negative claim about item-level structure is contingent on the reliability of the GWOT optimization and on the absence of a same-dataset human-human baseline for the larger dataset, so the headline conclusion needs additional support before it can be taken at face value.","major_comments":[{"comment":"The one-to-one matching rate of 1.69% for the 2,184-video Cowen & Keltner dataset is reported as evidence that MLLMs fail to capture item-level emotion structure, but it is obtained from only 200 random initializations of the unregularized GWOT problem, which is a non-convex quadratic assignment problem. The reported value is therefore the best solution found, i.e., an upper bound on achievable alignment, not a lower bound; if the optimizer is trapped in a locally optimal but item-level scrambled solution, the low rate would be an artifact of the solver rather than of the MLLM. Please provide evidence that the reported solutions are near-global, for example by reporting the distribution over initializations, comparing with entropic GWOT/Sinkhorn solutions, or running a synthetic recovery experiment with known ground-truth correspondences, or alternatively restrict the conclusion to 'the best GWOT alignment we found'.","section":"Methods, 'Gromov-Wasserstein Optimal Transport'; Table 4"},{"comment":"Before computing GWOT, the model RDM entries are rank-replaced by the human RDM's marginal distribution (v(r) <- u(r)). This is a strong preprocessing step that enforces identical univariate similarity distributions and can change the OT objective and the resulting matching rates. Please report GWOT results without this transformation and clarify explicitly whether the shuffled-human control pipeline applies the same transformation; if it does not, the chance-level comparisons in Tables 2 and 4 are not on equal footing.","section":"Methods, 'Histogram matching'"},{"comment":"The interpretation of category-level matching (54.6% for Gemini on all videos) as 'substantial' is calibrated against the Koide-Majima human-human upper bound (66.18%), but no human-human GWOT baseline exists for the Cowen & Keltner dataset, which differs in stimulus length, emotion taxonomy, and rating format. Please provide a same-dataset human-human comparison if participant-level data can be obtained, or explicitly state that the category-level interpretation relative to a human upper bound is provisional for this dataset.","section":"Results, 'Evaluation of MLLM's estimation ... Cowen & Keltner (2017)'; Table 4"},{"comment":"The top-video analyses (top 100/250/750 videos selected by per-video correlation with human ratings) are used to argue that models partially capture emotion structure. Because selection and evaluation share the same per-video correlation measure, the high matching rates on the selected subsets are partly by construction; the authors acknowledge this in the Koide-Majima section, but the later statement that 'Gemini was able to estimate the structure of a part of the videos well enough that unsupervised mapping is possible' should be explicitly framed as a descriptive, selection-conditioned quantity rather than as independent evidence of partial structural alignment.","section":"Results, 'Similarity structure of all and selected videos' (Cowen & Keltner section)"}],"minor_comments":[{"comment":"The word 'signle' should be corrected to 'single'.","section":"Abstract"},{"comment":"The heading 'Gromv-Wasserstein Optimal Transport' should read 'Gromov-Wasserstein Optimal Transport'.","section":"Methods heading"},{"comment":"The text refers to 'the left panel in Figure 5' when discussing the Cowen & Keltner RDM panels; this should be Figure 6B.","section":"Results, Cowen & Keltner subsection"},{"comment":"Panel D is labeled 'Transportation plan of top 250 selected videos' in the caption, but the main text describes top 750 results for panel D; please reconcile the caption with the text.","section":"Figure 6 caption"},{"comment":"Several shuffled-GWOT entries report a single value without a 95% interval (for example 0.667%, 0.229%, and 21.2% in Table 4); since the Methods state that only one shuffle was performed for datasets above 500 videos, please state this limitation next to each affected entry rather than only in the Methods.","section":"Tables 2 and 4"},{"comment":"The statistic that GPT-4V produced responses for only 507 of 2,185 videos is presented without a methods description or data citation; please document how this was measured and where the supporting data can be found.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the RSA/category-level results are likely acceptable after revision. The main technical risk is the unregularized GWOT optimization used for the item-level negative claim; validation via entropic GWOT or synthetic recovery would address my primary concern. I also suggest the editor ask the authors to verify whether participant-level data for Cowen & Keltner (2017) are actually unavailable, because if they are available, a same-dataset human-human GWOT baseline would meaningfully strengthen the calibration of the category-level result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful empirical paper. The main claim—that Gemini, GPT-4.1, and Molmo reproduce coarse human emotion categories from short video clips but fall far short at exact one-to-one alignment—looks true and is worth taking seriously. The RSA correlations (~0.55) and category matching (~50-55%) sit far above shuffled baselines on two datasets, and the pattern is consistent across three models. The human-human reference from Koide-Majima (RSA 0.86, category 66%, one-to-one 16%) gives a sensible ceiling. This is a new application of the GWOT+RSA toolkit to emotion structure, and the item/category dissociation is a concrete result that the emotion-MLLM papers I've seen haven't demonstrated.\n\nThe soft spots are real but not load-bearing.\n\nFirst, the GWOT on 2,184 videos uses only 200 random initializations without entropic regularization. The 1.69% one-to-one rate is therefore an upper bound on what the optimizer found, and there's no way to know whether a better local minimum would raise it. That said, the same dissociation shows up on the 550-video dataset with 10,000 initializations (one-to-one 2.36% vs human 16.36%), so the qualitative conclusion isn't resting on the large dataset alone. Still, the authors should report a robustness check—say, running more initializations on a 500–1000 video subset—before the item-level negative claim is treated as benchmark-grade.\n\nSecond, the histogram-matching step before GWOT replaces the model's RDM entries with the human marginal distribution. That does introduce human-derived information into the model comparison, and it's only the GWOT numbers affected; RSA is clean. It needs a sensitivity analysis or a clear justification.\n\nThird, the theoretical chance level for one-to-one matching in the 2,184-video dataset is wrong. They quote 0.229%, but 1/2184 is 0.0458%. The 0.229% appears to be from a single shuffle. That makes 'close to chance' an overstatement—1.69% is actually 37x the theoretical null—even though it's still far below the human upper bound.\n\nFourth, no code, data, or model outputs are posted, and they don't report variance across their three model repetitions. For a paper with this much pipeline, that's a review-blocker in practice.\n\nWho's it for: affective computing, MLLM benchmarking, anyone building emotion-labeled datasets from video. It deserves a serious referee; the core result is solid and interesting. My recommendation: send it out, but require code, corrected chance levels, and a GWOT robustness check before acceptance.","headline":"MLLMs capture category-level but not item-level emotion structure from video; the dissociation is real, but the GWOT optimization and a chance-level error need fixing.","tokens_in":25230,"tokens_out":5873,"would_cite":true,"duration_ms":59059,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art multimodal language models reproduce the category-level emotion structure of humans but fail at the level of individual videos.","keywords":["multimodal large language models","emotion structure","representational similarity analysis","Gromov-Wasserstein optimal transport","video emotion ratings","affective computing","zero-shot emotion inference","high-dimensional emotion"],"falsifier":"Rerun the alignment on the 2,184-video dataset with many more random restarts (for example, 10,000) or with entropic regularization and convergence checks, and compare the one-to-one match rate with the reported 1.69%; if the rate rises substantially toward the 16% human-human upper bound, the claim of no item-level correspondence is falsified. A simpler check is to verify on a few-hundred-video subset, where 10,000 restarts are feasible, that the 200-restart solution achieves the same Gromov-Wasserstein distance as the best solution found.","tokens_in":1584,"feed_emoji":"🎭","tokens_out":1785,"duration_ms":56987,"temperature":0.7,"pith_summary":"This paper tests whether current multimodal large language models (MLLMs) can reproduce the high-dimensional structure of human emotions elicited by short videos. The authors compare human emotion ratings from two large video datasets with ratings generated by three MLLMs, using both a supervised correlation between similarity matrices and an unsupervised alignment method. They find that model and human emotion structures correlate moderately strongly, that models match human ratings at the coarse level of emotion categories about half the time, and that strict one-to-one matching of individual videos remains close to chance. The paper concludes that current MLLMs broadly capture the category-level organization of human emotions but not the fine-grained item-level structure. If true, this suggests MLLMs can serve as useful coarse models of affective experience while remaining unreliable for predicting the emotion of a specific video.","feed_headline":"Multimodal AI captures emotion categories but misses individual videos","feed_subtitle":"Compared with human ratings on 2,700+ video clips, top models align ~55% at category level yet only ~2% one-to-one.","key_machinery":"The central machinery is Gromov-Wasserstein optimal transport (GWOT) applied to representational dissimilarity matrices (RDMs) of emotion ratings. Each RDM records how dissimilar every pair of videos is in emotion space, and GWOT finds the permutation of one set of videos that best matches the other's dissimilarity geometry without knowing which videos correspond. Contrasting the supervised RDM correlation with the unsupervised GWOT matching rates is what lets the authors separate item-level agreement from category-level agreement.","core_discovery":"Using two published datasets of human self-reported emotions during video viewing, the authors find that the representational dissimilarity matrix built from one leading MLLM's ratings correlates with the human matrix at about 0.55 for all videos in both datasets, with other models scoring 0.486 and 0.229. When the emotion structures are aligned by Gromov-Wasserstein optimal transport without using video labels, the one-to-one matching rate is 2.36% and 1.69% for the two datasets, close to chance, while category matching reaches 50.5% and 54.6%. The authors interpret this contrast as evidence that models capture the high-dimensional emotion structure at the level of video categories that elicit similar emotions, but do not yet capture the entire structure at the single-item level.","pith_inferences":["If the category-level alignment reflects genuine affective structure, MLLMs could serve as rapid coarse emotion annotators for large video corpora, with human validation reserved for item-level labels.","The low one-to-one matching rate could partly reflect the difficulty of optimizing GWOT on thousands of items, so a fair test of item-level correspondence would require stronger optimization or entropic regularization.","The same structural comparison could be run on audio-only or text-description inputs to test whether category-level emotion correspondence persists when visual features are removed.","A natural extension is to compare model RDMs against individual human participant RDMs, which would show whether a model sits inside the distribution of human emotion structures or at its edge."],"forward_implications":["Current MLLMs can be used in zero-shot settings to approximate the coarse category-level emotion structure of video collections.","The model ranking found in the paper implies that stronger visual recognition and instruction following improve emotion-structure inference.","Emotions that are readable from a single frame are estimated reliably, while context-dependent social emotions remain poorly captured.","Fine-grained item-level emotion prediction, where each video must be assigned its correct emotion profile, is not yet reliable and remains far below human-human agreement.","Selecting the videos a model estimates best raises one-to-one matching to near the human-human upper bound, indicating partial structural capture rather than uniform failure."],"supporting_citations":[{"why":"Supplies the 2,185-video human emotion rating dataset used for the main cross-model comparison.","marker":"[1]"},{"why":"Supplies the 550-video Japanese emotion rating dataset with 80 emotion dimensions and participant-level ratings.","marker":"[2]"},{"why":"Prior application of GWOT to compare human and model similarity structures, motivating the present analysis.","marker":"[17]"},{"why":"Provides the unsupervised alignment methodology for comparing neural and behavioral representational structures.","marker":"[24]"},{"why":"Provides the GWTune toolbox and random-initialization procedure used to compute GWOT alignments.","marker":"[25]"},{"why":"Identifies the primary MLLM whose emotion ratings are compared with human ratings.","marker":"[19]"},{"why":"Identifies the second MLLM evaluated on the English video dataset.","marker":"[27]"},{"why":"Identifies the open-weight MLLM used as a lower-performing comparison baseline.","marker":"[28]"}],"fun_headline_variants":["AI matches humans on emotion categories, not single clips","Emotion AI: 55% category match, 2% single-clip match","Multimodal models see emotion forest, miss the trees","Video emotion AI: broad category fit, poor item precision"],"cache_read_input_tokens":27136,"weakest_assumption_plain":"The central negative claim depends on the assumption that the matching algorithm, run with 200 random starting points and no smoothing, actually finds the best possible structural alignment; if it gets stuck in poor solutions, the low one-to-one match rate may reflect optimization failure rather than a genuine absence of item-level correspondence.","fun_headline_variants_meta":{"raw":{"variants":["AI matches humans on emotion categories, not single clips","Emotion AI: 55% category match, 2% single-clip match","Multimodal models see emotion forest, miss the trees","Video emotion AI: broad category fit, poor item precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1745,"prompt_tokens":960,"completion_tokens":785,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":713}},"tokens_in":576,"tokens_out":785,"duration_ms":8853,"temperature":1.0,"reasoning_tokens":713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:27:50.698803+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the alignment on the 2,184-video dataset with many more random restarts (for example, 10,000) or with entropic regularization and convergence checks, and compare the one-to-one match rate with the reported 1.69%; if the rate rises substantially toward the 16% human-human upper bound, the claim of no item-level correspondence is falsified. A simpler check is to verify on a few-hundred-video subset, where 10,000 restarts are feasible, that the 200-restart solution achieves the same Gromov-Wasserstein distance as the best solution found.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 2,185-video human emotion rating dataset used for the main cross-model comparison."},{"cited_title":"& Nishimoto, S","cited_arxiv_id":null,"evidence_quote":"Supplies the 550-video Japanese emotion rating dataset with 80 emotion dimensions and participant-level ratings."},{"cited_title":"& Oizumi, M","cited_arxiv_id":null,"evidence_quote":"Prior application of GWOT to compare human and model similarity structures, motivating the present analysis."},{"cited_title":"& Oizumi, M","cited_arxiv_id":null,"evidence_quote":"Provides the unsupervised alignment methodology for comparing neural and behavioral representational structures."},{"cited_title":"& Oizumi, M","cited_arxiv_id":null,"evidence_quote":"Provides the GWTune toolbox and random-initialization procedure used to compute GWOT alignments."},{"cited_title":"Introducing gemini: our largest and most capable AI model","cited_arxiv_id":null,"evidence_quote":"Identifies the primary MLLM whose emotion ratings are compared with human ratings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies the second MLLM evaluated on the English video dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies the open-weight MLLM used as a lower-performing comparison baseline."}],"review_version":1}