Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Correspondence of high-dimensional emotion structures elicited by video clips between humans and Multimodal LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read State-of-the-art multimodal language models reproduce the category-level emotion structure of humans but fail at the level of individual videos.

desk verdict MLLMs capture category-level but not item-level emotion structure from video; the dissociation is real, but the GWOT optimization and a chance-level error need fixing. read the letter →

arxiv 2505.12746 v2 pith:6KHXPW75 submitted 2025-05-19 cs.AI

Haruka Asanuma , Naoko Koide-Majima , Ken Nakamura , Takato Horii , Shinji Nishimoto , Masafumi Oizumi This is my paper · ORCID
classification cs.AI
keywords multimodallargelanguagemodelsemotionstructurerepresentationalsimilarityanalysisGromov-Wassersteinoptimaltransportvideoratingsaffectivecomputingzero-shotinferencehigh-dimensional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether current multimodal large language models (MLLMs) can reproduce the high-dimensional structure of human emotions elicited by short videos. The authors compare human emotion ratings from two large video datasets with ratings generated by three MLLMs, using both a supervised correlation between similarity matrices and an unsupervised alignment method. They find that model and human emotion structures correlate moderately strongly, that models match human ratings at the coarse level of emotion categories about half the time, and that strict one-to-one matching of individual videos remains close to chance. The paper concludes that current MLLMs broadly capture the category-level organization of human emotions but not the fine-grained item-level structure. If true, this suggests MLLMs can serve as useful coarse models of affective experience while remaining unreliable for predicting the emotion of a specific video.

What carries the argument

The central machinery is Gromov-Wasserstein optimal transport (GWOT) applied to representational dissimilarity matrices (RDMs) of emotion ratings. Each RDM records how dissimilar every pair of videos is in emotion space, and GWOT finds the permutation of one set of videos that best matches the other's dissimilarity geometry without knowing which videos correspond. Contrasting the supervised RDM correlation with the unsupervised GWOT matching rates is what lets the authors separate item-level agreement from category-level agreement.

What would settle it

Rerun the alignment on the 2,184-video dataset with many more random restarts (for example, 10,000) or with entropic regularization and convergence checks, and compare the one-to-one match rate with the reported 1.69%; if the rate rises substantially toward the 16% human-human upper bound, the claim of no item-level correspondence is falsified. A simpler check is to verify on a few-hundred-video subset, where 10,000 restarts are feasible, that the 200-restart solution achieves the same Gromov-Wasserstein distance as the best solution found.

Watch

Extended reading notes

Core claim

Using two published datasets of human self-reported emotions during video viewing, the authors find that the representational dissimilarity matrix built from one leading MLLM's ratings correlates with the human matrix at about 0.55 for all videos in both datasets, with other models scoring 0.486 and 0.229. When the emotion structures are aligned by Gromov-Wasserstein optimal transport without using video labels, the one-to-one matching rate is 2.36% and 1.69% for the two datasets, close to chance, while category matching reaches 50.5% and 54.6%. The authors interpret this contrast as evidence that models capture the high-dimensional emotion structure at the level of video categories that elicit similar emotions, but do not yet capture the entire structure at the single-item level.

Load-bearing premise

The central negative claim depends on the assumption that the matching algorithm, run with 200 random starting points and no smoothing, actually finds the best possible structural alignment; if it gets stuck in poor solutions, the low one-to-one match rate may reflect optimization failure rather than a genuine absence of item-level correspondence.

Editorial extensions

If this is right

  • Current MLLMs can be used in zero-shot settings to approximate the coarse category-level emotion structure of video collections.
  • The model ranking found in the paper implies that stronger visual recognition and instruction following improve emotion-structure inference.
  • Emotions that are readable from a single frame are estimated reliably, while context-dependent social emotions remain poorly captured.
  • Fine-grained item-level emotion prediction, where each video must be assigned its correct emotion profile, is not yet reliable and remains far below human-human agreement.
  • Selecting the videos a model estimates best raises one-to-one matching to near the human-human upper bound, indicating partial structural capture rather than uniform failure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the category-level alignment reflects genuine affective structure, MLLMs could serve as rapid coarse emotion annotators for large video corpora, with human validation reserved for item-level labels.
  • The low one-to-one matching rate could partly reflect the difficulty of optimizing GWOT on thousands of items, so a fair test of item-level correspondence would require stronger optimization or entropic regularization.
  • The same structural comparison could be run on audio-only or text-description inputs to test whether category-level emotion correspondence persists when visual features are removed.
  • A natural extension is to compare model RDMs against individual human participant RDMs, which would show whether a model sits inside the distribution of human emotion structures or at its edge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper compares human self-reported emotion ratings for video clips (Koide-Majima et al. 2020 and Cowen & Keltner 2017) with ratings predicted by three multimodal LLMs (Gemini-2.0-flash, GPT-4.1, and Molmo-7B-D). Using representational similarity analysis (RSA) and Gromov-Wasserstein optimal transport (GWOT) matching rates at one-to-one and category levels, it reports moderately high RSA correlations between human and model RDMs (around 0.55 for Gemini on both datasets), low one-to-one GWOT matching rates (around 2% or below), and substantially higher category-level matching rates (around 50-55% for Gemini on all videos). The authors conclude that current MLLMs capture high-dimensional emotion structure at a coarse categorical level but not at a fine item-level scale.

Significance. If the conclusions hold, the paper provides a useful benchmark and a methodological template for evaluating whether MLLMs align with human affective structure, a topic of growing interest in affective computing and cognitive AI. The main strengths are the use of two independent datasets, three models of different provenance, hypothesis-free structural comparisons, explicit shuffled baselines, and qualitative error analyses that connect performance to visual versus contextual video content. The RSA and category-level matching results appear robust: they are far above shuffled-baseline intervals and replicate across datasets and models. However, the central negative claim about item-level structure is contingent on the reliability of the GWOT optimization and on the absence of a same-dataset human-human baseline for the larger dataset, so the headline conclusion needs additional support before it can be taken at face value.

major comments (4)
  1. [Methods, 'Gromov-Wasserstein Optimal Transport'; Table 4] The one-to-one matching rate of 1.69% for the 2,184-video Cowen & Keltner dataset is reported as evidence that MLLMs fail to capture item-level emotion structure, but it is obtained from only 200 random initializations of the unregularized GWOT problem, which is a non-convex quadratic assignment problem. The reported value is therefore the best solution found, i.e., an upper bound on achievable alignment, not a lower bound; if the optimizer is trapped in a locally optimal but item-level scrambled solution, the low rate would be an artifact of the solver rather than of the MLLM. Please provide evidence that the reported solutions are near-global, for example by reporting the distribution over initializations, comparing with entropic GWOT/Sinkhorn solutions, or running a synthetic recovery experiment with known ground-truth correspondences, or alternatively restrict the conclusion to 'the best GWOT alignment we found'.
  2. [Methods, 'Histogram matching'] Before computing GWOT, the model RDM entries are rank-replaced by the human RDM's marginal distribution (v(r) <- u(r)). This is a strong preprocessing step that enforces identical univariate similarity distributions and can change the OT objective and the resulting matching rates. Please report GWOT results without this transformation and clarify explicitly whether the shuffled-human control pipeline applies the same transformation; if it does not, the chance-level comparisons in Tables 2 and 4 are not on equal footing.
  3. [Results, 'Evaluation of MLLM's estimation ... Cowen & Keltner (2017)'; Table 4] The interpretation of category-level matching (54.6% for Gemini on all videos) as 'substantial' is calibrated against the Koide-Majima human-human upper bound (66.18%), but no human-human GWOT baseline exists for the Cowen & Keltner dataset, which differs in stimulus length, emotion taxonomy, and rating format. Please provide a same-dataset human-human comparison if participant-level data can be obtained, or explicitly state that the category-level interpretation relative to a human upper bound is provisional for this dataset.
  4. [Results, 'Similarity structure of all and selected videos' (Cowen & Keltner section)] The top-video analyses (top 100/250/750 videos selected by per-video correlation with human ratings) are used to argue that models partially capture emotion structure. Because selection and evaluation share the same per-video correlation measure, the high matching rates on the selected subsets are partly by construction; the authors acknowledge this in the Koide-Majima section, but the later statement that 'Gemini was able to estimate the structure of a part of the videos well enough that unsupervised mapping is possible' should be explicitly framed as a descriptive, selection-conditioned quantity rather than as independent evidence of partial structural alignment.
minor comments (6)
  1. [Abstract] The word 'signle' should be corrected to 'single'.
  2. [Methods heading] The heading 'Gromv-Wasserstein Optimal Transport' should read 'Gromov-Wasserstein Optimal Transport'.
  3. [Results, Cowen & Keltner subsection] The text refers to 'the left panel in Figure 5' when discussing the Cowen & Keltner RDM panels; this should be Figure 6B.
  4. [Figure 6 caption] Panel D is labeled 'Transportation plan of top 250 selected videos' in the caption, but the main text describes top 750 results for panel D; please reconcile the caption with the text.
  5. [Tables 2 and 4] Several shuffled-GWOT entries report a single value without a 95% interval (for example 0.667%, 0.229%, and 21.2% in Table 4); since the Methods state that only one shuffle was performed for datasets above 500 videos, please state this limitation next to each affected entry rather than only in the Methods.
  6. [Discussion] The statistic that GPT-4V produced responses for only 507 of 2,185 videos is presented without a methods description or data citation; please document how this was measured and where the supporting data can be found.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main comparisons are zero-shot MLLM outputs against human data, and the GWOT matching rates are genuine optimization outcomes, not predetermined by the preprocessing or by self-citations.

full rationale

The paper's central derivation is not circular. Emotion ratings from MLLMs are collected zero-shot with no parameters fitted to human ratings, and the RSA correlation and GWOT matching rates are subsequently computed on those fixed outputs. The histogram-matching preprocessing equalizes the marginal distribution of pairwise dissimilarities between the human and model RDMs, but it does not fix the transport plan or the one-to-one matching rate: the GWOT objective is still a quadratic assignment problem whose solution depends on the model's rank-order structure, and the low item-level matching rates (e.g., 1.69% on the Cowen & Keltner dataset) are not forced by this equalization. The category-level analysis uses human-derived hierarchical clusters as ground-truth labels, which is a standard evaluation choice rather than a self-referential definition of the result. The self-citations to GWTune and earlier unsupervised-alignment papers are methodological and implementation references, not load-bearing uniqueness theorems or unverified premises that determine the conclusions. The reported use of only 200 random initializations for the largest GWOT problem raises a legitimate robustness concern about local minima, but that is a correctness risk, not a circularity: the matching rate is the best found solution, and any shortfall would be an optimization artifact rather than an input-output identity. No step in the paper equates a prediction with a fitted input by construction, renames a known result, or imports a conclusion solely through self-citation.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the representational validity of cosine-dissimilarity RDMs, the reliability of GWOT optimization, the treatment of selection proportions as intensity, the stability of repeated model outputs, the histogram-matching normalization, and the approximate independence of human split groups. These are domain assumptions, not standard mathematical axioms. The main free parameters are analysis hyperparameters; none are fitted to force the conclusion.

free parameters (4)
  • number of hierarchical clusters = 10
    Chosen for visual clarity; pilot analyses across 10-30 clusters gave 'virtually identical qualitative results' (Methods, Determination of video categories).
  • number of extracted frames per video = 6
    Chosen after exploring how many frames remain identifiable; affects GPT-4.1 and Molmo inputs for the Cowen & Keltner dataset (Methods, Selection of Multimodal LLMs).
  • number of response repetitions per video = 3 (non-sensitive), 10 (sensitive)
    A preliminary experiment showed a fourth response did not improve correlation with human ratings; sensitive videos were retried up to 10 times (Methods, Collecting responses from Models).
  • GWOT random initialization count = 10,000 (<=500 videos), 1,000 (~750), 200 (~2,000)
    Adjusted to matrix scale to balance computational efficiency and alignment accuracy (Methods, Gromov-Wasserstein Optimal Transport).
assumptions (6)
  • domain assumption Cosine dissimilarity of emotion-rating vectors is a valid representation of emotion similarity structure
    Used to construct RDMs (Methods, Representational Similarity Analysis). If this metric distorts perceived emotion relations, all structural comparisons inherit the distortion.
  • domain assumption GWOT optimization with random initializations finds near-global optima
    The paper treats the best of 200-10,000 runs as the optimal transport plan; local minima could lower matching rates (Methods, Gromov-Wasserstein Optimal Transport).
  • domain assumption Proportions of raters selecting each emotion in Cowen & Keltner can be treated as emotion intensity
    The paper uses averaged selection proportions as intensity values and compares them with model 0-9 ratings (Methods, Dealing with human ratings).
  • domain assumption Averaging up to three API responses yields a stable model estimate
    Model outputs are averaged over repeated calls without reporting variance; a preliminary experiment is cited as justification (Methods, Collecting responses from Models).
  • domain assumption Histogram matching preserves structural information while removing scale differences
    Rank-wise replacement of model RDM entries with the human marginal distribution is claimed not to alter pairwise ordering or introduce cross-sample correspondence (Methods, Histogram matching).
  • domain assumption The participant-group split in Koide-Majima data is sufficiently independent for an upper-bound estimate
    Participants may overlap across emotion categories between groups, though not within the same category (Methods, Dealing with human ratings).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Correspondence of high-dimensional emotion structures elicited by video clips between humans and Multimodal LLMs." pith.science (2026). https://pith.science/paper/6KHXPW75

@misc{pith2026250512746,
  author       = {Pith},
  title        = {Pith review of: Correspondence of high-dimensional emotion structures elicited by video clips between humans and Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KHXPW75}},
  note         = {Machine review of arXiv:2505.12746}
}
read the original abstract

Recent studies have revealed that human emotions exhibit a high-dimensional, complex structure. A full capturing of this complexity requires new approaches, as conventional models that disregard high dimensionality risk overlooking key nuances of human emotions. Here, we examined the extent to which the latest generation of rapidly evolving Multimodal Large Language Models (MLLMs) capture these high-dimensional, intricate emotion structures, including capabilities and limitations. Specifically, we compared self-reported emotion ratings from participants watching videos with model-generated estimates (e.g., Gemini or GPT). We evaluated performance not only at the individual video level but also from emotion structures that account for inter-video relationships. At the level of simple correlation between emotion structures, our results demonstrated strong similarity between human and model-inferred emotion structures. To further explore whether the similarity between humans and models is at the signle item level or the coarse-categorical level, we applied Gromov Wasserstein Optimal Transport. We found that although performance was not necessarily high at the strict, single-item level, performance across video categories that elicit similar emotions was substantial, indicating that the model could infer human emotional experiences at the category level. Our results suggest that current state-of-the-art MLLMs broadly capture the complex high-dimensional emotion structures at the category level, as well as their apparent limitations in accurately capturing entire structures at the single-item level.

Figures

Figures reproduced from arXiv: 2505.12746 by the authors.

Figure 1
Figure 1. Overview of the analytical framework for comparing similarity structures of emotions across two domains (e.g., humans vs. model). A: Acquisition of emotion ratings. Participants and models watch a series of video clips and report emotion ratings on multiple dimensions, such as calmness, joy, horror, anger. The elements of the matrix represent the intensity of each emotion category for each video reported by particip… view at source ↗
Figure 2
Figure 2. Histogram of the Pearson correlation for each video clip between human ratings in the Koide-Majima et al. dataset. The blue histogram represents the distribution of the correlation between the ratings of Participant group 1 and group 2 participants for each video, and the gray histogram represents the distribution of the correlation between the Participant group 1 ratings and the shuffled Participant group 1 ratings… view at source ↗
Figure 3
Figure 3. Unsupervised comparison of the similarity structures for all videos in the Koide-Majima et al. dataset between Participant group 1 and group 2 based on Gromov-Wasserstein Optimal Transport (GWOT). A: Representation Dissimilarity Matrices (RDMs) of Participant group 1 and group 2. The elements of the RDMs represent the dissimilarity between the emotion ratings of the videos, quantified by cosine similarity. B: Optima… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The histograms show the Pearson correlation of each video clip between the human ratings and Gemini’s estimation in the Koide-Majima et al. dataset. The blue histogram represents the distribution of the correlation between the human ratings and the Gemini’s estimation …
Figure 5
Figure 5. Figure 5: Unsupervised comparison of human similarity structures of videos in the Koide-Majima et al. dataset with similarity structures estimated by Gemini based on Gromov-Wasserstein Optimal Transport (GWOT). A: The Representation Dissimilarity Matrices (RDMs) of the human par…
Figure 6
Figure 6. Figure 6: Comparison of the human similarity structures of videos in the Cowen & Keltner dataset with the similarity structures estimated by Gemini. A: Histograms of the Pearson correlation of each video clip between the human ratings and Gemini’s estimation. The blue histogram …
Figure 7
Figure 7. Figure 7: Schematic of the Gromov-Wasserstein optimal transport. A: Each element of D and D ′ represents the dissimilarity between the emotion ratings of the videos. The optimal transportation plan Γ is obtained by minimizing the Gromov-Wasserstein distance (GWD) between the two…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 42 canonical work pages

  1. [1]

    Cowen, A. S. & Keltner, D. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proc. Natl. Acad. Sci. U. S. A. 114, E7900–E7909 (2017)

    work page 2017

  2. [2]

    & Nishimoto, S

    Koide-Majima, N., Nakai, T. & Nishimoto, S. Distinct dimensions of emotion in the human brain and their representation on the cortical surface. Neuroimage 222, 117258 (2020)

    work page 2020

  3. [3]

    & Friesen, W

    Ekman, P. & Friesen, W. V . Constants across cultures in the face and emotion.J. Pers. Soc. Psychol. 17, 124–129 (1971)

    work page 1971

  4. [4]

    Russell, J. A. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)

    work page 1980

  5. [5]

    Russell, J. A. & Barrett, L. F. Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. J. Pers. Soc. Psychol. 76, 805–819 (1999)

    work page 1999

  6. [6]

    Maithri, M. et al. Automated emotion recognition: Current trends and future perspectives. Comput. Methods Programs Biomed. 215, 106646 (2022)

    work page 2022

  7. [7]

    Designing sociable robots (MIT press, 2004)

    Breazeal, C. Designing sociable robots (MIT press, 2004)

    work page 2004

  8. [8]

    & Cowen, A

    Keltner, D., Sauter, D., Tracy, J. & Cowen, A. Emotional expression: Advances in basic emotion theory. J. Nonverbal Behav. 43, 133–160 (2019)

    work page 2019

Show all 43 references
  1. [9]

    Foteinopoulou, N. M. & Patras, I. Emoclip: A vision-language method for zero-shot video facial expression recognition. 2024 IEEE 18th Int. Conf. on Autom. Face Gesture Recognit. (FG)1–10 (2024)

    2024

  2. [10]

    Lian, Z. et al. GPT-4V with emotion: A zero-shot benchmark for generalized emotion recognition. Inf. Fusion 108, 102367 (2024)

    2024

  3. [11]

    Liang, P. et al. Holistic evaluation of language models. arXiv [cs.CL] (2022)

    2022

  4. [12]

    Zeng, A. et al. GLM-130B: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations (2023). 23/25

    2023

  5. [13]

    C., Thung, F

    Zhang, T., Irsan, I. C., Thung, F. & Lo, D. Revisiting sentiment analysis for software engineering in the era of large language models. ACM Trans. Softw. Eng. Methodol.34, 1–30 (2025)

    2025

  6. [14]

    & Wang, P

    Li, A., Xu, L., Ling, C., Zhang, J. & Wang, P. EmoVerse: Exploring multimodal large language models for sentiment and emotion understanding. arXiv [cs.CL] (2024)

    2024

  7. [15]

    Cheng, Z. et al. Emotion-LLaMA: Multimodal emotion recognition and reasoning with instruction tuning. arXiv [cs.AI] (2024)

    2024

  8. [16]

    Fei, H. et al. Video-of-thought: Step-by-step video reasoning from perception to cognition. arXiv [cs.AI] (2024)

    2024

  9. [17]

    & Oizumi, M

    Kawakita, G., Zeleznikow-Johnston, A., Tsuchiya, N. & Oizumi, M. Gromov–Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language models. Sci. Rep. 14, 1–10 (2024)

    2024

  10. [18]

    & Griffiths, T

    Marjieh, R., Sucholutsky, I., van Rijn, P., Jacoby, N. & Griffiths, T. L. Large language models predict human sensory judgments across six modalities. Sci. Rep. 14, 21445 (2024)

    2024

  11. [19]

    Introducing gemini: our largest and most capable AI model

    Pichai, S. Introducing gemini: our largest and most capable AI model. https://blog.google/technology/ai/google-gemini-ai/ (2023). Accessed: 2025-3-28

    2023

  12. [20]

    Open, A. I. Introducing ChatGPT. https://openai.com/index/chatgpt/ (2022). Accessed: 2025-3-28

    2022

  13. [21]

    A., Neupert, S

    DiGirolamo, M. A., Neupert, S. D. & Isaacowitz, D. M. Emotion regulation convoys: Individual and age differences in the hierarchical configuration of emotion regulation behaviors in everyday life. Affect. Sci. 4, 630–643 (2023)

    2023

  14. [22]

    Barrett, L. F. The theory of constructed emotion: an active inference account of interoception and categorization. Soc. Cogn. Affect. Neurosci. 12, 1–23 (2017)

    2017

  15. [23]

    & Garza, P

    Vaiani, L., Cagliero, L. & Garza, P. Emotion recognition from videos using multimodal large language models. Futur. Internet 16, 247 (2024)

    2024

  16. [24]

    & Oizumi, M

    Takeda, K., Abe, K., Kitazono, J. & Oizumi, M. Unsupervised alignment reveals structural commonalities and differences in neural representations of natural scenes across individuals and brain areas. iScience 28, 112427 (2025)

    2025

  17. [25]

    & Oizumi, M

    Takeda, K., Sasaki, M., Abe, K. & Oizumi, M. Unsupervised alignment in neuroscience: Introducing a toolbox for gromov-wasserstein optimal transport. J. Neurosci. Methods 419, 110443 (2025)

    2025

  18. [26]

    red” your “red

    Kawakita, G., Zeleznikow-Johnston, A., Takeda, K., Tsuchiya, N. & Oizumi, M. Is my “red” your “red”?: Evaluating structural correspondences between color similarity judgments using unsupervised alignment. iScience 28, 112029 (2025)

    2025

  19. [27]

    Open, A. I. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/ (2025). Accessed: 2025-5-17

    2025

  20. [28]

    Deitke, M. et al. Molmo and PixMo: Open weights and open data for state-of-the-art multimodal models. arXiv [cs.CV] (2024)

    2024

  21. [29]

    & Hoi, S

    Li, J., Li, D., Savarese, S. & Hoi, S. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 19730–19742 (PMLR, 2023)

    2023

  22. [30]

    & Lee, Y

    Liu, H., Li, C., Li, Y . & Lee, Y . J. Improved baselines with visual instruction tuning. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit.26286–26296 (2023)

    2023

  23. [31]

    Fu, C. et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv [cs.CV] (2023)

    2023

  24. [32]

    GPT-4 technical report

    OpenAI et al. GPT-4 technical report. arXiv [cs.CL] (2023)

    2023

  25. [33]

    Can’t see the forest for the trees: Benchmarking multimodal safety awareness for multimodal LLMs

    Wang, W.et al. Can’t see the forest for the trees: Benchmarking multimodal safety awareness for multimodal LLMs. arXiv [cs.CL] (2025)

    2025

  26. [34]

    Mallick, S. B. & Kilpatrick, L. Gemini 2.0: Flash, flash-lite and pro. https://developers.googleblog.com/en/ gemini-2-family-expands/ (2025). Accessed: 2025-4-28

    2025

  27. [35]

    & Parikh, D

    Goyal, Y ., Khot, T., Summers-Stay, D., Batra, D. & Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6904–6913 (IEEE, 2017)

    2017

  28. [36]

    Singh, A. et al. Towards VQA models that can read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2019)

    2019

  29. [37]

    A., Reddan, M

    Kragel, P. A., Reddan, M. C., LaBar, K. S. & Wager, T. D. Emotion schemas are embedded in the human visual system. Sci Adv 5, eaaw4358 (2019). 24/25

    2019

  30. [38]

    Barrett, L. F. How emotions are made: The secret life of the brain. Pan Macmillan (2017)

    2017

  31. [39]

    Predictive processing of interoception, decision-making, and allostasis: A computational framework and implications for emotional intelligence

    Ohira, H. Predictive processing of interoception, decision-making, and allostasis: A computational framework and implications for emotional intelligence. Psihol. Teme 29, 1–16 (2020)

    2020

  32. [40]

    Perception test: A diagnostic benchmark for multimodal video models

    Puatruaucean, V .et al. Perception test: A diagnostic benchmark for multimodal video models. Neural Inf Process. Syst abs/2305.13786, 42748–42761 (2023)

    2023 arXiv

  33. [41]

    Grok-1.5 vision preview

    xAI. Grok-1.5 vision preview. https://x.ai/news/grok-1.5v (2024). Accessed: 2025-4-25

    2024

  34. [42]

    Chiang, W.-L. et al. Chatbot arena: An open platform for evaluating LLMs by human preference. Forty-first Int. Conf. on Mach. Learn. (2024)

    2024

  35. [43]

    Grattafiori, A. et al. The llama 3 herd of models. arXiv [cs.AI] (2024). Acknowledgements M.O. was supported by JST Moonshot R&D Grant No. JPMJMS2012 and JSPS KAKENHI, Grant Number 20H05712. M.O. and T.H. were supported by JSPS KAKENHI Grant Number 23H04834. S.N. was supported...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.