Pith. sign in

REVIEW 4 major objections 6 minor 55 references

EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EmoSign, the first ASL video dataset with sentiment and emotion labels, shows that current multimodal LLMs recognize almost no emotion from signing alone.

desk verdict A genuinely first ASL emotion dataset with native-signer labels, but the benchmark's ground truth is internally inconsistent and partly noisy — fixable, but the per-emotion numbers shouldn't be trusted yet. read the letter →

arxiv 2505.17090 v1 pith:PZISBHZF submitted 2025-05-20 cs.CV

Phoebe Chua , Cathy Mengying Fang , Takehiko Ohkawa , Raja Kushalnagar , Suranga Nanayakkara , Pattie Maes This is my paper
classification cs.CV
keywords AmericanSignLanguageemotionrecognitionsentimentanalysismultimodaldatasetaffectivecomputingbenchmarkDeafannotators
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EmoSign is a new dataset of 200 American Sign Language video clips, each annotated by Deaf ASL signers for overall sentiment, ten emotion categories, and open-ended descriptions of the visual cues that carry emotion. The paper claims this is the first dedicated sign-language emotion dataset, and it uses the annotations to benchmark four multimodal large language models on sentiment analysis and emotion classification. The benchmark results show that, when given only the signing video, the models perform poorly and fall back on neutral or positive judgments, while access to the English caption substantially improves their scores. The paper also documents how emotion shows up in signing: facial expressions, head and body movements, and modified sign size, speed, or repetition are the main cues, with sentence context needed to disambiguate meaning. A sympathetic reader would take the central contribution to be a new resource and benchmark that tests whether AI systems can separate grammatical from affective functions of facial expression in sign language.

What carries the argument

The dataset itself is the central object. It was built by taking continuous signing videos from the ASLLRP corpus, segmenting them into utterance-level clips, using VADER sentiment scores on the English captions to select 100 clips with the most positive and 100 with the most negative text sentiment, and having three Deaf ASL signers annotate each clip on a 7-point sentiment scale, ten emotion intensity scales, and open-ended cue descriptions. Aggregation uses majority vote with annotator confidence as the tie-breaker, and inter-annotator agreement is measured with Krippendorff's $\alpha$ (average 0.593, but below 0.2 for some negative emotions). The benchmark part uses three tasks—sentiment analysis, single-label emotion classification, and a qualitative emotion-cue grounding inspection—to compare four video-capable multimodal LLMs against the human labels.

What would settle it

Re-annotate the same 200 clips with a larger panel of Deaf ASL signers and compare the resulting labels to EmoSign's ground truth; if the new majority labels flip for a substantial share of clips, especially in low-agreement classes like surprise (negative), disgust, and anger, then the benchmark's reference labels are not stable enough to support strong conclusions about model performance.

Watch

Extended reading notes

Core claim

The paper's claim is that EmoSign is the first sign-language video dataset with sentiment and emotion labels, and that it exposes a large gap between current multimodal LLMs and Deaf human annotators in reading emotion from sign language. On the dataset, the models' visual-only emotion recognition is close to chance-like in practice, with a marked bias toward neutral or positive labels, while adding the video caption improves performance substantially. Alongside the labels, the Deaf annotators' open-ended descriptions reveal that non-manual markers—furrowed brows, pursed lips, head thrusts, shoulder raises—are the primary cues, that sign modification (larger, faster, repeated signs, finger-spelling) carries emotional intensity, and that context is necessary to disambiguate some emotions. The authors present EmoSign as both a resource for studying emotional expression in ASL and a benchmark for multimodal AI.

Load-bearing premise

The load-bearing premise is that three Deaf ASL signers' majority-vote emotion labels are accurate ground truth, even though agreement among them was low for several negative emotion categories; if those labels are unreliable, the benchmark conclusions shift.

Editorial extensions

If this is right

  • EmoSign gives the ASL research community a first shared benchmark for sentiment and emotion classification on continuous signing, with defined single- and multi-expression sets.
  • The documented emotion cues give sign-language linguists and technologists a native-signer perspective on how affective meaning is carried by non-manual markers and sign modification, which can inform more expressive sign-to-speech and interpretation tools.
  • The baseline results show that caption-free emotion recognition is an unsolved problem for current MLLMs, so the dataset can serve as a diagnostic for visual understanding that does not rely on language shortcuts.
  • The large gap between visual-only and caption-conditioned performance suggests that progress on this benchmark will require models that are specifically trained on signed video rather than generic video-language pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because several negative emotion classes had very low inter-annotator agreement ($\alpha < 0.2$), the ground truth for those classes is fragile; a useful next step would be to re-annotate with a larger Deaf panel and per-clip confidence weighting to see which labels are stable.
  • The strong caption-dependence of the models suggests EmoSign could be repurposed as a test of whether a model is genuinely reading visual affect or just exploiting textual sentiment; a model that understands signing should keep most of its accuracy when captions are withheld.
  • The patterns observed here are specific to ASL and to a lab corpus; extending EmoSign-style annotation to other sign languages and to in-the-wild or multi-signer footage would test whether the cue inventory generalizes.
  • The models' neutral and positive bias is consistent with the paper's suggestion that safety-oriented training may push them away from negative judgments, but that hypothesis would need a controlled comparison with an unfiltered model to confirm.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces EmoSign, a dataset of 200 American Sign Language (ASL) video clips sampled from ASLLRP, annotated by three Deaf ASL signers with professional interpretation experience. Each clip receives a 7-point sentiment rating, intensity ratings for ten emotion categories, a confidence rating, and an open-ended description of emotion cues. The paper also presents baseline evaluations of four multimodal LLMs (GPT-4o, AffectGPT, Qwen2.5-VL-7B-Instruct, MiniGPT4) on sentiment analysis and single-label emotion classification, plus a manual inspection of reasoning outputs for emotion cue grounding. The central claim is that EmoSign is the first ASL video dataset with sentiment and emotion labels and that current MLLMs perform poorly on visual-only emotion recognition in ASL.

Significance. If the annotation ground truth is reliable, EmoSign fills a genuine gap: it is the first ASL video dataset designed specifically for affective understanding, annotated by native Deaf signers, and it includes open-ended descriptions of manual and non-manual emotion cues. The public release on Hugging Face, the documentation of prompts and model cards, and the inclusion of baseline comparisons are concrete strengths that support reproducibility. I do not see a circularity problem: the human labels are collected independently of the baseline models, and the VADER-based clip selection shapes the label distribution without defining the ground truth. However, the benchmark conclusions depend on the trustworthiness of the aggregated annotations, and the manuscript currently contains unresolved inconsistencies in annotation coverage, an apparent arithmetic error in the reported average inter-annotator agreement, and incomplete specification of benchmark subsets. These issues must be resolved before the central claims can be fully accepted.

major comments (4)
  1. [Section 3.3 vs Appendix A.1] The manuscript directly contradicts itself on annotation coverage. Section 3.3 states 'Each label of the clip was labeled by minimally 1, maximally 3 annotators, given certain clips were skipped,' while Appendix A.1 states 'All three annotators completed all annotation tasks for all sign videos in the dataset.' This distinction is load-bearing because the majority-vote and confidence tie-breaking aggregation described in Section 3.3 has very different reliability properties under the two scenarios. The authors should report the number of clips annotated by 1, 2, and 3 annotators, and if skipping was allowed, they should explain how skipped clips were handled in the aggregation and why the Appendix statement is worded differently.
  2. [Table 2] The reported average Krippendorff's alpha of 0.593 is not the mean of the eleven alpha values listed in Table 2; those values sum to 4.594, giving an average of 0.418 (or 0.386 if the sentiment row is excluded). The value 0.593 appears to be the average of only sentiment and the positive emotion labels (joy, excited, surprise_pos). This misreporting overstates overall inter-annotator agreement. More importantly, the alpha values for surprise_neg (0.119), disgust (0.166), and anger (0.370) are so low that a majority-vote ground truth for these classes effectively reflects one annotator's judgment, especially when ties are broken by self-reported confidence. The benchmark conclusions for these emotion classes should be re-analyzed or explicitly qualified, for example by agreement-filtered subsets or per-class uncertainty reporting.
  3. [Section 4.1] The construction of the single-expression and multi-expression evaluation sets is under-specified. The paper states that the single-expression set consists of 140 clips and the multi-expression set consists of 37 clips, but these sum to 177, leaving 23 of the 200 dataset clips unaccounted for. The text also says 'The multi-expression set only contains classes that have more than 2 samples,' which is not a definition of the set. The authors need to specify exactly how clips were assigned to these sets, why 23 clips are excluded, and whether the exclusion criteria could bias the benchmark results in Tables 4 and 5.
  4. [Section 4.2 and Section 5] The baseline comparisons do not account for the stochasticity of the evaluated MLLMs or the noise in the ground truth. The paper evaluates GPT-4o, AffectGPT, Qwen2.5-VL-7B-Instruct, and MiniGPT4 with no repeated inference runs, no reported decoding parameters (e.g., temperature), and no error bars or confidence intervals. The NeurIPS checklist justification that 'baseline evaluations did not include factors that require error bars' is not convincing for stochastic models. Since the ground truth for several emotion classes has very low inter-annotator agreement, a single-run model output that disagrees with the aggregated label may match the judgment of an excluded annotator. The claim that MLLMs exhibit significant limitations in visual-only emotion recognition should be supported by uncertainty-aware comparisons, such as multiple sampling runs or agreement-filtered evaluation.
minor comments (6)
  1. [Section 3.2] The sentence 'The annotation process took roughly x hours per individuals' contains an unresolved placeholder 'x' that must be filled in before publication.
  2. [Section 3.2] The text 'See Appendix ?? for details about the annotation interface' references an empty appendix; the cross-reference should be fixed.
  3. [Table 3] The 'Hearing person*' row has an asterisk with no corresponding footnote or explanation in the table caption; the methods for the hearing annotator's rating process are described only in the main text and should be linked explicitly.
  4. [Appendix A.2, Table 5] The model card table lists 'Qwen3-8B*' with no explanation of the asterisk, and it is not clear whether this model was used in the reported baselines; the entry should be clarified or removed.
  5. [Appendix A.5] Figures 22 and 23 appear to have identical captions ('Weighted Accuracy of Models on Single Expression Emotion Classification Task'); one of them likely refers to weighted F1 or another metric and should be relabeled.
  6. [Section 3.4] The paragraph describing common themes in the free-response emotion-cue annotations would benefit from explicit counts or example quotes linked to specific videos, since these qualitative findings are presented as dataset contributions.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity: benchmark outcomes rest on independent human labels; the only self-citation is minor and non-load-bearing.

full rationale

The paper's central contributions are the annotated dataset and the zero-shot MLLM baselines. No fitted parameter is renamed as a prediction: the annotator labels are collected independently of the models, and the baselines are off-the-shelf models evaluated against those labels. VADER is used only to select clips with emotionally salient captions, and the paper explicitly acknowledges this selection effect on the label distribution, so it does not constitute a fitted-input-called-prediction. There is no uniqueness theorem or ansatz imported from the authors' prior work. Two non-circular weaknesses should be weighed in a quality assessment: (1) Section 3.3 says 'Each label of the clip was labeled by minimally 1, maximally 3 annotators, given certain clips were skipped' while Appendix A.1 says 'All three annotators completed all annotation tasks for all sign videos in the dataset'; these statements directly contradict each other and the paper never reports per-clip annotator counts. (2) Low Krippendorff's alpha values (e.g., surprise_neg=0.119, disgust=0.166, anger=0.370 in Table 2) mean some majority-vote ground-truth labels may reflect a single annotator's judgment, so the reported model-human gap may overstate model failure on genuinely ambiguous clips. These are data-reliability issues, not circularity; they do not make the derivation equivalent to its inputs. The only self-citation is reference [9], the authors' own prior interview study, used to design the open-ended cue prompts; it is minor and non-load-bearing, so the score is 2 rather than 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no theoretical free parameters or invented entities. It relies on four domain assumptions: corpus annotation quality, VADER-based clip selection, annotator ground-truth reliability, and label-set coverage. The VADER selection and annotator agreement are the most consequential for interpreting benchmark scores.

assumptions (4)
  • domain assumption ASLLRP utterance segmentation and English captions are accurate enough to define clip boundaries and select emotionally salient content.
    Section 3.1 segments videos into utterances using ASLLRP-provided start and end frames and uses VADER on captions to select clips; errors here propagate into clip content and label distribution.
  • domain assumption VADER sentiment scores on English captions are a valid proxy for selecting emotionally expressive ASL utterances.
    Section 3.1 uses VADER to select the 100 most positive and 100 most negative utterances; the final sentiment distribution is therefore shaped by a text-based tool that the authors themselves describe as limited in emotion understanding.
  • domain assumption The three Deaf ASL signers' ratings, aggregated by majority vote with confidence tie-breaking, are reliable ground truth.
    Sections 3.2 and 3.3. Low Krippendorff's alpha for several labels (surprise_neg 0.119, disgust 0.166, anger 0.370) and the contradictory annotation-coverage statements make this premise fragile.
  • domain assumption The 10 emotion categories plus the 7-point sentiment scale are sufficient to capture emotional content in ASL.
    Section 3.2 builds the label set from Ekman's basic emotions and the circumplex model; this constrains what the dataset can represent and what the benchmark can measure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language." pith.science (2026). https://pith.science/paper/PZISBHZF

@misc{pith2026250517090,
  author       = {Pith},
  title        = {Pith review of: EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZISBHZF}},
  note         = {Machine review of arXiv:2505.17090}
}
read the original abstract

Unlike spoken languages where the use of prosodic features to convey emotion is well studied, indicators of emotion in sign language remain poorly understood, creating communication barriers in critical settings. Sign languages present unique challenges as facial expressions and hand movements simultaneously serve both grammatical and emotional functions. To address this gap, we introduce EmoSign, the first sign video dataset containing sentiment and emotion labels for 200 American Sign Language (ASL) videos. We also collect open-ended descriptions of emotion cues. Annotations were done by 3 Deaf ASL signers with professional interpretation experience. Alongside the annotations, we include baseline models for sentiment and emotion classification. This dataset not only addresses a critical gap in existing sign language research but also establishes a new benchmark for understanding model capabilities in multimodal emotion recognition for sign languages. The dataset is made available at https://huggingface.co/datasets/catfang/emosign.

Figures

Figures reproduced from arXiv: 2505.17090 by the authors.

Figure 1
Figure 1. Duration distribution of the clips in the [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Distribution of emotion categories based on binarized presence across clips. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Annotation Interface [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figures from the paper (16 more)
Figure 5
Figure 5. Figure 5: Jaccard Similarity of the orig￾inal set of Emotion Labels. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Sample frames from a sign video in the EmoSign dataset. The caption is: "If Mary gets [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Distribution of Sentiment Analysis Results (7-class). [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Distribution of Sentiment Analysis Results (3-class). [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Weighted Accuracy of Models on Sentiment Analysis Task (7-class) [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 11
Figure 11. Figure 11: Confusion Matrix of GPT (without caption) output on Sentiment Analysis Task (7-class) [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 13
Figure 13. Figure 13: Confusion Matrix of AffectGPT (without caption) output on Sentiment Analy￾sis Task (7-class) [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 15
Figure 15. Figure 15: Confusion Matrix of Qwen-2.5 (without caption) output on Sentiment Analy￾sis Task (7-class) [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 17
Figure 17. Figure 17: Confusion Matrix of MiniGPT4 (without caption) output on Sentiment Analy￾sis Task (7-class) [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 19
Figure 19. Figure 19: Distribution of Single Expression Emotion Classification Results. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Distribution of emotion categories within the single expression set. Numbers above the bars indicate count [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 22
Figure 22. Figure 22: Weighted Accuracy of Models on Single Expression Emotion Classification Task [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 25
Figure 25. Figure 25: Confusion Matrix of GPT with caption output on Single Expression Emotion Classification Task [PITH_FULL_IMAGE:figures/full_fig_p024_25.png]
Figure 26
Figure 26. Figure 26: Confusion Matrix of AffectGPT (without caption) output on Single Expression Emotion Classification Task [PITH_FULL_IMAGE:figures/full_fig_p024_26.png]
Figure 28
Figure 28. Figure 28: Confusion Matrix of Qwen-2.5 (without caption) output on Single Expression Emotion Classification Task [PITH_FULL_IMAGE:figures/full_fig_p025_28.png]
Figure 30
Figure 30. Figure 30: Confusion Matrix of MiniGPT4 (without caption) output on Single Expression Emotion Classification Task [PITH_FULL_IMAGE:figures/full_fig_p025_30.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 28 canonical work pages

  1. [1]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    arXiv 2024

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022. 9

    arXiv 2022

  4. [4]

    Openface: an open source facial behavior analysis toolkit

    Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: an open source facial behavior analysis toolkit. In2016 IEEE winter conference on applications of computer vision (WACV), pages 1–10. IEEE, 2016

    work page 2016

  5. [5]

    Sign pose-based transformer for word-level sign language recognition

    Matyáš Boháˇcek and Marek Hrúz. Sign pose-based transformer for word-level sign language recognition. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 182–191, 2022

    work page 2022

  6. [6]

    Iemocap: Interactive emotional dyadic motion capture database.Language resources and evaluation, 42:335–359, 2008

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database.Language resources and evaluation, 42:335–359, 2008

    2008

  7. [7]

    Multimodal sentiment analysis with word-level fusion and reinforcement learning

    Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Baltrušaitis, Amir Zadeh, and Louis-Philippe Morency. Multimodal sentiment analysis with word-level fusion and reinforcement learning. InProceedings of the 19th ACM international conference on multimodal interaction, pages 163–171, 2017

    work page 2017

  8. [8]

    Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.arXiv preprint arXiv:2406.11161, 2024

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.arXiv preprint arXiv:2406.11161, 2024

    arXiv 2024

Show all 55 references
  1. [9]

    Perspectives on capturing emotional expressiveness in sign language, 2025

    Phoebe Chua, Cathy Mengying Fang, Yasith Samaradivakara, Pattie Maes, and Suranga Nanayakkara. Perspectives on capturing emotional expressiveness in sign language, 2025. URLhttps://arxiv.org/abs/2505.08072

    2025 arXiv

  2. [10]

    University of California, Berkeley, 2014

    Daniel Thomas Cordaro.Universals and cultural variations in emotional expression. University of California, Berkeley, 2014

    2014

  3. [11]

    Self-report captures 27 distinct categories of emotion bridged by continuous gradients.Proceedings of the national academy of sciences, 114(38): E7900–E7909, 2017

    Alan S Cowen and Dacher Keltner. Self-report captures 27 distinct categories of emotion bridged by continuous gradients.Proceedings of the national academy of sciences, 114(38): E7900–E7909, 2017

    2017

  4. [12]

    Prosody in the comprehension of spoken language: A literature review.Language and speech, 40(2):141–201, 1997

    Anne Cutler, Delphine Dahan, and Wilma Van Donselaar. Prosody in the comprehension of spoken language: A literature review.Language and speech, 40(2):141–201, 1997

    1997

  5. [13]

    How2sign: a large-scale multimodal dataset for continuous american sign language

    Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2sign: a large-scale multimodal dataset for continuous american sign language. InProceedings of the IEEE/CVF conference on computer vision...

    2021

  6. [14]

    An argument for basic emotions.Cognition & emotion, 6(3-4):169–200, 1992

    Paul Ekman. An argument for basic emotions.Cognition & emotion, 6(3-4):169–200, 1992

    1992

  7. [15]

    Facial expressions, emotions, and sign languages.Frontiers in psychology, 4:115, 2013

    Eeva A Elliott and Arthur M Jacobs. Facial expressions, emotions, and sign languages.Frontiers in psychology, 4:115, 2013

    2013

  8. [16]

    Signllm: Sign languages production large language models.arXiv preprint arXiv:2405.10718, 2024

    Sen Fang, Lei Wang, Ce Zheng, Yapeng Tian, and Chen Chen. Signllm: Sign languages production large language models.arXiv preprint arXiv:2405.10718, 2024

    2024 arXiv

  9. [17]

    Perception of emotions in the hand movement quality of finnish sign language.Journal of nonverbal behavior, 28:53–64, 2004

    Jari K Hietanen, Jukka M Leppänen, and Ulla Lehtonen. Perception of emotions in the hand movement quality of finnish sign language.Journal of nonverbal behavior, 28:53–64, 2004

    2004

  10. [18]

    The platonic representation hypothesis.arXiv preprint arXiv:2405.07987, 2024

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis.arXiv preprint arXiv:2405.07987, 2024

    2024 arXiv

  11. [19]

    Vader: A parsimonious rule-based model for sentiment analysis of social media text

    Clayton Hutto and Eric Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. InProceedings of the international AAAI conference on web and social media, volume 8, pages 216–225, 2014

    2014

  12. [20]

    they’re not willing to accommodate deaf patients

    Tyler G James, Kyle A Coady, Jeanne-Marie R Stacciarini, Michael M McKee, David G Phillips, David Maruca, and JeeWon Cheong. “they’re not willing to accommodate deaf patients”: communication experiences of deaf american sign language users in the emergency department.Qualitati...

    2022

  13. [21]

    Dfew: A large-scale database for recognizing dynamic facial expressions in the wild

    Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. InProceedings of the 28th ACM international conference on multimedia, pages 2881–2889, 2020

    2020

  14. [22]

    Ms-asl: A large-scale data set and benchmark for understanding american sign language.arXiv preprint arXiv:1812.01053, 2018

    Hamid Reza Vaezi Joze and Oscar Koller. Ms-asl: A large-scale data set and benchmark for understanding american sign language.arXiv preprint arXiv:1812.01053, 2018

    2018 arXiv

  15. [23]

    Context-aware emotion recognition networks

    Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion recognition networks. InProceedings of the IEEE/CVF international conference on computer vision, pages 10143–10152, 2019

    2019

  16. [24]

    Eald-mllm: Emotion analysis in long-sequential and de-identity videos with multi-modal large language model.arXiv preprint arXiv:2405.00574, 2024

    Deng Li, Xin Liu, Bohao Xing, Baiqiang Xia, Yuan Zong, Bihan Wen, and Heikki Kälviäinen. Eald-mllm: Emotion analysis in long-sequential and de-identity videos with multi-modal large language model.arXiv preprint arXiv:2405.00574, 2024

    2024 arXiv

  17. [25]

    Mimeqa: Towards socially-intelligent nonverbal foundation models.arXiv preprint arXiv:2502.16671, 2025

    Hengzhi Li, Megan Tjandrasuwita, Yi R Fung, Armando Solar-Lezama, and Paul Pu Liang. Mimeqa: Towards socially-intelligent nonverbal foundation models.arXiv preprint arXiv:2502.16671, 2025

    2025

  18. [26]

    Mer 2023: Multi-label learning, modality robustness, and semi- supervised learning

    Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi- supervised learning. InProceedings of the 31st ACM international conference on multimedia, pages 9610...

    2023

  19. [27]

    Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models.arXiv preprint arXiv:2501.16566, 2025

    Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models.arXiv preprint arXiv:2501.16566, 2025

    2025 arXiv

  20. [28]

    Llava-slt: Visual language tuning for sign language translation.arXiv preprint arXiv:2412.16524, 2024

    Han Liang, Chengyu Huang, Yuecheng Xu, Cheng Tang, Weicai Ye, Juze Zhang, Xin Chen, Jingyi Yu, and Lan Xu. Llava-slt: Visual language tuning for sign language translation.arXiv preprint arXiv:2412.16524, 2024

    2024 arXiv

  21. [29]

    Hemm: Holistic evaluation of multimodal foundation models

    Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdi- nov, and Louis-Philippe Morency. Hemm: Holistic evaluation of multimodal foundation models. arXiv preprint arXiv:2407.03418, 2024

    2024 arXiv

  22. [30]

    Exploring the impact of emotional voice integration in sign-to-speech translators for deaf-to-hearing communication.arXiv preprint arXiv:2412.05738, 2024

    Hyunchul Lim, Minghan Gao, Franklin Mingzhe Li, Nam Anh Dang, Ianip Sit, Michelle M Olson, and Cheng Zhang. Exploring the impact of emotional voice integration in sign-to-speech translators for deaf-to-hearing communication.arXiv preprint arXiv:2412.05738, 2024

    2024 arXiv

  23. [31]

    Improved baselines with visual instruction tuning, 2024.URL https://arxiv

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024.URL https://arxiv. org/abs/2310.03744, 3(4):5, 2023

    2024 arXiv

  24. [32]

    Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. InProceedings of the 30th ACM international conference on multim...

    2022

  25. [33]

    emotion2vec: Self-supervised pre-training for speech emotion representation.arXiv preprint arXiv:2312.15185, 2023

    Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation.arXiv preprint arXiv:2312.15185, 2023

    2023 arXiv

  26. [34]

    Morevqa: Exploring modular reasoning models for video question answering

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13235–13245, 2024

    2024

  27. [35]

    New shared & interconnected asl resources: Signstream® 3 software; dai 2 for web access to linguistically annotated video corpora; and a sign bank

    Carol Neidle, Augustine Opoku, Gregory Dimitriadis, and Dimitris Metaxas. New shared & interconnected asl resources: Signstream® 3 software; dai 2 for web access to linguistically annotated video corpora; and a sign bank. In8th Workshop on the Representation and Processing of ...

    2018

  28. [36]

    Asl video corpora & sign bank: Resources available through the american sign language linguistic research project (asllrp)

    Carol Neidle, Augustine Opoku, and Dimitris Metaxas. Asl video corpora & sign bank: Resources available through the american sign language linguistic research project (asllrp). arXiv preprint arXiv:2201.07899, 2022

    2022 arXiv

  29. [37]

    De Gruyter Mouton, 2012

    Roland Pfau, Markus Steinbach, and Bencie Woll.Sign language. De Gruyter Mouton, 2012

    2012

  30. [38]

    Meld: A multimodal multi-party dataset for emotion recognition in conversa- tions.arXiv preprint arXiv:1810.02508, 2018

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversa- tions.arXiv preprint arXiv:1810.02508, 2018

    2018 arXiv

  31. [39]

    Affective prosody in american sign language.Sign Language Studies, 75(1):113–128, 1992

    Judy S Reilly, Marina L McIntire, and Howie Seago. Affective prosody in american sign language.Sign Language Studies, 75(1):113–128, 1992

    1992

  32. [40]

    A circumplex model of affect.Journal of personality and social psychology, 39(6):1161, 1980

    James A Russell. A circumplex model of affect.Journal of personality and social psychology, 39(6):1161, 1980

    1980

  33. [41]

    Open-domain sign language translation learned from online video.arXiv preprint arXiv:2205.12870, 2022

    Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video.arXiv preprint arXiv:2205.12870, 2022

    2022 arXiv

  34. [42]

    Eiji Taira and Shizuka Itagaki. How hearing people understand the deaf and some legal implications of their misinterpretation of visual expressions.International Journal for the Semiotics of Law-Revue internationale de Sémiotique juridique, 32:819–829, 2019

    2019

  35. [43]

    Roberta-lstm: a hybrid model for sentiment analysis with transformer and recurrent neural network.IEEE Access, 10:21517–21525, 2022

    Kian Long Tan, Chin Poo Lee, Kalaiarasi Sonai Muthu Anbananthen, and Kian Ming Lim. Roberta-lstm: a hybrid model for sentiment analysis with transformer and recurrent neural network.IEEE Access, 10:21517–21525, 2022

    2022

  36. [44]

    Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus.Advances in Neural Information Processing Systems, 36:29029–29047, 2023

    Dave Uthus, Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus.Advances in Neural Information Processing Systems, 36:29029–29047, 2023

    2023

  37. [45]

    Gallaudet University Press, 2000

    Clayton Valli and Ceil Lucas.Linguistics of American sign language: An introduction. Gallaudet University Press, 2000

    2000

  38. [46]

    The future of emotion in human- computer interaction

    Greg Wadley, Vassilis Kostakos, Peter Koval, Wally Smith, Sarah Webber, Anna Cox, James J Gross, Kristina Höök, Regan Mandryk, and Petr Slovák. The future of emotion in human- computer interaction. InCHI Conference on human factors in computing systems extended abstracts, page...

    2022

  39. [47]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13204–13214, 2024

    2024

  40. [48]

    Detecting depression severity from vocal prosody.IEEE transactions on affective computing, 4(2):142–150, 2012

    Ying Yang, Catherine Fairbairn, and Jeffrey F Cohn. Detecting depression severity from vocal prosody.IEEE transactions on affective computing, 4(2):142–150, 2012

    2012

  41. [49]

    Asl stem wiki: Dataset and benchmark for interpreting stem articles.arXiv preprint arXiv:2411.05783, 2024

    Kayo Yin, Chinmay Singh, Fyodor O Minakov, Vanessa Milan, Hal Daumé III, Cyril Zhang, Alex X Lu, and Danielle Bragg. Asl stem wiki: Dataset and benchmark for interpreting stem articles.arXiv preprint arXiv:2411.05783, 2024

    2024 arXiv

  42. [50]

    Tensor fusion network for multimodal sentiment analysis.arXiv preprint arXiv:1707.07250, 2017

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis.arXiv preprint arXiv:1707.07250, 2017

    2017 arXiv

  43. [51]

    Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Ling...

    2018

  44. [52]

    filename

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  45. [53]

    **You must choose ONLY ONE emotion**—the one that is most prominent in the clip

  46. [54]

    If multiple emotions seem present, select the strongest one

  47. [55]

    If Mary gets home late, John will probably be upset

    If unsure, make your best judgment—do not leave the response blank. **Penalties for not following instructions:** - If you output more than one emotion, you will fail the task. - If you refuse to answer or leave it blank, you will fail the task. **Output format:** [selected em...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.