Pith. sign in

REVIEW 3 major objections 5 minor 110 references

A new benchmark shows AI models understand mental health in long videos far worse than humans, especially from a first-person perspective.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:26 UTC pith:CON65OQL

load-bearing objection A serious benchmark with a genuinely new first/third-person task distinction and unusually careful construction; the production-label caveat is real but does not sink the central claim. the 3 major comments →

arxiv 2607.27895 v1 pith:CON65OQL submitted 2026-07-30 cs.AI cs.CV

MMHBench: A Multi-Perspective Benchmark for Mental Health Understanding in Long-Form Videos

Jinpeng Hu , Erqiang Wang , Shan Wang , Zhuo Li , Peipei Song , Xun Yang , Meng Wang This is my paper
classification cs.AI cs.CV
keywords mental health understandingmultimodal large language modelslong-form videobenchmarkperspective-takingfirst-person reasoningmulti-agent question generationexpert verification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MMHBench, a benchmark of 268 long-form videos and 2,184 multiple-choice questions that tests whether multimodal AI systems can actually understand mental health in context rather than just classify it. The authors argue that genuine understanding requires reasoning from two complementary perspectives: an external observer interpreting visible behavior, and a first-person perspective that infers latent emotions and intentions. To build the benchmark, they use a multi-agent question-generation pipeline that creates questions from four social roles, filters out items solvable from text alone, and has licensed counselors verify the answers. Across 22 models, the best system reaches 55% accuracy while humans reach 87%, and most models score markedly lower on first-person than third-person questions. The central claim is that perspective-aware psychological reasoning in long videos is a distinct capability that current models have not mastered.

Core claim

On its own terms, the paper's claim is that MMHBench validly measures multimodal mental-health understanding and that current MLLMs fail it. The benchmark contains 1,579 first-person questions, which ask a model to reason from the standpoint of a specific individual in the video, and 605 third-person questions, which ask for objective interpretation of observable behavior. Evaluating 22 representative open- and closed-source multimodal models in a zero-shot setting, the authors find the strongest model achieves 55.31% overall accuracy, far below the 87.44% of human respondents. Most models perform worse on first-person questions, and an error analysis shows the dominant failure is inferring

What carries the argument

The central machinery is the benchmark's perspective-conditioned question design, synthesized by the Multi-Agent Question Generation (MAQG) framework. MAQG uses a perception agent to extract video context and role profiles, then role-playing agents simulate four social roles—the Lived Experience Subject, Mental Health Professional, Family Member, and Others—to generate first-person questions, while a separate agent generates third-person questions. A textual evaluator and a multimodal evaluator iteratively push questions to require video evidence, a leakage filter discards any item that a text-only LLM can answer in four of six randomized trials, and four licensed counselors verify remaining

Load-bearing premise

The benchmark's validity rests on the assumption that each of the 2,184 questions has exactly one correct answer—the uniquely best-supported interpretation of the video—and that the licensed counselors' ratings reliably certify that unambiguity.

What would settle it

Have a second, independent panel of licensed psychologists answer a random sample of 200 MMHBench first-person questions without access to the official gold answers, then measure agreement with the published answers. If agreement falls well below the reported inter-rater reliability of about 0.79 for 'Answer Correctness', the ground truth is not as deterministic as claimed, and model–human gaps could be partly an artifact of option ambiguity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Long-form mental health understanding is far from solved: the best evaluated model leaves a 32-point gap behind human accuracy.
  • First-person perspective-taking is a measurably harder task than third-person observation; most models show a consistent first-person accuracy deficit.
  • The benchmark is not solvable from question priors: question-only accuracy is near the 25% chance level, so correct answers require video-grounded evidence.
  • More visual information helps: increasing frame density, keyframe extraction, and adding video to transcripts all raise accuracy, with the largest gains from the strongest models.
  • When models fail, the dominant error is missing the underlying psychological mechanism, not failing to recognize the surface event or emotion.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the consistent first-person deficit suggests that models trained largely on observed-behavior labels (for example, depression detection) learn to describe behavior but not to model subjective experience; testing this would require a training intervention that explicitly teaches perspective-taking.
  • Editorial inference: because transcripts alone outperform video frames alone, a useful stress test is to feed models only audio (prosody, pauses) plus transcript; if first-person accuracy jumps, the missing evidence is in how things are said rather than what is said.
  • Editorial inference: the benchmark's expert rubric asks for the 'most defensible' answer; a natural extension is to construct a sub-set of deliberately ambiguous questions to measure calibration, not just accuracy, of model confidence.
  • Editorial inference: the MAQG pipeline, with its role-conditioned agents and leakage filtering, could be transferred to other perspective-sensitive domains such as couple therapy, workplace conflict, or testimony analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MMHBench, a multimodal benchmark for mental-health understanding in long-form videos, consisting of 268 videos and 2,184 multiple-choice questions. Questions are split into 605 third-person items and 1,579 first-person items, the latter also stratified by social role (LES, MHP, FM, Others). The benchmark is constructed with a Multi-Agent Question Generation (MAQG) pipeline that uses role-playing agents, multi-role feedback, iterative optimization, a textual and a multimodal evaluator, answer-leakage filtering, and an expert-guided verification stage. The authors evaluate 22 open- and closed-source MLLMs and report that long-form mental-health understanding is challenging, with first-person questions consistently harder than third-person questions, a result they attribute to the need for perspective-conditioned reasoning over latent mental states. Secondary analyses cover frame density, video duration, question length, modality ablations, keyframe extraction, reasoning-quality evaluation with an LLM judge, and an error taxonomy.

Significance. If the benchmark is valid, it fills a real gap: most existing mental-health video datasets are built around coarse diagnostic labels or release only pre-extracted features, whereas MMHBench targets fine-grained, perspective-aware, video-grounded reasoning in long videos. The construction protocol has genuine strengths: a calibration phase for expert raters, a second calibration subset with Krippendorff alpha above 0.75, a six-trial text-only leakage filter, a modality ablation showing near-chance question-only accuracy, and explicit rubrics in the appendices. These are exactly the kinds of quality controls the field needs for LLM-generated benchmarks. However, the central empirical claim — that first-person questions are substantially harder than third-person questions — rests on the correctness of 2,184 gold labels, and the current verification protocol does not establish that those labels are unambiguous or free from single-expert noise. The paper's value therefore depends on whether the production items, not just the calibration subset, meet the reliability standard.

major comments (3)
  1. [§3.5 and Appendix F] The reported inter-rater reliability (Krippendorff's α ≈ 0.77–0.80) is computed only on the 200-item second calibration subset. After calibration, the remaining question-option pairs are partitioned among the four experts and each item is assessed by one expert only; only revised items receive a second review. The paper does not report per-expert item counts, rejection/acceptance rates, or any agreement statistic for the production items. Because the headline FP<TP gap depends on the correctness of all 2,184 labels, unmeasured single-expert judgment error could inflate or even create that gap if first-person items are more ambiguous than third-person items. Please provide independent double annotation on a random sample of production items (ideally stratified by perspective), report agreement on that sample, and present label-noise-adjusted accuracies.
  2. [§3.4–3.5, Appendix B, Appendix D] The ground-truth answers are generated by MAQG and refined by LLM evaluators Agentte (GPT-5) and Agentme (Qwen3-VL-Plus). Expert verification asks whether each generated answer is 'the most plausible and best-supported option' given the video and distractors, but it does not independently establish that the video has a unique correct answer absent the generation pipeline. This creates a low-to-moderate circularity risk: the benchmark may encode the interpretive preferences of the generating/evaluating model families. Using different model families and a leakage filter mitigates but does not eliminate this risk. As a concrete test, have independent clinicians write free-form answers for a random sample of items without seeing the generated options, then measure agreement with the benchmark gold answers; also report how many items experts marked as ambiguous, revised, or discarded during p
  3. [Table 2 and §4.2] The human performance row (87.44%) is evaluated on 300 randomly sampled questions, while all model accuracies are computed on the full set of 2,184 questions. This is not an apples-to-apples comparison: the human number may not be representative of the full benchmark, and the gap between human and model performance is therefore not a rigorous measure of how far models are from human-level understanding. Please evaluate the models on the same 300-item subset used for human raters, or report both full-set and subset accuracies with appropriate confidence intervals.
minor comments (5)
  1. [§4.2] The text says 'Qwen3.5 also demonstrates competitive performance' when referring to model families, but Table 2 shows Qwen3.6-35B-A3B as the best open-source model; please name the exact model variant to avoid ambiguity.
  2. [Appendix D] The retention criterion in items 3 and 4 is strict, but the paper never states how many items were discarded or revised under this rubric. Reporting these numbers would help readers assess the quality bar and the potential selection bias.
  3. [Figure 2] The panel labels 'a', 'b', 'c', 'd' in the caption do not clearly correspond to the subfigures; please align the caption with the actual panel layout.
  4. [Table 2] Some model names appear with inconsistent spacing (e.g., 'MiniCPM-V-4 5'); please standardize naming and add a version table if needed.
  5. [§3.2 and Appendix A] The definition of the 'Others' role is very broad ('friends, classmates, or colleagues'); consider reporting results for finer-grained Others subcategories if the sample sizes permit, because the current grouping may hide meaningful differences.

Circularity Check

0 steps flagged

No significant circularity: MMHBench is an empirical benchmark with independent expert verification and external checks; no derivation reduces to its own inputs.

full rationale

The paper does not advance a formal derivation in which an output equals an input by construction. It constructs a benchmark, verifies it, and reports model accuracy. The main empirical claims—low overall accuracy and lower first-person than third-person accuracy—are observations on held-out test items, not quantities fitted from the same data. The leakage filter in Section 3.5 and the modality ablation in Section 4.4 independently support that items require video grounding, and the expert calibration subset in Appendix F provides inter-rater agreement evidence. Although LLMs are used in the MAQG pipeline and GPT-5.5 is both evaluated and used as a reasoning-quality judge, this is a potential validity/contamination concern, not circularity: the ground-truth answers were generated by an LLM pipeline but reviewed by licensed counselors, and the evaluated models' outputs are not used to define the benchmark labels. Self-citations in the related work are not load-bearing. The single-expert partition after calibration is a reliability limitation, but it does not make the benchmark's findings true by definition. No circular step can be demonstrated from the paper's equations or construction protocol.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The benchmark rests on domain assumptions about video suitability, expert validity, and answer unambiguity; there are no fitted numerical parameters beyond a leakage filter threshold.

free parameters (1)
  • Answer leakage filter threshold = 4 of 6 trials
    Questions removed if text-only LLMs answer correctly in at least 4 of 6 randomized trials; threshold chosen without a principled calibration.
axioms (3)
  • domain assumption Expert rating of question-option pairs is a valid proxy for ground-truth correctness
    Section 3.5, Appendix D: all answer correctness is determined by counselor ratings of LLM-generated items, not by independent clinical annotation.
  • domain assumption Publicly available videos are representative and ethically usable
    Section 3.1: videos from YouTube/Bilibili pass a five-dimensional ethical/topic filter; no verification of representativeness.
  • domain assumption Multi-agent generation with different model families avoids generator bias
    Appendix B: Agentte=GPT-5, Agentme=Qwen3-VL-Plus, other agents=Seed-1.8; this mitigates but does not prove absence of model bias.

pith-pipeline@v1.3.0-daily-deepseek · 22665 in / 7208 out tokens · 67228 ms · 2026-07-31T23:26:24.036573+00:00 · methodology

0 comments
read the original abstract

Mental health understanding in long-form videos requires nuanced reasoning over observable behavior, interpersonal context, and latent psychological states. Existing benchmarks largely reduce this task to coarse-grained classification, providing limited insight into whether models truly understand psychological phenomena or rely on superficial correlations. To address this limitation, we introduce MMHBench, a comprehensive multimodal benchmark for multi-perspective mental health understanding, comprising 268 long-form videos and 2,184 carefully curated questions. MMHBench organizes the evaluation into two complementary settings: (1) third-person assessment, consisting of 605 questions that focus on the interpretation of observable behaviors and multimodal evidence, and (2) first-person perspective-taking, comprising 1,579 questions that require perspective-conditioned reasoning to identify the interpretation of the mental state supported by the available multimodal evidence. We propose a Multi-Agent Question Generation (MAQG) framework that simulates diverse social roles to synthesize questions from multiple perspectives. The generated questions are refined through multi-role feedback and iterative optimization, followed by expert-guided verification to ensure quality and validity. Extensive evaluation of 22 representative multimodal large language models (MLLMs), spanning both open-source and leading closed-source models, demonstrates that long-form video mental health understanding remains highly challenging.

Figures

Figures reproduced from arXiv: 2607.27895 by Erqiang Wang, Jinpeng Hu, Meng Wang, Peipei Song, Shan Wang, Xun Yang, Zhuo Li.

Figure 1
Figure 1. Figure 1: Examples from MMHBench, including both first-person and third-person questions. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An overview of MMHBench. a. The data construction process employs a multi-agent framework, including perception, multi-role feedback integration, iterative optimization, visual grounding filtering, and manual verification. b. Human-in-the-loop prompt refinement and seed pool expansion. c. Distributions of video types and perspective categories. d. Expert verification. tional nature of third-person tasks, t… view at source ↗
Figure 3
Figure 3. Figure 3: Impact of input frame density on model performance. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Modality ablation results on MMHBench. Question [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of reasoning errors across three represen [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Representative failure cases of MLLMs on MMHBench. The categories include role perspective (RP) errors, emotion recognition [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

110 extracted references · 8 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 2

    Pith/arXiv arXiv 2023

  2. [2]

    Ensemble hybrid learning methods for automated depression detection.IEEE transactions on computational social systems, 10(1):211–219, 2022

    Luna Ansari, Shaoxiong Ji, Qian Chen, and Erik Cambria. Ensemble hybrid learning methods for automated depression detection.IEEE transactions on computational social systems, 10(1):211–219, 2022. 2

    2022

  3. [3]

    Infinibench: A benchmark for large multi-modal models in long-form movies and tv shows

    Kirolos Ataallah, Eslam Mohamed Bakr, Mahmoud Ahmed, Chenhui Gou, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. Infinibench: A benchmark for large multi-modal models in long-form movies and tv shows. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19496–19523, 2025. 2

    2025

  4. [4]

    Qwen3-vl technical report, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhao- hai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Jun- yang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shix...

    2025

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 7

    Pith/arXiv arXiv 2025

  6. [6]

    Depression detection in clinical interviews with llm-empowered structural element graph

    Zhuang Chen, Jiawen Deng, Jinfeng Zhou, Jincenzi Wu, Tieyun Qian, and Minlie Huang. Depression detection in clinical interviews with llm-empowered structural element graph. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 8181–8...

    2024

  7. [7]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 2

    Pith/arXiv arXiv 2025

  8. [8]

    Tears or cheers? benchmarking LLMs via culturally elicited distinct affective responses

    Chongyuan Dai, Yaling Shen, Zihan Gao, Jia Li, Yishun Jiang, Yaxiong Wang, Liu Liu, Zongyuan Ge, and Jinpeng Hu. Tears or cheers? benchmarking LLMs via culturally elicited distinct affective responses. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 38171–38196, San Diego, California...

    2026

  9. [9]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24108–24118, 2025. 2, 3

    2025

  10. [10]

    The first mpdd challenge: Multimodal personality-aware depression detection

    Changzeng Fu, Zelin Fu, Qi Zhang, Xinhe Kuang, Jiacheng Dong, Kaifeng Su, Yikai Su, Wenbo Shi, Junfeng Yao, Yu- liang Zhao, et al. The first mpdd challenge: Multimodal personality-aware depression detection. InProceedings of the 33rd ACM International Conference on Multimedia, pages 13924–13929, 2025. 2, 3

    2025

  11. [11]

    The distress analysis interview corpus of human and computer interviews

    Jonathan Gratch, Ron Artstein, Gale M Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, et al. The distress analysis interview corpus of human and computer interviews. InLrec, pages 3123–3128. Reykjavik, 2014. 2, 3

    2014

  12. [12]

    Towards ai-assisted psychotherapy: Emotion-guided genera- tive interventions

    Kilichbek Haydarov, Youssef Mohamed, Emilio Goldenher- sch, Paul OCallaghan, Li-jia Li, and Mohamed Elhoseiny. Towards ai-assisted psychotherapy: Emotion-guided genera- tive interventions. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32724–32743, 2025. 3

    2025

  13. [13]

    Lmvd: A large-scale multimodal vlog dataset for depression detection in the wild.Information Fusion, page 103632, 2025

    Lang He, Kai Chen, Junnan Zhao, Yimeng Wang, Ercheng Pei, Haifeng Chen, Jiewei Jiang, Shiqing Zhang, Jie Zhang, Zhongmin Wang, et al. Lmvd: A large-scale multimodal vlog dataset for depression detection in the wild.Information Fusion, page 103632, 2025. 2, 3

    2025

  14. [14]

    Beyond emotion recognition: A multi-turn multimodal emotion understanding and reasoning benchmark

    Jinpeng Hu, Hongchang Shi, Chongyuan Dai, Zhuo Li, Peipei Song, and Meng Wang. Beyond emotion recognition: A multi-turn multimodal emotion understanding and reasoning benchmark. InProceedings of the 33rd ACM International Conference on Multimedia, pages 5814–5823, 2025. 2

    2025

  15. [15]

    Agentmental: An interactive multi-agent frame- work for explainable and adaptive mental health assessment

    Jinpeng Hu, Ao Wang, Qianqian Xie, Zhuo Li, Hui Ma, and Dan Guo. Agentmental: An interactive multi-agent frame- work for explainable and adaptive mental health assessment. InProceedings of the AAAI Conference on Artificial Intelli- gence, pages 31050–31058, 2026. 2

    2026

  16. [16]

    Smile: Multimodal dataset for understanding laughter in video with language models

    Lee Hyun, Kim Sung-Bin, Seungju Han, Youngjae Yu, and Tae-Hyun Oh. Smile: Multimodal dataset for understanding laughter in video with language models. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 1149–1167, 2024. 2, 3

    2024

  17. [17]

    De- pression detection from social network data using machine learning techniques.Health information science and systems, 6(1):8, 2018

    Md Rafiqul Islam, Muhammad Ashad Kabir, Ashir Ahmed, Abu Raihan M Kamal, Hua Wang, and Anwaar Ulhaq. De- pression detection from social network data using machine learning techniques.Health information science and systems, 6(1):8, 2018. 2

    2018

  18. [18]

    Depression detection on social media with large language models

    Xiaochong Lan, Zhiguang Han, Yiming Cheng, Li Sheng, Jie Feng, Chen Gao, and Yong Li. Depression detection on social media with large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2155–2171, 2025. 2

    2025

  19. [19]

    Sensemood: depression detection on social media

    Chenhao Lin, Pengwei Hu, Hui Su, Shaochun Li, Jing Mei, Jie Zhou, and Henry Leung. Sensemood: depression detection on social media. InProceedings of the 2020 international conference on multimedia retrieval, pages 407–411, 2020. 2

    2020

  20. [20]

    Ministral 3.arXiv preprint arXiv:2601.08584, 2026

    Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sad´e, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3.arXiv preprint arXiv:2601.08584, 2026. 7

    Pith/arXiv arXiv 2026

  21. [21]

    Bolt: Boost large vision-language model without training for long-form video understanding

    Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. Bolt: Boost large vision-language model without training for long-form video understanding. In2025 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 3318–3327, 2025. 9

    2025

  22. [22]

    Psyscam: A benchmark for psychological techniques in real-world scams.arXiv preprint arXiv:2505.15017, 2025

    Shang Ma, Tianyi Ma, Jiahao Liu, Wei Song, Zhenkai Liang, Xusheng Xiao, and Yanfang Ye. Psyscam: A benchmark for psychological techniques in real-world scams.arXiv preprint arXiv:2505.15017, 2025. 3

    arXiv 2025

  23. [23]

    Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 3

    2023

  24. [24]

    Deep learning for depression detection of twitter users

    Ahmed Husseini Orabi, Prasadith Buddhitha, Mahmoud Hus- seini Orabi, and Diana Inkpen. Deep learning for depression detection of twitter users. InProceedings of the fifth workshop on computational linguistics and clinical psychology: from keyboard to clinic, pages 88–97, 2018. 2

    2018

  25. [25]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. InProceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536,

  26. [26]

    Deepwell-adol: A scalable expert-based dialogue corpus for adolescent positive men- tal health and wellbeing promotion

    Wenyu Qiu, Yuxiong Wang, Jiajun Tan, Hanchao Hou, Qinda Liu, Wei Yao, and Shiguang Ni. Deepwell-adol: A scalable expert-based dialogue corpus for adolescent positive men- tal health and wellbeing promotion. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12797–12821, 2025. 3

    2025

  27. [27]

    Qwen3.5: Towards native multimodal agents,

    Qwen Team. Qwen3.5: Towards native multimodal agents,

  28. [28]

    Qwen3.6-35B-A3B: Agentic coding power, now open to all, 2026

    Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all, 2026. 7

    2026

  29. [29]

    Qwen3.6-Plus: Towards real world agents, 2026

    Qwen Team. Qwen3.6-Plus: Towards real world agents, 2026. 7

    2026

  30. [30]

    Qwen3.7: The agent frontier, 2026

    Qwen Team. Qwen3.7: The agent frontier, 2026. 7

    2026

  31. [31]

    M- help: Using social media data to detect mental health help- seeking signals

    MSVPJ Sathvik, Zuhair Hasan Shaik, and Vivek Gupta. M- help: Using social media data to detect mental health help- seeking signals. InFindings of the Association for Computa- tional Linguistics: EMNLP 2025, pages 22510–22520, 2025. 3

    2025

  32. [32]

    Automatic depres- sion detection: An emotional audio-textual corpus and a gru/bilstm-based model

    Ying Shen, Huiyu Yang, and Lin Lin. Automatic depres- sion detection: An emotional audio-textual corpus and a gru/bilstm-based model. InICASSP 2022-2022 IEEE In- ternational Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 6247–6251. IEEE, 2022. 2, 3

    2022

  33. [33]

    Video-mmlu: A massive multi- discipline lecture understanding benchmark.arXiv preprint arXiv:2504.14693, 2025

    Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video-mmlu: A massive multi- discipline lecture understanding benchmark.arXiv preprint arXiv:2504.14693, 2025. 2

    Pith/arXiv arXiv 2025

  34. [34]

    Emotional video captioning with vision-based emo- tion interpretation network.IEEE Transactions on Image Processing, 33:1122–1135, 2024

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. Emotional video captioning with vision-based emo- tion interpretation network.IEEE Transactions on Image Processing, 33:1122–1135, 2024. 2

    2024

  35. [35]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 7

    Pith/arXiv arXiv 2025

  36. [36]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, et al. Lvbench: An extreme long video understanding benchmark. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22958–22967, 2025. 2

    2025

  37. [37]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Informa- tion Processing Systems, 37:28828–28857, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Informa- tion Processing Systems, 37:28828–28857, 2024. 2, 3

    2024

  38. [38]

    Multimodal measurement of depression using deep learning models

    Le Yang, Dongmei Jiang, Xiaohan Xia, Ercheng Pei, Meshia C ´edric Oveneke, and Hichem Sahli. Multimodal measurement of depression using deep learning models. In Proceedings of the 7th annual workshop on audio/visual emo- tion challenge, pages 53–59, 2017. 2

    2017

  39. [39]

    D4: a chinese dialogue dataset for depression-diagnosis-oriented chat.arXiv preprint arXiv:2205.11764, 2022

    Binwei Yao, Chao Shi, Likai Zou, Lingfeng Dai, Mengyue Wu, Lu Chen, Zhen Wang, and Kai Yu. D4: a chinese dialogue dataset for depression-diagnosis-oriented chat.arXiv preprint arXiv:2205.11764, 2022. 2, 3

    Pith/arXiv arXiv 2022

  40. [40]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 7

    Pith/arXiv arXiv 2024

  41. [41]

    D-vlog: Multimodal vlog dataset for depression detec- tion

    Jeewoo Yoon, Chaewon Kang, Seungbae Kim, and Jinyoung Han. D-vlog: Multimodal vlog dataset for depression detec- tion. InProceedings of the AAAI Conference on Artificial Intelligence, pages 12226–12234, 2022. 3

    2022

  42. [42]

    Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe

    Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wen- shuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Ranchi Zhao, et al. Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11704–11715, 2026. 7

    2026

  43. [43]

    Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236–2246, 2018. 2

    2018

  44. [44]

    Mentalglm series: Explainable large lan- guage models for mental health analysis on chinese social media

    Wei Zhai, Nan Bai, Qing Zhao, Jianqiang Li, Fan Wang, Hongzhi Qi, Meng Jiang, Xiaoqin Wang, Bing Xiang Yang, and Guanghui Fu. Mentalglm series: Explainable large lan- guage models for mental health analysis on chinese social media. InProceedings of the 2025 Conference on Empiri- cal Methods in Natural Language Processing, pages 13599– 13614, 2025. 3

    2025

  45. [45]

    Auradial: A large-scale human-centric di- alogue dataset for chinese ai psychological counseling

    Xiantao Zhang. Auradial: A large-scale human-centric di- alogue dataset for chinese ai psychological counseling. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2847–2863, 2025. 3

    2025

  46. [46]

    M3ed: Multi-modal multi-scene multi-label emotional dialogue database

    Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou Li. M3ed: Multi-modal multi-scene multi-label emotional dialogue database. InPro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5699–5710, 2022. 2

    2022

  47. [47]

    Mlvu: Benchmarking multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 13691–13701, 2025. 2, 3

    2025

  48. [48]

    Bochao Zou, Jiali Han, Yingxue Wang, Rui Liu, Shenghui Zhao, Lei Feng, Xiangwen Lyu, and Huimin Ma. Semi- structural interview-based chinese multimodal depression cor- pus towards automatic preliminary screening of depressive disorders.IEEE Transactions on Affective Computing, 14(4): 2823–2838, 2022. 2 A. Details of Roles Detailed descriptions of these ro...

    2022

  49. [49]

    The reviewer first checks the source information and metadata of each candidate video, including its accessibility, duration, and source reliability

  50. [50]

    The reviewer watches the complete video to understand its topic, narrative context, psychological content, and potential ethical risks

  51. [51]

    Each dimension is assessed using a binary score, where 1 indicates that the eligibility requirement is satisfied and 0 indicates that it is not satisfied

    The reviewer independently evaluates each candidate video along the following five dimensions. Each dimension is assessed using a binary score, where 1 indicates that the eligibility requirement is satisfied and 0 indicates that it is not satisfied. (a)Topic Relevance. Does the video explicitly and substantively involve mental-health-related issues, such ...

  52. [52]

    Mental-health-related content is absent, merely peripheral, only briefly mentioned, or insufficiently developed to constitute a meaningful topic of the video

  53. [53]

    (b)Psychological Process Evidence

    Mental health or psychological difficulty constitutes a central and substantive topic of the video. (b)Psychological Process Evidence. Does the video contain sufficiently observable evidence of psychological processes, such as emotional changes, coping behavior, help-seeking, interpersonal conflict, counseling, medication, or self-regulation?

  54. [54]

    Observable psychological-process evidence is absent, insufficient, or too ambiguous to support reliable question generation

  55. [55]

    (c)Narrative Completeness

    The video contains sufficiently clear and observable emotional, behavioral, interpersonal, or psychological- process evidence to support meaningful question generation. (c)Narrative Completeness. Does the video provide sufficient contextual information and a coherent narrative structure for understanding the relevant events, character relationships, and p...

  56. [56]

    The video is incomplete, highly fragmented, or too ambiguous to support reliable interpretation and question generation

  57. [57]

    (d)Video Duration

    The video provides sufficient context and a coherent narrative for understanding the relevant events and psychological processes. (d)Video Duration. Does the duration of the candidate video fall within the predefined range of 8-60 minutes?

  58. [58]

    The video is shorter than 8 minutes or longer than 60 minutes

  59. [59]

    (e)Ethical Appropriateness

    The video duration falls within the required range of 8-60 minutes. (e)Ethical Appropriateness. Does the video satisfy the ethical, privacy, and source-verification requirements for benchmark construction? The reviewer checks whether the video: • is publicly accessible from a verifiable source; • contains no unresolved exposure of sensitive personal infor...

  60. [60]

    One or more ethical, privacy, or source-verification requirements are violated or remain unresolved

  61. [61]

    All ethical, privacy, and source-verification requirements are satisfied

  62. [62]

    Each evaluation dimension constitutes an independent mandatory eligibility requirement, and satisfactory performance on one dimension cannot compensate for failure on another

    We adopt a strict non-compensatory filtering criterion. Each evaluation dimension constitutes an independent mandatory eligibility requirement, and satisfactory performance on one dimension cannot compensate for failure on another

  63. [63]

    A score of 0 on any dimension results in the immediate exclusion of the candidate video

    A candidate video is retained only if it receives a score of 1 on all five evaluation dimensions. A score of 0 on any dimension results in the immediate exclusion of the candidate video. D. Expert Verification Rubric

  64. [64]

    A question-option pair is directly discarded if it receives a score of 1 on any evaluation dimension

  65. [65]

    For Dimensions (c), (d), and (e), an item receiving a score of 2 is revised by the annotator whenever the identified issue is amenable to modification

  66. [66]

    An unrevised item is retained only if it receives a score of 2 on both Dimensions (a) and (b), and a score of 3 on all of Dimensions (c), (d), and (e)

  67. [67]

    It is retained only if, after revision, it satisfies the same retention criterion: a score of 2 on Dimensions (a) and (b), and a score of 3 on Dimensions (c), (d), and (e)

    Each revised item is independently reviewed by another annotator. It is retained only if, after revision, it satisfies the same retention criterion: a score of 2 on Dimensions (a) and (b), and a score of 3 on Dimensions (c), (d), and (e). Otherwise, it is discarded

  68. [68]

    Are the question and its answer options consistent with the narrative perspective and character roles required by the item?

    Each question-option pair is evaluated along the following five dimensions: (a)Perspective Consistency. Are the question and its answer options consistent with the narrative perspective and character roles required by the item?

  69. [69]

    For a first-person item, the question is not posed from the questioner’s perspective, or the answer options do not represent responses from the intended respondent’s perspective

    The perspective is incorrect or internally inconsistent. For a first-person item, the question is not posed from the questioner’s perspective, or the answer options do not represent responses from the intended respondent’s perspective

  70. [70]

    For a first-person item, the question is posed from the questioner’s perspective, and the answer options are expressed from the intended respondent’s perspective

    The question and all answer options consistently follow the intended perspective. For a first-person item, the question is posed from the questioner’s perspective, and the answer options are expressed from the intended respondent’s perspective. (b)Psychological Value. Is the question suitable for evaluating reasonable psychological reasoning grounded in t...

  71. [71]

    The question is overly simple, lacks meaningful evaluation value, is unrelated to the relevant psychological process, or requires unsupported speculation beyond the available video evidence

  72. [72]

    The question is meaningful, relevant to the video, and effectively evaluates reasonable psychological reasoning grounded in observable events, behaviors, dialogue, or contextual evidence. (c) Answer Correctness.Does the provided reference answer represent the most plausible and best-supported option given the question, the available video evidence, and th...

  73. [73]

    The reference answer is contradicted by the video, unsupported by the available evidence, or less plausible than at least one distractor

  74. [74]

    The reference answer is generally plausible, but contains a minor inaccuracy, retains some ambiguity, or is not clearly better supported than all distractors

  75. [75]

    The reference answer is clearly the most plausible and best-supported option, is unambiguous relative to the distractors, and introduces no unsupported diagnostic, causal, or psychological claims. (d) Rationale Validity.Does the accompanying analysis reasonably explain the question and answer options based on factual and behavioral evidence presented in t...

  76. [76]

    The analysis contradicts the video, incorrectly identifies the key emotion or key event, or provides an incorrect explanation of the gold answer or answer options

  77. [77]

    The analysis is generally reasonable but contains minor unverified interpretations, unnecessary psychological reasoning, or incomplete differentiation among the answer options

  78. [78]

    (e)Video Necessity

    The analysis accurately uses relevant events, behaviors, dialogue, or contextual evidence from the video to identify the key emotion and key event, correctly explain the gold answer, and appropriately analyze each answer option, without introducing unsupported psychological speculation. (e)Video Necessity. Is watching the video necessary to answer the que...

  79. [79]

    The question can be answered correctly without watching the video, based primarily on textual cues, answer- option patterns, or general commonsense

  80. [80]

    For example, the question may reveal relevant information, or one option may be excessively absolute or otherwise easily identifiable

    The video provides useful information, but the correct answer can still be substantially narrowed down based solely on the question text and answer options. For example, the question may reveal relevant information, or one option may be excessively absolute or otherwise easily identifiable

Showing first 80 references.