Pith. sign in

REVIEW 3 major objections 5 minor 37 references

Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Chain-of-thought reasoning for social audio-visual question answering is expensive and ineffective: a plain direct-answer fine-tuned baseline matches or beats reasoning models, and a text caption alone performs as well as the full video…

desk verdict A genuinely useful benchmark audit and a convincing cheap-baseline result, but the caption-parity claim hinges on a training-condition ambiguity in Table 5 that the authors need to resolve before it is taken as established. read the letter →

arxiv 2608.13239 v1 pith:36IJ4FXK submitted 2026-08-13 cs.CV

Koen P. de Vries , Xavier Alameda-Pineda , Estefanía Talavera , Stéphane Lathuilière This is my paper · ORCID
classification cs.CV
keywords socialaudio-visualquestionansweringchain-of-thoughtreasoningmultimodallargelanguagemodelssupervisedfine-tuningbenchmarkcurationIntentBench-Primetext-onlypriorscaption-basedvideoQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the field's emphasis on chain-of-thought reasoning for social audio-visual question answering is misplaced. A minimal baseline, Vanilla SFT—standard supervised fine-tuning that directly outputs the answer with no reasoning trace—matches or outperforms existing reasoning-based systems on IntentBench-Prime, WorldSense, and Daily-Omni, while training and inference are far cheaper. The paper also shows that the IntentBench benchmark itself is noisy: about 7% of its questions are broken and about 23% can be answered from text alone; the cleaned re-release, IntentBench-Prime, removes both classes. Finally, replacing the video and audio with a generic, question-independent caption yields accuracy essentially equal to the full multimodal input, and a question-only model still captures most of the gain. The practical upshot is that direct-answer SFT should be the mandatory baseline for any new method in this area, and that current omni-models are not yet extracting much question-specific social information from video.

What carries the argument

The central instrument is the Vanilla SFT baseline: supervised fine-tuning (with LoRA on the language module) of Qwen2.5-Omni so that, given the video, audio, and question, it emits only the answer letter(s), with no chain-of-thought trace. It serves as the minimal control against which all CoT methods are judged. Around it sit two controlled ablations trained in the same way but with reduced input: Question SFT, which sees only the question and answer options, and Caption SFT, which sees a generic question-independent video caption generated by ASID-Captioner instead of the audiovisual stream. Comparing the three isolates three contributions: the benchmark's learnable language priors, the information contained in a condensed textual scene description, and whatever extra the raw video and audio provide. This same three-way design produces the paper's deployment proposal: precompute captions once per video, then answer many queries from text at low latency.

What would settle it

Retrain the video-based Vanilla SFT model on exactly the Social-IQ 2.0 + EMER subset used for Caption SFT and Question SFT, with identical hyperparameters and training steps, and compare its IntentBench-Prime (Hard) accuracy to Caption SFT's 68.6%. If the video model beats the caption model by a clear margin (say, more than 3 points) under this matched-data condition, then the paper's claim that a generic caption is on par with the full multimodal input would be refuted; if the gap stays within one point, the claim holds. The retrained model should also be re-evaluated on WorldSense and Daily-Omni to confirm that the restricted data, not the video modality, is what drives the earlier baseline gains.

Watch

Extended reading notes

Core claim

The paper establishes three findings. Finding 1: IntentBench, the standard benchmark, is contaminated—four LLMs unanimously answer 22.9% of its Social-IQ 2.0 questions from the question and answer text alone, and manual ranking removes 7.1% of questions as broken; the released IntentBench-Prime excludes both, retaining a Hard split of 1,899 questions. Finding 2: Vanilla SFT, a LoRA fine-tune of Qwen2.5-Omni trained on the same data and protocol as HumanOmniV2 but with direct answers, achieves 70.4% accuracy on IntentBench-Prime Hard, 65.2% on Daily-Omni, and 48.8% on WorldSense, matching or beating HumanOmniV2, AVATAR, and AffectOmni in most columns, while using roughly 11x less training compute and 356x less decoding time. Finding 3: the modality ablations show that a question-only SFT model reaches 62.0% on IntentBench-Prime Hard, and a caption-only SFT model reaches 68.6%—within a point of the 69.5% video-based Vanilla SFT—so language priors and a generic caption cover nearly all of the benchmark performance. The authors argue that reasoning traces are therefore not the bottleneck, that new methods must be measured against direct-answer SFT, and that the bottleneck is the model's ability to extract question-specific information from the video and audio.

Load-bearing premise

The caption-vs-video comparison assumes that the video-based Vanilla SFT model in that experiment was trained on the same restricted dataset (Social-IQ 2.0 plus EMER) as the caption and question models; if it was instead the main Section 4 model trained on the full 20K-video set, the apparent parity is confounded by training-data size.

Editorial extensions

If this is right

  • New training methods for social AV-QA should be reported together with a direct-answer SFT baseline trained on the same data; otherwise accuracy gains cannot be attributed to the method rather than to plain fine-tuning.
  • Researchers can re-score their existing IntentBench results against IntentBench-Prime by applying the released exclusion list, avoiding new evaluation runs and making the cleaned benchmark immediately usable.
  • Benchmark builders should screen questions for text-only answerability and broken options before release; the paper's 22.9% text-answerable and 7.1% broken rates show how large the contamination can be.
  • A two-stage caption-then-answer architecture becomes a practical alternative: the caption is computed once per video, so query-time latency drops dramatically and one caption can serve many questions.
  • The results imply that progress in social AV-QA will come less from better reasoning schedules and more from models that can pull question-specific cues out of the video and audio streams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the caption-parity result holds beyond this exact setup, it suggests the bottleneck is the interface between the video encoder and the language model: the visual stream is currently being reduced to scene-level features that carry little question-relevant social detail. A direct test would be to feed question-conditioned or socially annotated captions (gaze, gesture, prosody) and see whether ca
  • The 22.9% text-answerability estimate is conservative because it requires four LLMs to agree unanimously; a single strong LLM would likely flag a higher fraction as text-answerable, which would further shrink the 'Hard' split and raise the bar for what counts as a multimodal question.
  • Because ASID-Captioner and the QA models share the same base model, part of the caption advantage may come from shared tokenization and format alignment rather than from caption content; swapping in a captioner from a different model family would isolate that effect.
  • The paper measures accuracy and latency, but not the explanatory value of the reasoning trace; a complementary study would ask whether the 356x decode-time saving is worth giving up the human-inspectable rationales that CoT models produce.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper audits the IntentBench benchmark, introduces a curated version (IntentBench-Prime), and reports three findings: (1) IntentBench is noisy, with about 7% broken questions and about 23% of questions answerable from text alone; (2) a simple direct-answer supervised fine-tuning baseline (Vanilla SFT) matches or outperforms current chain-of-thought/GRPO-based reasoning methods on IntentBench-Prime, WorldSense, and Daily-Omni at a fraction of the training and inference cost; and (3) training a model to answer from a question-independent textual caption yields accuracy comparable to training on the full video+audio input, suggesting that current omni-models extract limited additional question-relevant information from the video and audio streams. The authors release the IntentBench-Prime exclusion list, the Vanilla SFT model, and code.

Significance. If the findings hold, they are practically important for the audio-visual question answering community: they challenge the necessity of expensive CoT/GRPO training for social AV-QA, advocate for a mandatory direct-SFT baseline, and expose a surprising ceiling on the use of video input by current MLLMs. The paper's strengths include a controlled setup for Finding 2 (same base model, same training data, conservative use of the original published numbers for HumanOmniV2), concrete efficiency measurements, and the practical design of the curated benchmark as an exclusion list that allows re-scoring of existing results. The main risk to the paper's central claims is the training-data ambiguity in the Table 5 comparison, which is load-bearing for Finding 3.

major comments (3)
  1. [Section 5, Table 5] The caption-parity finding depends on whether the Vanilla SFT row in Table 5 was trained on the same restricted Social-IQ 2.0 + EMER data as Caption SFT and Question SFT. Section 5 says 'We limit the training data to Social-IQ 2.0 and EMER' and 'Apart from the modality setup, training is identical between the three variants,' but it also says the baselines are 'compared with the Vanilla SFT method from Section 4,' where Vanilla SFT is trained on the full 20K-video/10K-image mixture. The reported Table 5 value (69.5) differs from both Section 4 values (70.4 LoRA, 69.0 full FT), so the row cannot be unambiguously identified. If the Table 5 Vanilla SFT is the Section 4 model, the comparison is confounded by training-data volume and is not a clean modality ablation. Please state explicitly which training set was used for each row in Table 5, and if Vanilla SFT was retrained on the restricted data, report that run's training-data composition and hyperparameters.
  2. [Section 3.2, broken-question curation] The headline '7.1% broken questions' in Finding 1 is presented without the caveat that it is a lower bound. The curation procedure manually verifies questions in ranked order and stops after 300 consecutive non-broken questions, having verified 'more than half' of the S-IQ questions. The unverified remainder could still contain broken questions, so the 7.1% figure is not a measured rate on the full benchmark. Please report the number of questions actually manually inspected and explicitly describe the 7.1% statistic as a lower bound, or provide an estimate that accounts for the stopping rule.
  3. [Section 5, Table 5 and 'on par' claim] Finding 3 is stated as 'a generic textual caption yields QA performance on par with processing the full multimodal data,' but the table-level support is a 0.9-point difference (Caption SFT 68.6 vs Vanilla SFT 69.5) on a single benchmark split, with no confidence intervals, significance test, or repeated-seed variation. Category-level differences are larger (e.g., How: 64.1 vs 67.0; Why: 68.2 vs 68.6), so the aggregate parity could be consistent with noise. Please provide variance estimates, multiple seeds, or a statistical test to support the parity claim, or soften the claim accordingly.
minor comments (5)
  1. [Abstract and throughout] The name of the curated benchmark is written inconsistently as 'Intentbench-Prime' in the abstract and 'IntentBench-Prime' in the body; please unify the capitalization.
  2. [Figure 1 and surrounding text] The block containing 'FINDING 1' and Figure 1 appears to be duplicated in the introduction; one copy should be removed for clarity.
  3. [Section 4.2, Table 3] The claim that 'there is no reasoning method with a comparable base-model and size that outperforms our Vanilla SFT baseline' should be made precise by listing the base model and parameter count of AVATAR and AffectOmni; as written, the reader cannot verify comparability.
  4. [Section 3.1, Table 1] The finding that the option-swapping makes questions easier for LLMs would be more convincing if the analysis reported per-model text-only accuracies rather than only the unanimous-agreement rate, since the latter is a strict criterion and may compress the effect.
  5. [Section 5] The sentence 'We limit the training data to Social-IQ 2.0 and EMER to reduce the cost of the experiment' should specify which of the compared models the restriction applies to; this is the same ambiguity raised in the first major comment and should be resolved in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: all three findings are empirical comparisons against external benchmarks, with no fitted parameter or self-referential derivation chain.

full rationale

The paper's three findings are empirical results, not derivations. Finding 1 (IntentBench is noisy) is a data-quality audit: broken questions are identified by manual inspection, programmatic checks, and an LLM ensemble, and the text-answerability rate is measured by running four pretrained LLMs on question-answer pairs. Finding 2 (Vanilla SFT matches or outperforms reasoning methods) is an experimental comparison: a simple SFT model is trained on the same data as HumanOmniV2 and evaluated on IntentBench-Prime, WorldSense, and Daily-Omni, which are external benchmarks whose scores are not inputs to the training procedure. Finding 3 (caption-based answering is on par with video-based answering) is a modality ablation: Caption SFT is trained on question-independent ASID captions and compared to Vanilla SFT, with both training runs stated to be identical except for modality. The possible ambiguity about whether the Table 5 Vanilla SFT row was retrained on the restricted Social-IQ 2.0 + EMER set is an experimental-conditions concern, not a circularity concern; the paper explicitly says 'We limit the training data to Social-IQ 2.0 and EMER' and 'Apart from the modality setup, training is identical between the three variants,' which reads as applying to all three finetuned models. No fitted parameter is renamed as a prediction, no load-bearing claim is justified solely by a self-citation, and no equation reduces to an input by construction. The ASID-Captioner sharing a base model with the evaluated video model is a legitimate tool choice for producing text, not a circular argument. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

Free parameters are limited to the hand-chosen curation thresholds, which directly determine the composition of IntentBench-Prime. No mathematical constants are fitted. The central empirical claims are governed by domain assumptions about what constitutes a fair benchmark and a faithful caption.

free parameters (2)
  • text-answerability threshold = unanimous agreement of 4 LLMs
    Hand-chosen strict criterion for removing questions answerable without video. Affects the size and remaining priors of IntentBench-Prime; loosening to 3-of-4 would remove more questions.
  • manual curation stopping rule = 300 consecutive non-broken questions
    Hand-chosen stopping criterion; manual verification covers just over half of the S-IQ questions, so some broken questions may remain in the released benchmark.
assumptions (5)
  • domain assumption Multiple-choice accuracy on the curated benchmark measures social audio-visual understanding.
    The paper uses accuracy or F1 on IntentBench-Prime, WorldSense, and Daily-Omni as the dependent variable for all conclusions.
  • ad hoc to paper Unanimous agreement among Gemma-2-9B, Llama-3.1-8B, Mistral-7B, and Qwen3.5-9B identifies questions trivially answerable from text alone.
    This operationalization defines the 23% text-answerable estimate and the Hard split; it is specific to this paper's curation pipeline.
  • domain assumption ASID-Captioner captions are question-independent and faithfully represent the video and audio content.
    The caption-vs-video comparison in Section 5 treats these captions as a sufficient textual proxy for the multimodal input.
  • domain assumption Vanilla SFT's training data matches HumanOmniV2's training data.
    Section 4.1 says the authors aim to mirror and follow the HumanOmniV2 code-base; the exact match is not independently verified.
  • domain assumption Numbers taken from prior papers (HumanOmniV2, AVATAR, AffectOmni) are comparable to this paper's evaluation pipeline.
    Section 4.2 uses original reported numbers, noting small reproduction differences due to prompting and frame settings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?." pith.science (2026). https://pith.science/paper/36IJ4FXK

@misc{pith2026260813239,
  author       = {Pith},
  title        = {Pith review of: Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36IJ4FXK}},
  note         = {Machine review of arXiv:2608.13239}
}
abstract

Training Multimodal Large Language Models for audio-visual social understanding is a crucial step toward embodied social intelligence. Chain-of-thought (CoT) reasoning has become the dominant approach, with HumanOmniV2 and its IntentBench benchmark as a prominent reference point. In this context, we report three findings. First, IntentBench is highly noisy: $\sim$7% of questions are broken and $\sim$23% are trivially answerable without the video input. We remove the affected questions and release Intentbench-Prime. Second, current reasoning approaches are expensive and surprisingly ineffective. A simple Vanilla SFT baseline matches or outperforms existing reasoning methods across three benchmarks at a fraction of the cost, establishing it as an essential baseline for evaluating novel fine-tuning techniques. Third, our analysis reveals that substantial priors can be learned solely from the text modality and that using a textual caption instead of the video yields performance on par with Vanilla SFT. These surprising findings reveal the limitations of current MLLMs when it comes to social understanding. IntentBench-Prime, Vanilla SFT model, and code are publicly available.

Figures

Figures reproduced from arXiv: 2608.13239 by the authors.

Figure 1
Figure 1. Social Audio-Visual Question Answering (AV-QA): Our three key findings [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Curation process of IntentBench-Prime and benchmark statistics. The two very small categories ‘when’ (14) and ‘who/which’ (25) are merged into ‘other’ in both IntentBench-Prime variants. 616 questions are flagged as answerable from text alone; 18 of them were already removed as broken. 87% of the questions. Examples of issues are displayed in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Examples from Social-IQ 2.0, EMER and MDPE [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Examples of issues in IntentBench. More examples across all types of issues are provided in the supplementary material, Appendix A [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Additional examples of Social-IQ 2.0 questions used in IntentBench that are ambiguous or otherwise broken. It is also clear that sometimes distractor options have issues, but this is never grounds for removal of the question [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Additional examples of Social-IQ 2.0 questions used in IntentBench where the swapping of options made the correct answer ambiguous. Note that, at first sight, some of these questions might not seem broken, but the fact that the swapped-in options are GT answers to othe…
Figure 7
Figure 7. Figure 7: Additional examples of Social-IQ 2.0 questions used in IntentBench. Some￾times due to the swapping of distractor options, the ground truth answer in these questions is easily identifiable from text alone [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 22 canonical work pages

  1. [1]

    In: Proceedings of the 33rd ACM International Confer- ence on Multimedia

    Cai, C., Liang, S., Liu, X., Zhu, K., Wen, Z., Tao, J., Xie, H., Cui, J., Ma, Y., Cheng, Z., et al.: Mdpe: A multimodal deception dataset with personality and emotional characteristics. In: Proceedings of the 33rd ACM International Confer- ence on Multimedia. pp. 12957–12964 (2025)

    work page 2025

  2. [2]

    In: Belgrave, D., Zhang, C., Lin, H., Pascanu, R., Koniusz, P., Ghassemi, M., Chen, N

    Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., Wu, J., Zhang, X., Wang, B., Yue, X.: Video-r1: Reinforcing video reasoning in mllms. In: Belgrave, D., Zhang, C., Lin, H., Pascanu, R., Koniusz, P., Ghassemi, M., Chen, N. (eds.) Advances in Neural Information Processing Systems. vol. 38, pp. 99114–99137. Curran Associates, Inc. (2025),https://proc...

    work page 2025

  3. [3]

    arXiv preprint arXiv:2408.00118 (2024)

    Gemma Team, Riviere, M., Pathak, S., Andreev, A., et al.: Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024)

    arXiv 2024

  4. [4]

    Nature645(8081), 633–638 (2025).https://doi.org/10.1038/s41586-025-09422-z,http://dx

    Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z.F., Gou, Z., et al.: Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature645(8081), 633–638 (2025).https://doi.org/10.1038/s41586-025-09422-z,http://dx. doi.org/10.1038/s41586-025-09422-z

    doi:10.1038/s41586-025-09422-z 2025

  5. [5]

    pre- ferring shorter thinking chains for improved llm reasoning

    Hassid, M., Synnaeve, G., Adi, Y., Schwartz, R.: Don’t overthink it. pre- ferring shorter thinking chains for improved llm reasoning. arXiv preprint arXiv:2505.17813 (2025)

    arXiv 2025

  6. [6]

    In: The Fourteenth Inter- national Conference on Learning Representations (2026),https://openreview

    Hong, J., Yan, S., Cai, J., Jiang, X., Hu, Y., Xie, W.: Worldsense: Evaluating real- world omnimodal understanding for multimodal LLMs. In: The Fourteenth Inter- national Conference on Learning Representations (2026),https://openreview. net/forum?id=YxsfxAvJv4

    work page 2026

  7. [7]

    IBM Research: Granite 4.1 language models.https://www.ibm.com/granite/ docs/models/granite4-1(2026)

    work page 2026

  8. [8]

    In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)

    Kulkarni, Y., Fazli, P.: Avatar: Reinforcement learning to see, hear, and reason over video. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)

    work page 2026

Show all 37 references
  1. [9]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Li, H., Fei, H., Hu, Z., Yang, Z., Wang, Z.: Vegas: Towards visually explainable and grounded artificial social intelligence. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 4707–4715 (2025)

    2025

  2. [10]

    Transactions on Machine Learning Re- search (TMLR) (2026) 16 K

    Li, X., Deng, S., Lai, B., Pian, W., Rehg, J.M., Tian, Y.: Towards online multi- modal social interaction understanding. Transactions on Machine Learning Re- search (TMLR) (2026) 16 K. P. de Vries et al

    2026

  3. [11]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2026), https://openreview.net/forum?id=SSF4qgsNYE

    LI, Y., Ma, Y., Zhang, G., Yuan, R., Zhu, K., Guo, H., Liang, Y., Liu, J., Wang, Z.M.,Yang,J.,Wu,S.,Qu,X.,Shi,J.,Zhang,X.,Yang,Z.,WEN,Y.,Wang,Y.,Li, S., Zhang, Z., Liu, R., Benetos, E., Huang, W., Lin, C.: Omnibench: Towards the future of universal omni-language models. In: Th...

    2026

  4. [12]

    Li, Y., Zhang, H., Guo, M.H., Gao, W., Jia, S., Jiao, S., Hou, Q., Cheng, M.M.: Towards universal video mllms with attribute-structured and quality-verified in- structions (2026),https://arxiv.org/abs/2602.13013

    2026

  5. [13]

    arXiv preprint arXiv:2306.15401 (2023)

    Lian, Z., Sun, H., Sun, L., Gu, H., Wen, Z., Zhang, S., Chen, S., Xu, M., Xu, K., Chen, K., et al.: Explainable multimodal emotion recognition. arXiv preprint arXiv:2306.15401 (2023)

    2023 arXiv

  6. [14]

    Advances in Neural Information Processing Systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in Neural Information Processing Systems36, 34892–34916 (2023)

    2023

  7. [15]

    arXiv preprint arXiv:2407.21783 (2024),https://ai.meta.com/blog/meta-llama-3-1/

    Llama Team, Meta: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024),https://ai.meta.com/blog/meta-llama-3-1/

    2024 arXiv

  8. [16]

    International Conference on Learning Representations (ICLR) (2026)

    Luo, M., Li, B., Xu, S., Zhang, S., Chen, Q., Han, M., Chen, W., Huang, Y., Fei, H., Lee, M.L., et al.: Unveiling the cognitive compass: Theory-of-mind-guided mul- timodal emotion reasoning. International Conference on Learning Representations (ICLR) (2026)

    2026

  9. [17]

    arXiv preprint arXiv:2606.12018 (2026)

    Ma, S., Dang, J., Zhang, W., Zhang, Y., Wang, B., Peng, H., Hu, B., Tian, Q., Chua, T.S.: Modf-sir: A multi-agent omni-modal distilled framework for social intelligence reasoning. arXiv preprint arXiv:2606.12018 (2026)

    2026 arXiv

  10. [18]

    In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    Mathur, L., Qian, M., Liang, P.P., Morency, L.P.: Social genome: Grounded social reasoning abilities of multimodal models. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 24879–24902 (2025)

    2025

  11. [19]

    Mistral AI Team: Mistral-7b-instruct-v0.3 open-weight model.https : / / huggingface.co/mistralai/Mistral- 7B- Instruct- v0.3(2024), official Model Repository and Specification

    2024

  12. [20]

    OpenAI: Introducing ChatGPT.https://openai.com/blog/chatgpt(2022)

    2022

  13. [21]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Qin, Z., Zheng, R., Wang, Y., Li, T., Yuan, Y., Chen, J., Wang, L.: Humansense: From multimodal perception to empathetic context-aware responses through rea- soning mllms. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 24973–24981 (2026)

    2026

  14. [22]

    arXiv preprint arXiv:2604.15804 (2026)

    Qwen Team: Qwen3.5 technical report: Towards native multimodal agents. arXiv preprint arXiv:2604.15804 (2026)

    2026 arXiv

  15. [23]

    Proceedings of the AAAI Conference on Arti- ficial Intelligence40(3), 2029–2037 (Mar 2026).https://doi.org/10.1609/aaai

    Rha, H., Yeo, J.H., Kim, Y., Ro, Y.M.: Emotion-coherent reasoning for multimodal llms via emotional rationale verifier. Proceedings of the AAAI Conference on Arti- ficial Intelligence40(3), 2029–2037 (Mar 2026).https://doi.org/10.1609/aaai. v40i3.37184,https://ojs.aaai.org/ind...

    2026 doi

  16. [24]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., Wu, Y., Guo, D.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models (2024),https://arxiv.org/abs/2402.03300

    2024 arXiv

  17. [25]

    IEEE Transactions on Affective Computing pp

    Wang, Y., Yang, R., Dang, J., Wang, B., Wu, Y., Cao, P., Zhang, W., Peng, H., Hu, B., Chua, T.S.: Affectomni: Rl-verifiable people-centric grounded affective reason- ing for social and art-related scenes. IEEE Transactions on Affective Computing pp. 1–12 (01 2026).https://doi....

    2026

  18. [26]

    NIPS ’22, Curran Associates Inc., Red Hook, NY, USA (2022) Reasoning for Social AV-QA: Where Do We Stand? 17

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models.In:Proceedingsofthe36thInternationalConferenceonNeuralInformation Processing Systems. NIPS ’22, Curran Associ...

    2022

  19. [27]

    Wilf, A., Mathur, L., Mathew, S., Ko, C., Kebe, Y., Liang, P.P., Morency, L.P.: Social-iq 2.0 challenge: Benchmarking multimodal social understanding.https: //github.com/abwilf/Social-IQ-2.0-Challenge(2023)

    2023

  20. [28]

    In: The Fourteenth International Conference on Learning Representations (2026),https://openreview.net/forum?id=KttCXdjj4w

    Xiao, T., Xu, X., Huang, Z., Gao, H., Liu, Q., Liu, Q., Chen, E.: Perception- r1: Advancing multimodal reasoning capabilities of MLLMs via visual perception reward. In: The Fourteenth International Conference on Learning Representations (2026),https://openreview.net/forum?id=K...

    2026

  21. [29]

    Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., Zhang, B., Wang, X., Chu, Y., Lin, J.: Qwen2.5-omni technical report (2025), https://arxiv.org/abs/2503.20215

    2025 arXiv

  22. [30]

    arXiv preprint arXiv:2506.21277 (2025)

    Yang, Q., Yao, S., Chen, W., Fu, S., Bai, D., Zhao, J., Sun, B., Yin, B., Wei, X., Zhou, J.: Humanomniv2: From understanding to omni-modal reasoning with context. arXiv preprint arXiv:2506.21277 (2025)

    2025 arXiv

  23. [31]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zadeh, A., Chan, M., Liang, P.P., Tong, E., Morency, L.P.: Social-iq: A ques- tion answering benchmark for artificial social intelligence. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8807– 8817 (2019)

    2019

  24. [32]

    In: The Fourteenth International Confer- ence on Learning Representations (2026),https://openreview.net/forum?id= xindJJLSr1

    Zhang, C., Wang, Z., Ma, Y., Peng, J., Wang, Y., Zhou, Q., Song, J., Zheng, B.: Rewatch-r1: Boosting complex video reasoning in large vision-language mod- els through agentic data synthesis. In: The Fourteenth International Confer- ence on Learning Representations (2026),https...

    2026

  25. [33]

    In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

    Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual lan- guage model for video understanding. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 543–553 (2023)

    2023

  26. [34]

    arXiv preprint arXiv:2503.05379 (2025)

    Zhao, J., Wei, X., Bo, L.: R1-omni: Explainable omni-multimodal emotion recog- nition with reinforcement learning. arXiv preprint arXiv:2503.05379 (2025)

    2025 arXiv

  27. [35]

    arXiv preprint arXiv:2512.09616 (2025)

    Zhong, Y., Hu, Z.Y., Li, Y., Wang, L.: Rethinking chain-of-thought reasoning for videos. arXiv preprint arXiv:2512.09616 (2025)

    2025

  28. [36]

    arXiv preprint arXiv:2504.21277 (2025)

    Zhou, G., Qiu, P., Chen, C., Wang, J., Yang, Z., Xu, J., Qiu, M.: Reinforced mllm: A survey on rl-based reasoning in multimodal large language models. arXiv preprint arXiv:2504.21277 (2025)

    2025 arXiv

  29. [37]

    arXiv preprint arXiv:2505.17862 (2025) 18 K

    Zhou, Z., Wang, R., Wu, Z., Jiang, Y.G.: Daily-omni: Towards audio-visual rea- soning with temporal alignment across modalities. arXiv preprint arXiv:2505.17862 (2025) 18 K. P. de Vries et al. Reasoning for Social Audio-Visual Question Answering: Where Do We Stand? Supplementa...

    2025

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.