Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper builds Friends-MMC, a multi-modal multi-party conversation dataset from the TV series Friends, and shows that a modular baseline with a quadratic binary optimization solver identifies speakers at 83.21% accuracy, beating…

desk verdict Useful new multimodal dataset, but the face-label threshold is tuned on the test season; fix that and this is a solid contribution. read the letter →

arxiv 2412.17295 v1 pith:OVV6BNXQ submitted 2024-12-23 cs.CL

Yueqian Wang , Xiaojun Meng , Yuxuan Wang , Jianxin Liang , Qun Liu , Dongyan Zhao This is my paper · ORCID
classification cs.CL
keywords multi-modalconversationmulti-partyspeakeridentificationresponsepredictionactivedetectionquadraticbinaryoptimizationFriendsdatasetsituateddialogue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces multi-modal multi-party conversation (MMC) as a research area and releases Friends-MMC, a dataset of more than 24,000 utterances from the TV series Friends, each paired with video, audio, speaker labels, face bounding boxes, and face names. It defines two tasks: conversation speaker identification and conversation response prediction. The paper claims that a modular baseline—visual active-speaker detection plus a text model for same-speaker relations plus a quadratic binary optimization solver—identifies the speaker of each turn at 83.21% accuracy on the clean 5-turn test set, beating fine-tuned multimodal pretrained models (LLaVA 46.30%, Emu 61.76%, GPT-4o 66.36%) and roughly matching human accuracy (82.25%). It also claims that adding speaker information, even when predicted automatically, improves response prediction in both text-only and multimodal generative models. The dataset and code are released to encourage further work.

What carries the argument

The load-bearing object is the three-module baseline with the quadratic binary optimization solver at its center. From the visual model it builds a vision reward matrix $B \in \mathbb{R}^{l \times m}$, where $B_{ij}$ is the probability that face $c_i$ is the speaking face for turn $u_j$; from the text model it builds a text reward matrix $A = p_{\text{sim}} - \text{mean}(p_{\text{sim}})$, the mean-subtracted pairwise same-speaker probabilities. The solver maximizes $f(X) = (1-\alpha)X^T A X + \alpha X B$ over binary assignment matrices $X$, with the constraint that each turn gets exactly one speaker, and $\alpha$ weighting the two modalities. The mean subtraction prevents the solver from assigning all turns to one speaker, and $\alpha$ is chosen on a held-out validation set.

What would settle it

Re-run face labeling with the similarity threshold tuned on seasons 1-2 instead of season 3 and measure the 5-turn test accuracy on season 3; a material drop below 83.21% would show the reported result depends on tuning the training-data construction to the test season.

Watch

Extended reading notes

Core claim

The central claim is that speaker identification in situated multi-party dialogue can be solved accurately by modular component models plus global optimization, without end-to-end multimodal fusion. The baseline uses a visual model (TalkNet for video or a CNN for single frames) to score each visible face as the current speaker, a text model (DeBERTa-v3) to estimate whether each pair of utterances was spoken by the same person, and a quadratic binary optimization solver that assigns every turn exactly one speaker by balancing the two reward matrices. On the clean 5-turn test set this reaches 83.21% accuracy, above LLaVA-13B (46.30%), Emu-14B (61.76%), and GPT-4o (66.36%), and slightly above the measured human accuracy (82.25%). For response prediction, the paper shows that feeding ground-truth or automatically inferred speaker names into Llama2-7B and Emu-14B consistently improves the accuracy of selecting the true next utterance among ten candidates, and argues that the main benefit is speaker names acting as a local-context indicator rather than a global speaking style signal.

Load-bearing premise

The automatic face-labeling threshold t=0.6 is chosen by validating on Season 3, the same season that forms the test set, so the reported test accuracy may be inflated by tuning training-data construction to the test distribution.

Editorial extensions

If this is right

  • Speaker identification in situated multi-party conversation can be decomposed into per-face speaking probability and pairwise same-speaker relations, then solved globally, without requiring end-to-end multimodal pretrained fusion.
  • The text-based same-speaker model provides a meaningful boost, especially when the speaker is not visible in the current clip or frame, and the benefit grows when the visual model is stronger.
  • Including speaker information, even from an automatic speaker-identification system, improves response prediction in both text-only and multimodal generative models.
  • Speaker names act primarily as a local-context indicator, not a global speaking style tag, so models should use speaker identity to anchor whose perspective the next utterance takes.
  • The released dataset enables future work on character-centered dialogue understanding in a natural, situated, multi-party setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the reported test accuracy survives a cleaner validation split: the face-labeling threshold t=0.6 was tuned on Season 3, which is also the test season, so the training-data construction may be indirectly optimized for the test distribution.
  • A natural extension would be to replace the discrete quadratic binary solver with a differentiable neural assignment layer, allowing end-to-end training of the visual and textual modules while preserving the global constraint.
  • The modular approach could transfer to real-world meeting analysis, where face tracks and dialogue overlap, though active-speaker detection in meetings is typically more challenging than in a sitcom with clear face tracks.
  • The response-selection evaluation, which picks one of ten candidates by perplexity, may reward shallow cues; a generative evaluation with human judgments could reveal whether the speaker-information gains persist in free-form response generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Friends-MMC, a large multi-modal multi-party conversation dataset built from the TV series Friends, containing 24,000+ unique utterances paired with video clips, speaker labels, face bounding boxes, and face names. It proposes two tasks: conversation speaker identification and conversation response prediction. For speaker identification, the paper designs a modular baseline combining a visual model (CNN for frames or TalkNet for video), a same-speaker text model (DeBERTa-v3), and a quadratic binary optimization solver; it reports 83.21% accuracy on the 5-turn clean test set, outperforming fine-tuned LLaVA, Emu, and GPT-4o, and close to human performance. For response prediction, it fine-tunes Llama2-7B and Emu-14B and shows that speaker information, especially ground-truth speaker names, improves response selection accuracy.

Significance. The dataset fills a genuine gap: existing multi-modal dialogue datasets are mostly two-party and bystander-style, whereas Friends-MMC supports multi-party, situated conversation understanding with speaker and face annotations. The paper is transparent about its construction pipeline and releases code and data, which is commendable. The modular baseline is clearly described, and the ablations (M1 vs. M1+M2) are consistent with the claim that both visual and textual modalities contribute. The response-prediction analysis of speaker information, including random/shuffled controls, is a useful contribution. However, the benchmark-integrity issue with threshold tuning tempers the significance; if resolved, the dataset would be a solid and reusable resource.

major comments (3)
  1. [Automatic Face Labelling / Test & Test-noisy Set] The face-labeling threshold t = 0.6 is selected by maximizing the validation accuracy on Season 3, which is the same season used as the test set. Although the test set uses human-annotated C1C faces, t controls which face tracks are kept and labeled in the training set (Seasons 1, 2, 4–10). Tuning this threshold on the test-season distribution leaks information about the test set into training-data construction, potentially inflating the reported downstream accuracies (Table 2). Please retune t on a held-out season (or on training-season data only), report the sensitivity of the results to t, and re-evaluate the main claims with the resulting threshold. If the effect is small, that should be demonstrated quantitatively.
  2. [Main Results / Table 2] The claim that M1(TalkNet)+M2 (83.21%) 'rivals' human performance (82.25%) is not an apples-to-apples comparison. The human study provided only static frames plus text (no audio or video), whereas TalkNet uses video and audio. The appropriate human comparison for the video+text setting would require a video/audio setting or a clear caveat. Please either temper the claim or report human performance under the same video+audio setting.
  3. [Experiment Results / Table 3] The statement 'regardless of which pre-trained model is used, or the length of dialogue context, in all cases adding speaker information always improves the performance' is contradicted by the Emu-14B row for 5 turns, where the 'Random' condition (29.35) is lower than 'No' (30.49). Please correct this claim or clarify that random speaker names are considered a form of speaker information that may not help.
minor comments (5)
  1. [Quadratic Binary Optimization Problem Solver] The objective is written as f(X) = (1−α)X^T A X + α X B; as written, X^T A X is l×l and X B is m×m, so the objective is not a scalar. Presumably the intended scalar objective uses traces (e.g., tr(X^T A X) + α tr(X B)). Please correct the equation to make the optimization problem well-defined.
  2. [Abstract] The abstract uses 'Friend-MMC' where 'Friends-MMC' is intended; please fix the typo.
  3. [Appendix / Details of Baseline Methods] The appendix contains 'paramter-efficient'; it should be 'parameter-efficient'.
  4. [Table 2 caption] The notation 'LLaV A' appears with an unintended space; please use 'LLaVA' consistently throughout the paper.
  5. [Human Performance appendix] There is a discrepancy between the main text, which says 80 dialogue sessions were sampled, and the appendix instructions, which refer to two folders of 20 pieces and annotation of 60.jpg–79.jpg. Please clarify the sampling and annotation procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the baseline and dataset claims are empirical and are not forced by construction; the Season-3 threshold tuning is a benchmark-integrity concern, not a circular derivation.

full rationale

Friends-MMC is an empirical benchmark paper rather than a derivation from first principles, so the enumerated circularity patterns do not apply. The central baseline (M1 visual model + M2 text model + quadratic binary optimization solver) is trained on Seasons 1, 2, and 4-10 and evaluated on Season 3, with model parameters and the fusion weight alpha selected on held-out training/validation splits. The test labels come from human-annotated C1C faces, not from the automatic pipeline. No equation in the paper defines a predicted output in terms of the target quantity: the vision reward B is built from face-speaking probabilities, the text reward A from pairwise same-speaker relations, and the solver maximizes a stated objective over candidate assignments. The 'prediction' of response selection is likewise evaluated against held-out test candidates. The one legitimate concern is benchmark integrity, not circularity: the automatic face-labelling threshold t = 0.6 is set to maximize face-labelling accuracy on Season 3, which is the same season as the test distribution (Section 'Automatic Face Labelling'), and the reported accuracy could be influenced by this choice. However, this tunes training-data construction, not a fitted parameter that is then renamed as a prediction; the reported 83.21% is not algebraically or statistically forced by that threshold. The human-performance comparison also mixes settings (static frames plus text versus video, audio, and text), which is a soundness caveat but again not circular. Self-citations, such as following Wang, Wang, and Zhao (2023) for the response-selection evaluation, are methodological references and are not load-bearing in the sense of importing an unverified conclusion that the paper then claims to derive. On the specific question of circularity, the paper's empirical claims stand independently of their inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the quality of the dataset construction pipeline, which depends on several domain assumptions about transcript accuracy, character coverage, and face-labeling reliability. The face-labeling threshold is tuned on the test season, a significant concern. No new physical or theoretical entities are introduced.

free parameters (4)
  • Face labeling threshold t = 0.6
    Set to maximize face-labeling accuracy on Season 3 (C1C), which is the test set. This leaks test information into training-data construction.
  • Reward weight alpha = 0.8 (frame), 0.7 (video), 0.2 (with M2-dagger)
    Selected on a validation set held out from the train set; controls trade-off between visual and textual rewards in the solver.
  • Sliding window time gap = 8 seconds
    Hand-chosen heuristic to prevent selecting turns from different scenes; not validated against scene-change annotations.
  • Number of top similarities in face labeling = 5
    The mean of the largest 5 cosine similarities is compared to threshold t; hand-chosen.
assumptions (5)
  • domain assumption Fan-maintained transcripts and subtitles are correctly aligned with the video and accurately attribute utterances to speakers.
    The dataset construction relies on these sources without independent verification beyond the fan community.
  • domain assumption The 18 selected main characters cover all speakers in the chosen sessions.
    Session selection requires all speakers to be from the main characters, so conversations with guest characters are excluded.
  • domain assumption FaceNet-512 embeddings and cosine similarity with the 'largest 5' averaging provide reliable automatic face labeling.
    The method is validated at 95% on Season 3, but the threshold is chosen on that same season, so the validation is not fully independent.
  • domain assumption The 8-second gap heuristic ensures that adjacent turns belong to the same scene.
    This hand-chosen threshold is not validated against scene-change annotations.
  • domain assumption TV series dialogue is a valid proxy for real-world face-to-face multi-party conversation.
    The paper justifies the choice with 'well fits real-world scenarios', but the data is scripted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding." pith.science (2026). https://pith.science/paper/OVV6BNXQ

@misc{pith2026241217295,
  author       = {Pith},
  title        = {Pith review of: Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OVV6BNXQ}},
  note         = {Machine review of arXiv:2412.17295}
}
read the original abstract

Multi-modal multi-party conversation (MMC) is a less studied yet important topic of research due to that it well fits real-world scenarios and thus potentially has more widely-used applications. Compared with the traditional multi-modal conversations, MMC requires stronger character-centered understanding abilities as there are many interlocutors appearing in both the visual and textual context. To facilitate the study of this problem, we present Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique utterances paired with video context. To explore the character-centered understanding of the dialogue, we also annotate the speaker of each utterance, the names and bounding bboxes of faces that appear in the video. Based on this Friends-MMC dataset, we further study two fundamental MMC tasks: conversation speaker identification and conversation response prediction, both of which have the multi-party nature with the video or image as visual context. For conversation speaker identification, we demonstrate the inefficiencies of existing methods such as pre-trained models, and propose a simple yet effective baseline method that leverages an optimization solver to utilize the context of two modalities to achieve better performance. For conversation response prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze the benefits of speaker information. The code and dataset is publicly available at https://github.com/yellow-binary-tree/Friends-MMC and thus we call for more attention on modeling speaker information when understanding conversations.

Figures

Figures reproduced from arXiv: 2412.17295 by the authors.

Figure 1
Figure 1. An example of multi-modal multi-party conversation. The task of conversation speaker identification is to infer the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of the construction process of Friends-MMC dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Model overview of the three modules in different [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The change of accuracy with respect to α. The dotted horizontal line shows the performance of only using the visual model. Given the names and bounding boxes of the faces for the people that appear in the visual context where the dialog takes place: [bbox 1] [face 1], …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 22 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    K.; and Hori, C

    AlAmri, H.; Cartillier, V.; Das, A.; Wang, J.; Lee, S.; Anderson, P.; Essa, I.; Parikh, D.; Batra, D.; Cherian, A.; Marks, T. K.; and Hori, C. 2019. Audio Visual Scene-Aware Dialog. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 7550--7559

    work page 2019

  4. [4]

    M.; and Zisserman, A

    Cao, Q.; Shen, L.; Xie, W.; Parkhi, O. M.; and Zisserman, A. 2017. VGGFace2: A Dataset for Recognising Faces across Pose and Age. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 67--74

    work page 2017

  5. [5]

    Das, A.; Kottur, S.; Gupta, K.; Singh, A.; Yadav, D.; Moura, J. M. F.; Parikh, D.; and Batra, D. 2016. Visual Dialog. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1080--1089

    work page 2016

  6. [6]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics

    2019

  7. [7]

    Feng, J.; Sun, Q.; Xu, C.; Zhao, P.; Yang, Y.; Tao, C.; Zhao, D.; and Lin, Q. 2023. MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation. In Annual Meeting of the Association for Computational Linguistics

    work page 2023

  8. [8]

    Y.; Wang, L.; and Liu, Z

    Fu, T.-J.; Li, L.; Gan, Z.; Lin, K.; Wang, W. Y.; Wang, L.; and Liu, Z. 2021. VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling. ArXiv, abs/2111.12681

    arXiv 2021

Show all 44 references
  1. [9]

    Ganesh, A.; Palmer, M.; and Kann, K. 2023. A Survey of Challenges and Methods in the Computational Modeling of Multi-Party Dialog. Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023)

    2023

  2. [10]

    Gu, J.-C.; Ling, Z.; LIU, Q.; Liu, C.; and Hu, G. 2023. GIFT: Graph-Induced Fine-Tuning for Multi-Party Conversation Understanding. In Annual Meeting of the Association for Computational Linguistics

    2023

  3. [11]

    Gu, J.-C.; Tao, C.; and Ling, Z. 2022. Who Says What to Whom: A Survey of Multi-Party Conversations. In International Joint Conference on Artificial Intelligence

    2022

  4. [12]

    Gu, J.-C.; Tao, C.; Ling, Z.; Xu, C.; Geng, X.; and Jiang, D. 2021. MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding. In Annual Meeting of the Association for Computational Linguistics

    2021

  5. [13]

    Gurobi Optimization, LLC . 2023. Gurobi Optimizer Reference Manual

    2023

  6. [14]

    He, P.; Gao, J.; and Chen, W. 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. arXiv:2111.09543

    2021 arXiv

  7. [15]

    J.; Shen, Y.; Wallis, P.; Allen - Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen - Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net

    2022

  8. [16]

    Hu, W.; Chan, Z.; Liu, B.; Zhao, D.; Ma, J.; and Yan, R. 2019. GSN: A Graph-Structured Network for Multi-Party Dialogues. In International Joint Conference on Artificial Intelligence

    2019

  9. [17]

    S.; and Zisserman, A

    Kalogeiton, V. S.; and Zisserman, A. 2020. Constrained Video Face Clustering using1NN Relations. In British Machine Vision Conference

    2020

  10. [18]

    Li, K.; He, Y.; Wang, Y.; Li, Y.; Wang, W.; Luo, P.; Wang, Y.; Wang, L.; and Qiao, Y. 2023 a . VideoChat: Chat-Centric Video Understanding. ArXiv, abs/2305.06355

    2023 arXiv

  11. [19]

    Li, K.; Wang, Y.; He, Y.; Li, Y.; Wang, Y.; Liu, Y.; Wang, Z.; Xu, J.; Chen, G.; Luo, P.; Wang, L.; and Qiao, Y. 2023 b . MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. ArXiv, abs/2311.17005

    2023 arXiv

  12. [20]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. ArXiv, abs/2304.08485

    2023 arXiv

  13. [21]

    Lowe, R.; Pow, N.; Serban, I.; and Pineau, J. 2015. The U buntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In Koller, A.; Skantze, G.; Jurcicek, F.; Araki, M.; and Rose, C. P., eds., Proceedings of the 16th Annual Meeting of the ...

    2015

  14. [22]

    A.; Khan, S

    Maaz, M.; Rasheed, H. A.; Khan, S. H.; and Khan, F. S. 2023. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. ArXiv, abs/2306.05424

    2023 arXiv

  15. [23]

    Meng, Y.; Wang, S.; Han, Q.; Sun, X.; Wu, F.; Yan, R.; and Li, J. 2020. OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts. ArXiv, abs/2012.15015

    2020 arXiv

  16. [24]

    Meng, Z.; Mou, L.; and Jin, Z. 2018. Towards Neural Speaker Modeling in Multi-Party Conversation: The Task, Dataset, and Models. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)

    2018

  17. [25]

    B.; Galley, M.; Gao, J.; Spithourakis, G

    Mostafazadeh, N.; Brockett, C.; Dolan, W. B.; Galley, M.; Gao, J.; Spithourakis, G. P.; and Vanderwende, L. 2017. Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation. ArXiv, abs/1701.08251

    2017 arXiv

  18. [26]

    OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2024-12-10

    2024

  19. [27]

    Ouchi, H.; and Tsuboi, Y. 2016. Addressee and Response Selection for Multi-Party Conversation. In Conference on Empirical Methods in Natural Language Processing

    2016

  20. [28]

    Poria, S.; Hazarika, D.; Majumder, N.; Naik, G.; Cambria, E.; and Mihalcea, R. 2018. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. ArXiv, abs/1810.02508

    2018 arXiv

  21. [29]

    Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015. FaceNet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 815--823

    2015

  22. [30]

    Shuster, K.; Humeau, S.; Bordes, A.; and Weston, J. 2020. Image-Chat: Engaging Grounded Conversations. In Annual Meeting of the Association for Computational Linguistics

    2020

  23. [31]

    Su, Z.; and Zhou, Q. 2022. Speaker Clustering in Textual Dialogue with Pairwise Utterance Relation and Cross-corpus Dialogue Act Supervision. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-...

    2022

  24. [32]

    Sun, Q.; Yu, Q.; Cui, Y.; Zhang, F.; Zhang, X.; Wang, Y.; Gao, H.; Liu, J.; Huang, T.; and Wang, X. 2023. Generative Pretraining in Multimodality. CoRR, abs/2307.05222

    2023 arXiv

  25. [33]

    E.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A

    Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S. E.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2014. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1--9

    2014

  26. [34]

    K.; Qian, X.; Shou, M

    Tao, R.; Pan, Z.; Das, R. K.; Qian, X.; Shou, M. Z.; and Li, H. 2021. Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection. Proceedings of the 29th ACM International Conference on Multimedia

    2021

  27. [35]

    Wang, S.; Meng, Y.; Li, X.; Sun, X.; Ouyang, R.; and Li, J. 2021. OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts. ArXiv, abs/2109.12761

    2021 arXiv

  28. [36]

    Wang, Y.; Wang, Y.; and Zhao, D. 2023. Overview of the NLPCC 2023 Shared Task 10: Learn to Watch TV: Multimodal Dialogue Understanding and Response Generation. In 12th National CCF Conference, NLPCC 2023, Foshan, China, October 12-15, 2023, Proceedings, Part III , volume 14304...

    2023

  29. [37]

    Wang, Y.; Zheng, Z.; Zhao, X.; Li, J.; Wang, Y.; and Zhao, D. 2023. VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume...

    2023

  30. [38]

    Xu, Y.; Wang, S.; Li, P.; Luo, F.; Wang, X.; Liu, W.; and Liu, Y. 2023. Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf. ArXiv, abs/2309.04658

    2023 arXiv

  31. [39]

    Zang, X.; Liu, L.; Wang, M.; Song, Y.; Zhang, H.; and Chen, J. 2021. PhotoChat: A Human-Human Dialogue Dataset With Photo Sharing Behavior For Joint Image-Text Modeling. ArXiv, abs/2108.01453

    2021 arXiv

  32. [40]

    B.; Shu, T.; and Gan, C

    Zhang, H.; Du, W.; Shan, J.; Zhou, Q.; Du, Y.; Tenenbaum, J. B.; Shu, T.; and Gan, C. 2023. Building Cooperative Embodied Agents Modularly with Large Language Models. CoRR, abs/2307.02485

    2023 arXiv

  33. [41]

    Zhang, S.; Zhu, X.; Lei, Z.; Shi, H.; Wang, X.; and Li, S. 2017. S3FD: Single Shot Scale-Invariant Face Detector. 2017 IEEE International Conference on Computer Vision (ICCV), 192--201

    2017

  34. [42]

    Zheng, Y.; Chen, G.; Liu, X.; and Lin, K. W. 2022. MMChat: Multi-Modal Chat Dataset on Social Media. ArXiv, abs/2108.07154

    2022 arXiv

  35. [43]

    P.; Gupta, A.; McKee, K

    Zhou, P.; Madaan, A.; Potharaju, S. P.; Gupta, A.; McKee, K. R.; Holtzman, A.; Pujara, J.; Ren, X.; Mishra, S.; Nematzadeh, A.; Upadhyay, S.; and Faruqui, M. 2023. How FaR Are Large Language Models From Agents with Theory-of-Mind? ArXiv, abs/2310.03051

    2023 arXiv

  36. [44]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. ArXiv, abs/2304.10592

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.