Pith. sign in

REVIEW 4 major objections 5 minor 58 references

Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces VENUS, a 14,910-hour corpus of two-person podcast videos with time-aligned 3D face and body annotations, and MARS, a language model that learns to generate words, facial expressions, and gestures in one token stream.

desk verdict VENUS is a genuinely useful new dataset; MARS's evaluation does not support the paper's strongest claims, so the paper needs substantial revision, not rejection. read the letter →

arxiv 2506.00958 v1 pith:GHJ6VYBT submitted 2025-06-01 cs.AI cs.CLcs.CV

Youngmin Kim , Jiwan Chung , Jisoo Kim , Sunghyun Lee , Sangkyu Lee , Junhyeok Kim , Cheoljong Yang , Youngjae Yu This is my paper
classification cs.AIcs.CLcs.CV
keywords multimodaldialoguenonverbalcommunicationVQ-VAEtokenization3Dfacialexpressionbodylanguagegenerationvideo-groundedconversationslargemodelpodcastdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models converse fluently in text but are blind to the facial expressions and gestures that carry much of a real conversation's meaning. The paper attempts to close that gap by building VENUS, a large corpus of two-person podcast videos in which transcripts are aligned with pseudo-3D estimates of facial expression and body gesture, and by training MARS, a language model that treats nonverbal cues as discrete tokens interleaved with ordinary words. The central claim is that nonverbal behavior is learnable with the same next-token prediction used for text: given a conversational turn, the model emits the next words plus appropriate face and body codes in one autoregressive pass. If that holds, conversational agents, virtual humans, and game characters could produce visible, context-appropriate behavior rather than text alone. Quantitative results show MARS assigns lower negative log-likelihood to held-out face and body tokens than text-only baselines, with better text scores, which the authors read as evidence that the model has learned conversational nonverbal patterns.

What carries the argument

The load-bearing mechanism is the discretization and interleaving of nonverbal cues. Face parameters (53 values per frame: expression coefficients and jaw pose) and body parameters (117 values per frame: upper body and both hands) are smoothed and passed through two VQ-VAEs, discrete-token autoencoders, that compress each motion sequence into codebook indices. Those indices are inserted into the text stream at each word position, yielding one sequence in which speech and gesture share timestamps. Understanding and generation then reduce to a single next-token prediction problem over that mixed sequence, which is what lets a model trained for text be fine-tuned to emit gesture as well as words. The pseudo-label extraction step is what makes the dataset large: monocular video frames are cheap, and the two reconstruction models, EMOCA-v2 for faces and OSX for bodies, supply 3D motion without motion-capture suits.

What would settle it

Run the VENUS extraction pipeline on a set of dyadic conversations recorded simultaneously with marker-based motion capture or multi-view high-resolution video, and compare the reconstructed 3D face and body parameters against the motion-capture ground truth. If the pseudo-label error is comparable to or larger than the measured quality gap between MARS and a text-only baseline, then pseudo-label noise, not the modeling approach, is what carries the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that the obstacle to non-textual dialogue is data, not architecture. VENUS supplies that data at scale, with 89,459 ten-minute dialogues, 1,114,328 turns, and about one billion nonverbal parameter entries, by running a fully automatic pipeline on YouTube podcasts: speaker diarization, time-aligned transcription, active-speaker detection, per-frame extraction of FLAME facial parameters and SMPL-X body parameters (parametric models of the head and whole body), and VQ-VAE compression of both streams into 512-entry codebooks. MARS then treats a conversation as one token sequence ordered word, face, body, word, and so on, and predicts all three autoregressively. On a held-out test set, MARS achieves lower perplexity and higher BERTScore (a semantic similarity metric) for text than the same base language models without nonverbal tokens, and substantially lower negative log-likelihood for face and body tokens, indicating that it generates gestures that are plausible and contextually aligned. The paper frames this as evidence that real conversational behavior, not just language, can be learned from in-the-wild video.

Load-bearing premise

The load-bearing premise is that the 3D face and body parameters auto-extracted from ordinary YouTube video frames are accurate enough to act as teacher signals for natural nonverbal behavior; if those reconstructions are noisy or systematically wrong, the error is baked into both the dataset and everything trained on it.

Editorial extensions

If this is right

  • A single next-token objective can carry both linguistic and gestural generation, so gesture output needs no separate synthesis stage once the VQ codebooks exist.
  • The data pipeline is scalable: filtering and transcribing audio first, then reconstructing 3D motion from monocular frames, avoids motion-capture equipment and can be pointed at other video sources or languages.
  • Face and body tokens can be added as special tokens to an existing instruction-tuned language model, so conversational nonverbal behavior is acquired by supervised fine-tuning rather than by redesigning the model.
  • Performance on both text and nonverbal tokens improves with model size, suggesting that the usual scaling behavior of language models carries over to the combined modality stream.
  • Decoded MARS outputs are renderable 3D parameters, so the same trained model can drive an avatar in virtual-human or game settings directly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the corpus is podcast-based, the learned nonverbal style is the host-and-guest register: relaxed postures, chin rests, and explanatory hand gestures; a likely testable behavior is that MARS will under-generate strong emotions such as crying or anger, a gap the paper itself acknowledges.
  • The same interleaved-token design suggests a natural extension the paper names but does not test: adding vocal prosody as a third nonverbal stream, which would give the model tone of voice alongside face and body.
  • The reported metrics measure token likelihood and reconstruction fidelity, not social appropriateness; a stricter test would ask human viewers whether a generated gesture fits the conversational context better than a random or text-only baseline, which is a different and harder question than negative log-likelihood.
  • Since only video IDs and template meshes are released, an independent audit is possible: rerun the pseudo-labeling pipeline on a sample of VENUS videos, compare against manual pose and expression annotation, and quantify how much downstream MARS quality degrades when pseudo-label noise is artificially amplified.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VENUS, a large-scale dataset of YouTube podcast dialogues with time-aligned text, FLAME-based facial parameters, and SMPL-X body parameters extracted by monocular pseudo-labeling, and MARS, an LLM fine-tuned to interleave text with VQ-VAE-quantized face and body tokens under a next-token objective. The dataset analysis, reconstruction ablations, and MARS results are presented as evidence that the corpus is large and that the model can generate text, face, and body outputs. The manuscript explicitly promises user studies in the introduction, but no user study or any human evaluation appears in the experimental sections; the generation-quality claims therefore rest on token-likelihood scores and a single qualitative example.

Significance. If the dataset is released as promised and its pseudo-label quality is validated, VENUS would be a substantial resource: it is larger in dialogue length and in 3D nonverbal annotations than comparable conversational and gesture datasets, and it unifies text, facial expression, and body language in one corpus. The VQ-VAE discretization and the MARS hierarchical token interleaving are reasonable design choices, and the acknowledged limitations (podcast-domain bias, pseudo-labeling inaccuracy, incomplete metric coverage) are stated candidly. However, the central claim that MARS produces 'natural and contextually aligned nonverbal expressions' is not established by the reported evidence, because no human evaluation is conducted and the quantitative baselines in Table 4 are set up in a way that makes the comparison unsurprising. The paper is a promising resource contribution whose model-evaluation claims require substantial additional support.

major comments (4)
  1. [§1, §5.3, Fig. 4] The introduction's final contribution bullet states that the effectiveness of MARS is 'supported by user studies, quantitative evaluations, and qualitative analyses,' but Section 5 contains no user study or any human evaluation; the only qualitative evidence is the single illustrative example in Figure 4. The NLL-F and NLL-B numbers in Table 4 are computed on ground-truth code indices and therefore measure fit to the VENUS pseudo-labels, not the naturalness, diversity, contextual appropriateness, or temporal plausibility of sampled motions. This is load-bearing for the paper's central claim, so the revision should either add a human evaluation (e.g., preference or naturalness rating of sampled face/body sequences) or add generation-level automatic metrics and remove the user-study promise.
  2. [Table 4, §5.3] The zero-shot LLaMA and Qwen baselines were never trained on the <FACE_*> and <BODY_*> special tokens, so their much higher perplexity and NLL values are expected from tokenizer and vocabulary mismatch; this table therefore does not test whether MARS's nonverbal outputs are good. The comparison also confounds architecture with training, because MARS is supervised-fine-tuned on VENUS while the baselines are zero-shot. The revision should include matched baselines, for example the same base LLM fine-tuned on VENUS text only, or fine-tuned with nonverbal tokens but with the token embeddings frozen, and should compute NLL over the same token vocabulary for all models.
  3. [§3.1 (P5/P6), §7] The pseudo-labeled FLAME parameters from EMOCA-v2 and SMPL-X parameters from OSX are treated as ground truth for all downstream analyses and for the NLL evaluation, yet the manuscript reports no validation of reconstruction accuracy, no filtering by confidence or tracking quality, and no error analysis on the extracted 3D parameters. Since the paper claims 'substantial scale and high effectiveness' for VENUS and since both VENUS annotations and MARS training targets inherit errors from these estimators, the revision should quantify reconstruction reliability, for example by reporting error on benchmark data with ground-truth 3D capture or by providing a sample-level validation study.
  4. [Table 3] The VQ-VAE ablation reports only point estimates for VMSE, LVD, w-VL2, diversity, and variance, with no variance across runs or statistical significance testing. The conclusion that the chosen L1 loss, codebook size 512, and embedding dimensions 8/16 'outperformed previous works' is therefore based on single training runs; error bars or repeated-run statistics are needed to make the configuration choices robust.
minor comments (5)
  1. [§3.1] The sentence 'If the number of predicted speaker frames is less than the more number of predicted words from WhisperX' appears to contain a typo; 'the more number' should likely be 'the number.'
  2. [§3.1 vs. Appendix A.7] Section 3.1 states that facial parameters mf_l have dimension 156, while Appendix A.7 states that the facial expression features total 153; the discrepancy should be resolved or explained.
  3. [Appendix A.3, Algorithm 1] The re-alignment condition 'if k = 2' appears to trigger the cosine-similarity comparison only for the second frame; if the intent is to re-align periodically or whenever a new speaker starts, the condition and surrounding logic need clarification.
  4. [§7] The limitation section contains a typo, 'nameed MARS,' which should read 'named MARS.'
  5. [§4, Table 1] The footnote for Table 1 explains the '*' entries as estimates, but the exact estimation procedure for '# Turns' and 'Length (hrs)' of YTD-18M is not clearly described in the main text; a sentence in the caption or the footnote would improve clarity.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation chain: MARS is evaluated by held-out NLL/PPL on VENUS code indices, and the self-cited Champagne/MERLOT filtering pipeline is only a non-load-bearing data-collection heuristic; the missing user study is an evaluation gap, not circularity.

full rationale

VENUS is constructed from YouTube podcasts using external ASR, diarization, active-speaker detection, and pseudo-3D reconstruction tools (EMOCA-v2, OSX), then quantized by separately trained VQ-VAEs; MARS is trained and evaluated with a standard next-token objective on held-out VENUS utterances. Equation (11) defines the factorization of the training objective, and Table 4 reports NLL/PPL on a 1,000-utterance held-out subset, so the quantitative evaluation measures generalization to unseen VENUS tokens rather than fitting a parameter to the test quantity. The paper's Section 1 promise of 'user studies' is not delivered in Section 5, and Section 7 concedes that the evaluation metrics 'may not fully capture the nonverbal communication'; these are substantive evaluation limitations, but they are not instances of a derived claim reducing to its own input. The only self-citation of note is the reuse of the filtering process from Champagne (Han et al., 2023) and MERLOT (Zellers et al., 2021a), papers sharing the last author; this pipeline is a data-collection heuristic and is not used to justify MARS's generation quality, so it is a minor, non-load-bearing self-citation rather than circularity. No equation defines a target quantity in terms of itself, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 10 free parameters · 6 assumptions · 0 invented entities

VENUS assumes a chain of automatic tools: WhisperX, PyAnnote, LightASD, EMOCA-v2, and OSX. The most fragile is pseudo-labeled 3D reconstruction, which the authors themselves flag. The model also depends on many empirically chosen loss weights and codebook sizes. No new physical entities are introduced.

free parameters (10)
  • Face VQ-VAE codebook size = 512
    Chosen by reconstruction validation loss; reported in Table 3 ablation.
  • Face VQ-VAE embedding dimension = 8
    Selected via Table 3 ablation as the best face configuration.
  • Body VQ-VAE codebook size = 512
    Chosen by reconstruction validation loss; reported in Table 3 ablation.
  • Body VQ-VAE embedding dimension = 16
    Selected via Table 3 ablation as the best body configuration.
  • VQ-VAE downsampling ratio q = 8
    Architecture choice from prior VQ-VAE motion works; controls token sequence length.
  • Sequence window W = 512 frames
    Set in Appendix B.1 to cover utterance-level sequences, with shorter utterances zero-padded.
  • Commitment loss weight beta = 0.02
    Set empirically in Appendix B.1.
  • Reconstruction and velocity loss weights = lambda_psi=1, lambda_jaw=5, lambda_face_vel=0.5, lambda_theta=5, lambda_body_vel=0.5
    Reported as empirically chosen in Appendix B.1.
  • MARS SFT epochs = 50
    Training choice in Appendix C.1, not justified against alternatives.
  • MARS maximum sequence length = 4096 training, 512 inference
    Training and inference settings in Appendix C.1.
assumptions (6)
  • domain assumption EMOCA-v2 and OSX pseudo-labels are accurate proxies for facial expressions and body language.
    Invoked in Section 3.1 P5 and P6; explicitly flagged as potentially inaccurate in Section 7.
  • domain assumption WhisperX transcripts and PyAnnote diarization correctly identify exactly two English speakers and align timestamps.
    Used in Section 3.1 filters F3 and F4 and processing step P2.
  • domain assumption LightASD active speaker detection and cosine-similarity re-identification keep crops on the correct speaker.
    Used in Section 3.1 P3 and P4 and detailed in Algorithm 1 in Appendix A.3.
  • domain assumption Podcast videos from YouTube channels are representative of natural dyadic conversation.
    Collection strategy in Section 3.1 and Appendix A.2; diversity limitation acknowledged in Section 7.
  • domain assumption SMPL-X and FLAME parameter spaces span the nonverbal cues relevant to dialogue.
    Face and body vectors defined in Eq. 1; no independent perceptual validation is provided.
  • standard math The autoregressive factorization in Eq. 11 is a valid probability model for interleaved text and motion tokens.
    Standard next-token factorization, but ordering text before face and body per position is a modeling choice, not proven optimal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues." pith.science (2026). https://pith.science/paper/GHJ6VYBT

@misc{pith2026250600958,
  author       = {Pith},
  title        = {Pith review of: Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHJ6VYBT}},
  note         = {Machine review of arXiv:2506.00958}
}
read the original abstract

Nonverbal communication is integral to human interaction, with gestures, facial expressions, and body language conveying critical aspects of intent and emotion. However, existing large language models (LLMs) fail to effectively incorporate these nonverbal elements, limiting their capacity to create fully immersive conversational experiences. We introduce MARS, a multimodal language model designed to understand and generate nonverbal cues alongside text, bridging this gap in conversational AI. Our key innovation is VENUS, a large-scale dataset comprising annotated videos with time-aligned text, facial expressions, and body language. Leveraging VENUS, we train MARS with a next-token prediction objective, combining text with vector-quantized nonverbal representations to achieve multimodal understanding and generation within a unified framework. Based on various analyses of the VENUS datasets, we validate its substantial scale and high effectiveness. Our quantitative and qualitative results demonstrate that MARS successfully generates text and nonverbal languages, corresponding to conversational input.

Figures

Figures reproduced from arXiv: 2506.00958 by the authors.

Figure 1
Figure 1. Overview of VENUS collection pipeline. (a) and (b) use only audio information, while (c) and (d) also utilize visual information. The blue boxes contain filtering criteria (F), and the yellow boxes pertain to the processing steps (P). The final box shown in (d) represents the facial expression and body language combined and represented using SMPL-X parameters. For more details, refer to the Section 3.1. interactions… view at source ↗
Figure 2
Figure 2. System overview. Our system consists of two main parts: (a) the VQ-VAE model trained to quantize nonverbal cues, and (b) a MARS trained to process quantized nonverbal expressions alongside text. The output generated by the assistant is visualized by replacing both face and body parameters with SMPL-X. For codebook learning, we use commitment loss, Lvq, in the proposed (Van Den Oord et al., 2017). Lvq = β||z − sg(ˆz)… view at source ↗
Figure 3
Figure 3. Visualization of the distribution of nonverbal-cues. (a) Facial expression embeddings are well-clustered despite the absence of emotion class la￾bels, capturing meaningful emotion patterns. (b) Body language embeddings are similarly well-clustered, rep￾resenting common conversational gestures that enhance communication or naturally occur during dialogue. Rep￾resentative examples are provided for each cluster. from a… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative results for MARS. Qualitative re￾sults showcasing inputs and outputs of our MARS model. Inputs include the user’s text, face, and body language, while MARS outputs corresponding text, facial expres￾sions, and body language. Underlined text indicates where M…
Figure 5
Figure 5. Figure 5: The diversity of topics of videos in VENUS, displayed as a word cloud. Larger words indicate more videos from that topic. mension of the image is resized to S, maintaining the asepct ratio. The image is then resized to new dimensions h ′ = s × h and w ′ = s × w. After …
Figure 7
Figure 7. Figure 7: Overview of VQ-VAE architecture. Encoder (left) quantizes the speaker’s noverbal-cues, while the decoder (right) projcets the learned discrete codebook tokens back into continuous nonverbal-cues sequence space. The downsampling block consists of 1D convolu￾tional layer…
Figure 8
Figure 8. Figure 8: Visualization for VENUS dataset. This demonstrates the capability of the VENUS dataset to capture multimodal communication, encompassing speech, body language, and facial expressions. Words are time-aligned using WhisperX, with YouTube IDs providing access to ground tr…
Figure 9
Figure 9. Figure 9: VENUS annotation format. This is an example of an annotation for a single segmented video. We provide the VENUS dataset in JSON format [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 36 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    work page 2023

  3. [3]

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whisperx: Time-accurate speech transcription of long-form audio. arxiv. arXiv preprint arXiv:2303.00747

    arXiv 2023

  4. [4]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Comput...

    2005

  5. [5]

    Yoshua Bengio, R \'e jean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in neural information processing systems, 13

    work page 2000

  6. [6]

    Herv \'e Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. 2020. Pyannote. audio: neural building blocks for speaker diarization. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7124--7128. IEEE

    2020

  7. [7]

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335--359

    work page 2008

  8. [8]

    Jose Camacho-Collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthenis Antypas, Joanne Boisson, Luis Espinosa-Anke, Fangyu Liu, Eugenio Mart \' nez-C \'a mara, et al. 2022. Tweetnlp: Cutting-edge natural language processing for social media. arXiv preprint arXiv:2206.14774

    arXiv 2022

Show all 58 references
  1. [9]

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793

    2023 arXiv

  2. [10]

    Neeraj Cherakara, Finny Varghese, Sheena Shabana, Nivan Nelson, Abhiram Karukayil, Rohith Kulothungan, Mohammed Afil Farhan, Birthe Nesset, Meriam Moujahid, Tanvi Dinkar, et al. 2023. Furchat: An embodied conversational agent using llms, combining open and closed-domain dialog...

    2023

  3. [11]

    Glen Coppersmith and Erin Kelly. 2014. Dynamic wordclouds and vennclouds for exploratory data analysis. In Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces, pages 22--29

    2014

  4. [12]

    Radek Dan e c ek, Michael J Black, and Timo Bolkart. 2022. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20311--20322

    2022

  5. [13]

    Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, Yao Feng, and Michael J Black. 2024. Tokenhmr: Advancing human mesh recovery with a tokenized pose representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1323--1333

    2024

  6. [14]

    Martin Ester, Hans-Peter Kriegel, J \"o rg Sander, Xiaowei Xu, et al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226--231

    1996

  7. [15]

    Peter A Gorry. 1990. General least-squares smoothing and differentiation by the convolution (savitzky-golay) method. Analytical Chemistry, 62(6):570--573

    1990

  8. [16]

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. 2024. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900--1910

    2024

  9. [17]

    Seungju Han, Jack Hessel, Nouha Dziri, Yejin Choi, and Youngjae Yu. 2023. Champagne: Learning real-world conversation from large-scale web videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15498--15509

    2023

  10. [18]

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406.18495

    2024 arXiv

  11. [19]

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023. https://github.com/ultralytics/ultralytics Ultralytics YOLO

    2023

  12. [20]

    Yoon Kyung Lee, Yoonwon Jung, Gyuyi Kang, and Sowon Hahn. 2023. Developing social robots with empathetic non-verbal cues using large language models. arXiv preprint arXiv:2308.16529

    2023 arXiv

  13. [21]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355

    2023 arXiv

  14. [22]

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194--1

    2017

  15. [23]

    Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, and Liangyin Chen. 2023. A light weight model for active speaker detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22932--22941

    2023

  16. [24]

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. 2024. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36

    2024

  17. [25]

    Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. 2023. One-stage 3d whole-body mesh recovery with component aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21159--21168

    2023

  18. [26]

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J Black. 2024 a . Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In Proceedings of the IE...

    2024

  19. [27]

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. 2022. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. In European conference on computer vision, pages 612--630. Springer

    2022

  20. [28]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  21. [29]

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. 2024. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  22. [30]

    Liying Lu, Tianke Zhang, Yunfei Liu, Xuangeng Chu, and Yu Li. 2023 a . Audio-driven 3d facial animation from in-the-wild videos. arXiv preprint arXiv:2306.11541

    2023 arXiv

  23. [31]

    Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. 2023 b . Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978

    2023 arXiv

  24. [32]

    Meta. 2024. Llama 3 & 2 connect 2024: Vision for edge and mobile devices. [Online] https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/. Accessed: 2024-12-16

    2024

  25. [33]

    Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. 2022. Learning to listen: Modeling non-deterministic dyadic facial motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20395--20405

    2022

  26. [34]

    Evonne Ng, Sanjay Subramanian, Dan Klein, Angjoo Kanazawa, Trevor Darrell, and Shiry Ginosar. 2023. Can language models learn to listen? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10083--10093

    2023

  27. [35]

    Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. 2024. Let's go real talk: Spoken dialogue model for face-to-face conversation. arXiv preprint arXiv:2406.07867

    2024 arXiv

  28. [36]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. 2019. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern reco...

    2019

  29. [37]

    Deepika Phutela. 2015. The importance of non-verbal communication. IUP Journal of Soft Skills, 9(4):43

    2015

  30. [38]

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics...

    2019

  31. [39]

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32

    2019

  32. [40]

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510--4520

    2018

  33. [41]

    Zoya Shafique, Haiyan Wang, and Yingli Tian. 2023. Nonverbal communication cue recognition: A pathway to more accessible communication. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5666--5674

    2023

  34. [42]

    Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30

    2017

  35. [43]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11)

    2008

  36. [44]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  37. [45]

    Qi Wu, Yubo Zhao, Yifan Wang, Yu-Wing Tai, and Chi-Keung Tang. 2024. Motionllm: Multimodal motion-language learning with large language models. arXiv preprint arXiv:2405.17013

    2024 arXiv

  38. [46]

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  39. [47]

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. 2023. Generating holistic 3d human motion from speech. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 469--480

    2023

  40. [48]

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computation...

    2018

  41. [49]

    Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Com...

    2022

  42. [50]

    Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021 a . Merlot: Multimodal neural script knowledge models. Advances in neural information processing systems, 34:23634--23651

    2021

  43. [51]

    Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021 b . Merlot: Multimodal neural script knowledge models. Advances in neural information processing systems, 34:23634--23651

    2021

  44. [52]

    Hang Zhang, Xin Li, and Lidong Bing. 2023 a . Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858

    2023 arXiv

  45. [53]

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. 2023 b . Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogn...

    2023

  46. [54]

    Sitao Zhang, Yimu Pan, and James Z Wang. 2023 c . Learning emotion representations from verbal and nonverbal communication. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18993--19004

    2023

  47. [55]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.