Pith. sign in

REVIEW 4 major objections 5 minor 93 references

InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read InteracVid is a 454K-sample dataset pairing live-chat queries with the real streamer reactions that follow, built to train models that respond with audio and video rather than captions.

desk verdict A genuinely useful dataset for interactive audio-visual generation, with the 'real interactive' label overstating the reconstructed branch — but the real branch and careful evaluation carry the paper. read the letter →

arxiv 2608.01157 v1 pith:CNDEOMZK submitted 2026-08-02 cs.CV

Chi Zhang , Haoyang Shi , Yueyi Liu , Zhaokun Yan , Yishu Yin , Yuhang Wu , Miao Liu This is my paper
classification cs.CV
keywords interactiveaudio-visualgenerationlive-chatvideosdatasetcurationcontext-query-responsetripletstwo-stagemultimodalinteractionhumanevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InteracVid addresses a supervision gap in multimodal generation: audio-video models are mostly trained to render captions, not to produce responses caused by an external user query. The paper builds a dataset of over 454K context-query-response triplets from more than 59K livestreams, each pairing a preceding audio-visual context and a user query with the real streamer reaction that follows. Roughly 39K pairs come from genuine live-chat metadata; the remaining 414K come from a reconstructed-query branch in which a vision-language model identifies reactive response moments and synthesizes plausible triggering queries, while the response footage is always real. A ten-rater human study finds both branches causal, natural, and temporally complete, and fine-tuning a two-stage planner-plus-generator pipeline on the dataset improves both stages on a held-out set of 100 genuine queries. The paper concludes that interaction-structured data of this kind is a critical foundation for models that answer with synchronized speech and video.

What carries the argument

The load-bearing mechanism is the metadata-aware curation pipeline, which first turns raw subtitles into a sentence-level semantic timeline for each livestream — a searchable discourse backbone that lets the pipeline judge whether a sentence is a proactive monologue, a reactive response, or a continuation. Videos with live-chat metadata run through a real-query branch that matches time-stamped comments to subsequent streamer sentences; videos without metadata run through a reconstructed-query branch that classifies reactive response spans and asks a vision-language model to synthesize a plausible triggering query from the local video and transcript. On the modeling side, the paper decomposes

What would settle it

Take a set of livestreams that do have live-chat metadata, hide the metadata, run the reconstructed-query branch, and compare each reconstructed query against the actual live-chat comment that triggered that response moment; if the semantic similarity between reconstructed and actual triggers is no higher than between randomly paired queries, the reconstruction branch is not producing faithful proxies.

Watch

Extended reading notes

Core claim

The paper's central claim is that genuine interactive audio-visual response generation can be supervised at scale by mining livestreams. It formalizes the task as modeling $p(Y \mid C, Q)$: given a preceding audio-visual context $C$ and a user query $Q$, the model should produce the real audio-visual response $Y$ that follows. The curation pipeline builds a sentence-level semantic timeline for each livestream, then splits videos by whether time-stamped live-chat metadata exists. With metadata, comments are matched to subsequent streamer sentences, yielding 39K real-query pairs; without it, a vision-language model classifies transcript sentences as proactive or reactive and, for reactive span

Load-bearing premise

The dataset's scale rests on the reconstructed-query branch (414K of 454K samples), which assumes that a vision-language model can reliably distinguish reactive responses from proactive monologues and then synthesize a query that faithfully reproduces the unseen trigger the streamer was actually answering.

Editorial extensions

If this is right

  • Fine-tuning the audio-video co-generator on InteracVid improves all semantic metrics across three different interaction planners, while fine-tuning the planner adds smaller gains, and oracle captions still outperform learned planners — indicating that response planning, not rendering, is the current bottleneck.
  • The reconstructed-query branch is synthetic only in the query condition; the response footage is always real, and human raters score reconstructed queries at least as high as real queries on causality, naturalness, and completeness.
  • Replacing the user query while holding the visual context fixed leaves generated relevance essentially unchanged (4.03 to 3.98), which the paper reads as evidence the model responds to the query rather than producing a scene-driven generic reaction.
  • Supplying the preceding audio-visual context instead of only a single first frame improves audio quality and speaker similarity, indicating continuous temporal context carries speaker identity and acoustic state that a static frame cannot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the 42% of clips with non-speech sound and the 4.3% navigation-like camera-motion clips are released as per-clip annotations; these could be turned into conditioning variables, letting a generator learn when to include music or camera movement in its interactive response.
  • Inference: because reconstructed queries are clean and well-formed while genuine live chat is full of slang and ellipsis, the two branches could support a curriculum — pretrain on reconstructed queries for scale, then fine-tune on real queries to adapt the model to authentic viewer phrasing.
  • Inference: the two-stage decomposition predicts that as audio-video generators improve, the planner becomes the dominant bottleneck; a direct test is to scale only the planner (more data, larger model) on the same 100-query benchmark and check whether the gap between oracle captions and learned planners shrinks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. InteracVid is a dataset paper targeting interactive audio-visual response generation. The authors collect 454K context-query-response triplets from YouTube livestreams: 39K from streams with live-chat metadata ('real' queries) and 414K from streams without such metadata, where a VLM classifies streamer utterances as reactive responses and synthesizes a plausible triggering query. A two-stage pipeline (a VLM planner producing an interaction-aware caption, with MOVA as the audio-video generator) is fine-tuned on the dataset. On 100 genuine-query held-out examples, fine-tuning improves planner and generator metrics; a ten-rater human study reproduces the ordering of configurations, and a counterfactual query replacement test indicates query-driven behavior.

Significance. The dataset addresses a real gap: audio-video generation supervision is mostly descriptive, and InteracVid provides a large corpus with explicit (C, Q, Y) structure. The paper has notable methodological strengths: channel-level train/test disjointness, a counterfactual query test, fixed evaluation prompts and model versions, per-clip acoustic and camera-motion annotations, and a human evaluation with high configuration-level agreement with the VLM judge (Pearson r > 0.94). If the interaction structure of the reconstructed branch were validated, this would be a substantial contribution. However, the central claim that the dataset provides 'real interactive' supervision is currently supported mainly by the 39K real-query branch; the majority branch is generated in a way that risks circularity, and the experiments do not isolate its contribution. With additional ablations and reframing, the contribution could be solid.

major comments (4)
  1. [§3.3, A.4] The reconstructed-query branch inverts the construction: for 414K of 454K samples (91%), a VLM classifies a transcript sentence as 'reactive response' and then generates a plausible query from the response transcript and video. Thus Q is synthesized from Y, so the triple (C, Q, Y) does not record an observed interaction but a model's guess about a hidden cause. Training p(Y|C,Q) on such pairs can learn the correlation between two views of the same response rather than a causal link. The real branch (39K) is independent but small, and no experiment in §5 trains on the real branch alone or measures the marginal contribution of the reconstructed branch. Please add ablations: (i) real-only, (ii) reconstructed-only, (iii) both, evaluated on the held-out genuine-query set; report whether reconstructed-only improves over no fine-tuning and whether it adds to real-only. Without this, the headlin
  2. [§4.3, B.1, Table 2] The human validation does not establish causality for reconstructed queries. The rating axis is defined as 'could this query plausibly have caused this response?' (B.1). A query generated by a VLM to be a plausible trigger will trivially score high on this axis; the high reconstructed causality (4.20) is therefore expected and cannot distinguish faithful reconstruction from a plausible but wrong trigger. The naturalness result (reconstructed > real) is explained by the authors as VLM-generated clean queries, which itself indicates the reconstructed branch is not distributionally identical to real live chat. A stronger validation would compare reconstructed queries against actual chat messages for videos that do have metadata, e.g., hold out a subset of the real branch and measure reconstruction accuracy; otherwise the reported human scores should be described as plausibility, not causali
  3. [§5.2, §5.3, Table 3] The evaluation design does not isolate where the gains come from. All training uses the full dataset (both branches), while the held-out set is 100 genuine-query examples from the real branch (E.1). Improvements over the pretrained baseline could come from the 39K verified branch, from generic adaptation to livestream speech/visual style, or from the reconstructed branch's interaction structure. The paper's conclusion that InteracVid 'provides effective supervision' is stated for the whole dataset. Please add a training-data ablation (real-only vs reconstructed-only vs both) and, if possible, a fixed-size comparison to a non-interactive livestream captioning baseline to rule out domain adaptation as the sole driver. This is load-bearing because the abstract's strongest claim is about 'genuine live-chat queries' and the majority branch is unverified.
  4. [§5.4/E.4] The counterfactual query test is a good evaluation of the trained system: replacing the query while holding the context fixed leaves relevance essentially unchanged (4.03 -> 3.98, Table 12), which shows the model is not purely scene-driven. However, it does not validate the causal status of the training queries. The model could learn to follow query text even if the training Q was not the actual trigger, because the reconstruction function maps observed responses to plausible queries. Please state this limitation explicitly and connect it to the real-only ablation recommended above.
minor comments (5)
  1. [Table 3] OVRL is reported without uncertainty despite being the main aggregate; the claim that dimensions are strongly correlated should be quantified (e.g., average pairwise correlation) or bootstrapped CIs reported.
  2. [F.2 and F.3] The prompts are nearly identical, but F.3 (discourse role classification for the reconstructed branch) still asks for trigger_bullet_index and candidate lists, although no chat metadata exists. This makes the prompt confusing and suggests the pipeline may rely on a matching prompt copy-pasted from the real branch.
  3. [Abstract and §4.3] 'Causal' in the abstract and Section 4.3 overstates what the human study measures; recommend replacing with 'plausibly causal' or similar.
  4. [Figure 5 and Figure 9] Transcript examples show ASR errors (e.g., 'body paint' for 'body pain'); consider noting that ASR errors are a known limitation.
  5. [Abstract] The 'first open-source large-scale dataset' claim may need qualification given LiveStar [64] and SpeakerVid-5M [72]; the Table 1 comparison is useful but the abstract's 'first' claim is not substantiated with a clear definition of scope.

Circularity Check

2 steps flagged · score 6.0 of 10

Reconstructed-query branch (91% of the dataset) generates the query from the response it annotates, and the human 'causality' validation re-tests that same generation criterion; the causal-interaction claim for the majority of the corpus is circular, though the genuine-query benchmark is not.

  1. self definitional [Section 3.3 (Metadata-Aware Interaction Construction); Appendix A.4 (Reconstructed-Query Branch)]
    "For each such span, a vision-language model observes the local video context together with the response transcript, and reconstructs a plausible user query that could have triggered the response."

    The query Q is generated from the response Y (and the video context C), so for the 414K reconstructed samples Q is a function of Y. The paper's data schema requires 'the query Q should serve as the trigger or semantic cause of the response Y' (Section 3.2), but this cause is not observed; it is synthesized after the response is known. Thus the interaction structure in 91% of the corpus is imposed by construction: training on (C, Q_synth, Y) teaches the model to map a query that was inferred from the target back to that target. The response footage is real, but the external stimulus is not.

  2. other [Section 4.3 (Human Validation of Interaction Quality); Appendix B.1 (Human Validation Protocol)]
    "Causality: could this query plausibly have caused this response? A score of 1 means the two are unrelated; 5 means the response is clearly an answer or reaction to the query."

    The reconstructed query was generated by the same plausibility criterion used here: the VLM was asked to 'reconstruct a plausible user query that could have triggered the response' (Section 3.3). Asking human raters whether the query 'could plausibly have caused' the response re-tests the generator's objective; it does not establish that the response was actually triggered by an external user. Because the query is produced from the response, high 'causality' scores are expected by construction. The paper's own observation that reconstructed queries score higher than real ones—because the VLM 'produces a clean, well-formed question'—confirms that the validation is measuring generation quality, not observed interaction.

full rationale

The paper is transparent about the reconstruction, and the held-out evaluation uses genuine live-chat queries, so the fine-tuning results are not themselves circular. The counterfactual query test shows genuine query-following on held-out examples, and the human evaluation of generated videos is independent. However, the dataset's central claim that every sample 'couples a preceding audio-visual context and an external stimulus with the real interactive response that follows' is not supported for the reconstructed branch: the 'external stimulus' is synthesized from the response it is supposed to cause. The human validation of reconstructed queries asks raters to judge exactly the plausibility criterion used to generate them, so it is a consistency check, not causal evidence. The genuine-query branch (39K samples) provides some independent grounding, but no experiment isolates the contribution of the reconstructed branch, so the reported gains are compatible with the reconstructed branch acting as a large set of paraphrase-style pairs rather than verified interaction structure. This is a partial circularity: the causal-interaction claim for the majority of the corpus reduces to the generation objective, while the genuine-query benchmark and its conclusions remain independently meaningful.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The main auxiliary assumptions are about the reliability of proprietary models (Seed-1.6, Gemini-3.1-Pro) used for curation and evaluation, and about the faithfulness of the reconstructed-query branch. Free parameters are curation thresholds and taxonomy quotas, chosen by hand and affecting dataset composition.

free parameters (5)
  • live-chat matching window t_window = 60 s
    Chosen by hand; controls which comments are candidates for the real-query branch. Section A.3.
  • Response span extension length = 5 transcript sentences
    Chosen by hand; defines how far the response extends after the trigger sentence. Sections 3.3 and A.3.
  • Clip padding and minimum length = 0.5 s offset; min 3 s
    Chosen by hand; trims response boundaries and removes very short clips. Section A.5.
  • ASR consistency threshold = 40% word error rate
    Chosen by hand; discards clips where subtitle-derived text disagrees with Seed-ASR. Section A.6.
  • Crawling taxonomy quotas = per-subclass percentages (15%, 5%, 8%, 10%, etc.)
    Chosen by hand; balances domain coverage. Section A.1.
assumptions (4)
  • domain assumption YouTube live-chat metadata and VTT subtitles are accurate enough for temporal alignment and semantic matching.
    The real-query branch relies on comment timestamps and subtitles to pair queries with responses; errors would propagate to response boundaries. Sections 3.3, A.2-A.3.
  • domain assumption LLM/VLM discourse-role classification and semantic correspondence detection are reliable at scale.
    Both branches depend on Seed-1.6 judgments (reactive vs proactive, query matching), which are not independently verified per sample. Sections 3.3, A.3-A.4.
  • domain assumption The VLM judge (Gemini-3.1-Pro) is a valid proxy for human perception of interactive response quality.
    Used for all semantic metrics in the 100-example benchmark; validated on 8 configuration-level means (r > 0.94), a small sample. Sections 5.2, E.3.
  • domain assumption The two-stage factorization p(Y|C,Q) ≈ p(Y|T,C)p(T|C,Q) preserves the interactive semantics.
    Eq. 3 decomposes the joint probability; if the intermediate caption T discards interaction-relevant information, the measured gains would be underestimated. Section 5.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos." pith.science (2026). https://pith.science/paper/CNDEOMZK

@misc{pith2026260801157,
  author       = {Pith},
  title        = {Pith review of: InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNDEOMZK}},
  note         = {Machine review of arXiv:2608.01157}
}
read the original abstract

Large language models have made text the default medium for human--AI interaction, buttext alone cannot express the full range of responses required by multimodal assistants,avatars, and embodied agents. While recent audio-video generative models can synthesizehigh-fidelity synchronized content, existing supervision is largely \emph{descriptive}:models are trained to render captions rather than to produce audio-visual responsescaused by external user interactions. We introduce \textbf{InteracVid}, \emph{the firstopen-source large-scale dataset that addresses this missing supervision}, so that everysample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows. We design a metadata-aware pipeline that extractsinteractive clips from long, noisy livestreams, yielding over \textbf{454K}context-query-response triplets from more than \textbf{59K} livestream videos andspanning conversation-centered, object-centric, procedural, embodied, and screen-basedscenarios. A ten-rater human study confirms that the extracted interactions are causal,natural, and temporally complete for both genuine and reconstructed queries. On aheld-out benchmark of \textbf{100} genuine live-chat queries, fine-tuning on InteracVidimproves both interaction planning and audio-video response generation, and anindependent human evaluation reproduces the system ranking and the conclusions obtainedwith our automatic judge. These results highlight interaction-structured data as acritical foundation for interactive multimodal generation.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

93 extracted references · 38 canonical work pages

  1. [1]

    Technical report / model release page, 2025

    Seed1.6 tech introduction. Technical report / model release page, 2025. URLhttps://seed. bytedance.com/en/seed1_6. Accessed: 2026-05-07

    2025

  2. [2]

    Technical report / model release, 2026

    Wan-Streamer v0.1: End-to-end real-time interactive foundation models. Technical report / model release, 2026

    2026

  3. [3]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Audio visual scene-aware dialog

    Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7558–7567, 2019

    2019

  5. [5]

    Marks, Dhruv Batra, and Devi Parikh

    Huda AlAmri, Chiori Hori, Tim K. Marks, Dhruv Batra, and Devi Parikh. Audio visual scene-aware dialog (avsd) track for natural language generation in dstc7. 2019. URLhttps: //api.semanticscholar.org/CorpusID:221068548

    2019

  6. [6]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Tsinghua University | 15 Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    arXiv 2025

  7. [7]

    Seed-ASR: Understanding diverse speech and contexts with LLM-based speech recognition.arXiv preprint arXiv:2407.04675, 2024

    Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, et al. Seed-ASR: Understanding diverse speech and contexts with LLM-based speech recognition.arXiv preprint arXiv:2407.04675, 2024

    arXiv 2024

  8. [8]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

Show all 93 references
  1. [9]

    Towards interactive intelligence for digital humans

    Yiyi Cai, Xuangeng Chu, Xiwei Gao, Sitong Gong, Yifei Huang, Caixin Kang, Kunhang Li, Haiyang Liu, Ruicong Liu, Yun Liu, et al. Towards interactive intelligence for digital humans. arXiv preprint arXiv:2512.13674, 2025

    2025

  2. [10]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE, 2020

    2020

  3. [11]

    Knowledge enhanced model for live video comment generation

    Jieting Chen, Junkai Ding, Wenping Chen, and Qin Jin. Knowledge enhanced model for live video comment generation. In2023 IEEE International Conference on Multimedia and Expo (ICME), pages 2267–2272. IEEE, 2023

    2023

  4. [12]

    Hunyuanvideo-avatar: High-fidelity audio-driven human animation for multiple characters.arXiv preprint arXiv:2505.20156, 2025

    Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, and Qinglin Lu. Hunyuanvideo-avatar: High-fidelity audio-driven human animation for multiple characters.arXiv preprint arXiv:2505.20156, 2025

    2025 arXiv

  5. [13]

    Qwen2-Audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-Audio technical report. arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  6. [14]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil...

    2025 arXiv

  7. [15]

    ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification

    Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Interspeech, pages 3830–3834, 2020

    2020

  8. [16]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  9. [17]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  10. [18]

    Omniavatar: Efficient audio- driven avatar video generation with adaptive body animation.arXiv preprint arXiv:2506.18866, 2025

    Qijun Gan, Ruizi Yang, Jianke Zhu, Shaofei Xue, and Steven Hoi. Omniavatar: Efficient audio- driven avatar video generation with adaptive body animation.arXiv preprint arXiv:2506.18866, 2025. Tsinghua University | 16

    2025 arXiv

  11. [19]

    Livechat: A large-scale personalized dialogue dataset automatically constructed from live streaming

    Jingsheng Gao, Yixin Lian, Ziyi Zhou, Yuzhuo Fu, and Baoyuan Wang. Livechat: A large-scale personalized dialogue dataset automatically constructed from live streaming. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2023

  12. [20]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In2017 IEEE international conference on acoustics, speech and signal processing (I...

    2017

  13. [21]

    Gemini 3.1 pro model card

    Google DeepMind. Gemini 3.1 pro model card. Model card, 2026. URLhttps://deepmind. google/models/model-cards/gemini-3-1-pro/

    2026

  14. [22]

    Liveportrait: Efficient portrait animation with stitching and retargeting control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168, 2024

    2024 arXiv

  15. [23]

    Implicit identity representation conditioned memory compensation network for talking head video generation

    Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. InProceedings of the IEEE/CVF international conference on computer vision, pages 23062–23072, 2023

    2023

  16. [24]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, 2022

    2022

  17. [25]

    Jova: Unified multimodal learning for joint video-audio generation.arXiv preprint arXiv:2512.13677, 2025

    Xiaohu Huang, Hao Zhou, Qiangpeng Yang, Shilei Wen, and Kai Han. Jova: Unified multimodal learning for joint video-audio generation.arXiv preprint arXiv:2512.13677, 2025

    2025 arXiv

  18. [26]

    VBench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. I...

    2024

  19. [27]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  20. [28]

    Sentiavatar: Towards expressive and interactive digital humans.arXiv preprint arXiv:2604.02908, 2026

    Chuhao Jin, Rui Zhang, Qingzhe Gao, Haoyu Shi, Dayu Wu, Yichen Jiang, Yihan Wu, and Ruihua Song. Sentiavatar: Towards expressive and interactive digital humans.arXiv preprint arXiv:2604.02908, 2026

    2026 arXiv

  21. [29]

    LTX-2: Joint audio-video latent diffusion

    Lightricks. LTX-2: Joint audio-video latent diffusion. Technical report / model release, 2026. Version 2.3

    2026

  22. [30]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, Chao Liang, Yuan Zhang, and Jingtuo Liu. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13847–13858, 2025

    2025

  23. [31]

    Emage: Towards unified holistic co- speech gesture generation via expressive masked audio gesture modeling

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: Towards unified holistic co- speech gesture generation via expressive masked audio gesture modeling. InProceedings of the IEEE/CVF co...

    2024

  24. [32]

    Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization.arXiv preprint arXiv:2503.23377, 2025

    Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Jiebo Luo, Ziwei Liu, Hao Fei, et al. Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization.arXiv preprint arXiv:2503.23377, 2025

    2025

  25. [33]

    Elasticttt: Prior-preserving test-time tuning for video editing.arXiv preprint arXiv:2607.21529, 2026

    Yueyi Liu, Chi Zhang, Sen Cui, and Miao Liu. Elasticttt: Prior-preserving test-time tuning for video editing.arXiv preprint arXiv:2607.21529, 2026

    2026 arXiv

  26. [34]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In7th International Conference on Learning Representations, 2019

    2019

  27. [35]

    Talkingmachines: Real-time audio-driven facetime-style video via autoregressive diffusion models.arXiv preprint arXiv:2506.03099, 2025

    Chetwin Low and Weimin Wang. Talkingmachines: Real-time audio-driven facetime-style video via autoregressive diffusion models.arXiv preprint arXiv:2506.03099, 2025

    2025 arXiv

  28. [36]

    Ovi: Twin backbone cross-modal fusion for audio-video generation.arXiv preprint arXiv:2510.01284, 2025

    Chetwin Low, Weimin Wang, and Calder Katyal. Ovi: Twin backbone cross-modal fusion for audio-video generation.arXiv preprint arXiv:2510.01284, 2025

    2025 arXiv

  29. [37]

    Livebot: Generating live video comments based on visual and textual contexts

    Shuming Ma, Lei Cui, Damai Dai, Furu Wei, and Xu Sun. Livebot: Generating live video comments based on visual and textual contexts. InProceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6810–6817, 2019

    2019

  30. [38]

    Principal components analysis (pca).Computers & Geosciences, 19(3):303–342, 1993

    Andrzej Maćkiewicz and Waldemar Ratajczak. Principal components analysis (pca).Computers & Geosciences, 19(3):303–342, 1993

    1993

  31. [39]

    Mavid: A multimodal framework for audio-visual dialogue understanding and generation.arXiv preprint arXiv:2512.03034, 2025

    Youxin Pang, Jiajun Liu, Lingfeng Tan, Yong Zhang, Feng Gao, Xiang Deng, Zhuoliang Kang, Xiaoming Wei, and Yebin Liu. Mavid: A multimodal framework for audio-visual dialogue understanding and generation.arXiv preprint arXiv:2512.03034, 2025

    2025

  32. [40]

    Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  33. [41]

    Mm-sonate: Multimodal controllable audio-video generation with zero-shot voice cloning.arXiv preprint arXiv:2601.01568, 2026

    Chunyu Qiang, Jun Wang, Xiaopeng Wang, Kang Yin, Yuxin Guo, Xijuan Zeng, Nan Li, Zihan Li, Yuzhe Liang, Ziyu Zhang, et al. Mm-sonate: Multimodal controllable audio-video generation with zero-shot voice cloning.arXiv preprint arXiv:2601.01568, 2026

    2026

  34. [42]

    Sentence-bert: Sentence embeddings using siamese bert- networks.ArXiv, abs/1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks.ArXiv, abs/1908.10084, 2019. URL https://api.semanticscholar.org/CorpusID: 201646309

    1908 arXiv

  35. [43]

    Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation

    Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. InProceedings of the IEEE/CVF conference on computer vision and pattern re...

    2023

  36. [44]

    Seedance 1.5 pro: A native audio-visual joint generation foundation model.arXiv preprint arXiv:2512.13507, 2025

    Team Seedance, Heyi Chen, Siyan Chen, Xin Chen, Yanfei Chen, Ying Chen, Zhuo Chen, Feng Cheng, Tianheng Cheng, Xinqi Cheng, et al. Seedance 1.5 pro: A native audio-visual joint generation foundation model.arXiv preprint arXiv:2512.13507, 2025

    2025 arXiv

  37. [45]

    Seedance 2.0: Advancing video generation for world complexity.arXiv preprint arXiv:2604.14148, 2026

    Team Seedance, De Chen, Liyang Chen, Xin Chen, Ying Chen, Zhuo Chen, Zhuowei Chen, Feng Cheng, Tianheng Cheng, Yufeng Cheng, et al. Seedance 2.0: Advancing video generation for world complexity.arXiv preprint arXiv:2604.14148, 2026. Tsinghua University | 18

    2026 arXiv

  38. [46]

    Openai gpt-5 system card

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025

    2025 arXiv

  39. [47]

    Streamavatar: Streaming diffusion models for real-time interactive human avatars.arXiv preprint arXiv:2512.22065, 2025

    Zhiyao Sun, Ziqiao Peng, Yifeng Ma, Yi Chen, Zhengguang Zhou, Zixiang Zhou, Guozhen Zhang, Youliang Zhang, Yuan Zhou, Qinglin Lu, et al. Streamavatar: Streaming diffusion models for real-time interactive human avatars.arXiv preprint arXiv:2512.22065, 2025

    2025

  40. [48]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. InEuropean Conference on Computer Vision, pages 244–260. Springer, 2024

    2024

  41. [49]

    Meta Audiobox Aesthetics: Unified automatic assessment for speech, music and sound

    Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, Carleigh Wood, Ann Lee, and Wei-Ning Hsu. Meta Audiobox Aesthetics: Unified automatic assessment for speech, music and sound. InIEEE Automatic Spe...

    2025

  42. [50]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    2008

  43. [51]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  44. [52]

    Universe-1: Unified audio-video generation via stitching of experts.arXiv preprint arXiv:2509.06155, 2025

    Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. Universe-1: Unified audio-video generation via stitching of experts.arXiv preprint arXiv:2509.06155, 2025

    2025 arXiv

  45. [53]

    VGGT: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  46. [54]

    Av-dit: Effi- cient audio-visual diffusion transformer for joint audio and video generation.arXiv preprint arXiv:2406.07686, 2024

    Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, and Yapeng Tian. Av-dit: Effi- cient audio-visual diffusion transformer for joint audio and video generation.arXiv preprint arXiv:2406.07686, 2024

    2024 arXiv

  47. [55]

    Few-shot video-to-video synthesis.arXiv preprint arXiv:1910.12713, 2019

    Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis.arXiv preprint arXiv:1910.12713, 2019

    1910 arXiv

  48. [56]

    Videoic: A video interactive comments dataset and multimodal multitask learning for comments generation

    Weiying Wang, Jieting Chen, and Qin Jin. Videoic: A video interactive comments dataset and multimodal multitask learning for comments generation. InProceedings of the 28th ACM International Conference on Multimedia, pages 2599–2607, 2020

    2020

  49. [57]

    InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    2025 arXiv

  50. [58]

    Omnitalker: One-shot real-time text-driven talking audio-video generation with multimodal style mimicking.arXiv preprint arXiv:2504.02433, 2025

    Zhongjian Wang, Peng Zhang, Jinwei Qi, Guangyuan Wang, Chaonan Ji, Sheng Xu, Bang Zhang, and Liefeng Bo. Omnitalker: One-shot real-time text-driven talking audio-video generation with multimodal style mimicking.arXiv preprint arXiv:2504.02433, 2025

    2025 arXiv

  51. [59]

    X-streamer: Unified human world modeling with audiovisual interaction.arXiv preprint arXiv:2509.21574, 2025

    You Xie, Tianpei Gu, Zenan Li, Chenxu Zhang, Guoxian Song, Xiaochen Zhao, Chao Liang, Jianwen Jiang, Hongyi Xu, and Linjie Luo. X-streamer: Unified human world modeling with audiovisual interaction.arXiv preprint arXiv:2509.21574, 2025. Tsinghua University | 19

    2025

  52. [60]

    Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    2025 arXiv

  53. [61]

    Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Processing Systems, 37:660–684, 2024

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Processing Systems, 37:660–684, 2024

    2024

  54. [62]

    Hunyuanportrait: Implicit condition control for enhanced portrait animation

    Zunnan Xu, Zhentao Yu, Zixiang Zhou, Jun Zhou, Xiaoyu Jin, Fa-Ting Hong, Xiaozhong Ji, Junwei Zhu, Chengfei Cai, Shiyu Tang, et al. Hunyuanportrait: Implicit condition control for enhanced portrait animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2025

  55. [63]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  56. [64]

    Livestar: Live streaming assistant for real-world online video understanding.arXiv preprint arXiv:2511.05299, 2025

    Zhenyu Yang, Kairui Zhang, Yuhang Hu, Bing Wang, Shengsheng Qian, Bin Wen, Fan Yang, Tingting Gao, Weiming Dong, and Changsheng Xu. Livestar: Live streaming assistant for real-world online video understanding.arXiv preprint arXiv:2511.05299, 2025

    2025

  57. [65]

    Generating holistic 3d human motion from speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 469–480, 2023

    2023

  58. [66]

    MOVA: Towards scalable and synchronized video-audio generation.arXiv preprint arXiv:2602.08794, 2026

    Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, Wenming Tu, Xiangyu Peng, Yang Gao, Yanru Huo, Ying Zhu, Yinze Luo, Yiyang Zhang, Yuerong Song, Zhe Xu, Zhiyu Zhang, Chenchen Yang, Cheng Chang, Chushu Zhou, Hanfu Chen,...

    2026

  59. [67]

    Voicebridge: Designing latent bridge models for general speech restoration at scale.arXiv preprint arXiv:2509.25275, 2025

    Chi Zhang, Zehua Chen, Kaiwen Zheng, and Jun Zhu. Voicebridge: Designing latent bridge models for general speech restoration at scale.arXiv preprint arXiv:2509.25275, 2025

    2025

  60. [68]

    Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions.arXiv preprint arXiv:2511.03334, 2025

    Guozhen Zhang, Zixiang Zhou, Teng Hu, Ziqiao Peng, Youliang Zhang, Yi Chen, Yuan Zhou, Qinglin Lu, and Limin Wang. Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions.arXiv preprint arXiv:2511.03334, 2025

    2025

  61. [69]

    Towards multimodal empathetic response generation: A rich text-speech-vision avatar-based benchmark

    Han Zhang, Zixiang Meng, Meng Luo, Hong Han, Lizi Liao, Erik Cambria, and Hao Fei. Towards multimodal empathetic response generation: A rich text-speech-vision avatar-based benchmark. Proceedings of the ACM on Web Conference 2025, 2025. URLhttps://api.semanticscholar. org/Corp...

    2025

  62. [70]

    Towards multimodal empathetic response generation: A rich text-speech-vision avatar-based benchmark

    Han Zhang, Zixiang Meng, Meng Luo, Hong Han, Lizi Liao, Erik Cambria, and Hao Fei. Towards multimodal empathetic response generation: A rich text-speech-vision avatar-based benchmark. InProceedings of the ACM on Web Conference 2025, pages 2872–2881, 2025

    2025

  63. [71]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. InProceedings of the IEEE/CVF conference on computer vision and pat...

    2023

  64. [73]

    Speakervid-5m: A large-scale high-quality dataset for audio-visual dyadic interactive human generation.arXiv preprint arXiv:2507.09862, 2025

    Youliang Zhang, Zhaoyang Li, Duomin Wang, Jiahe Zhang, Deyu Zhou, Zixin Yin, Xili Dai, Gang Yu, and Xiu Li. Speakervid-5m: A large-scale high-quality dataset for audio-visual dyadic interactive human generation.arXiv preprint arXiv:2507.09862, 2025

    2025 arXiv

  65. [74]

    Talking face generation by adversarially disentangled audio-visual representation

    Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentangled audio-visual representation. InProceedings of the AAAI conference on artificial intelligence, volume 33, pages 9299–9306, 2019

    2019

  66. [75]

    Responsive listening head generation: a benchmark dataset and baseline

    Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: a benchmark dataset and baseline. InEuropean conference on computer vision, pages 124–142. Springer, 2022. Tsinghua University | 21 Contents 1 Introduction 1 2 Related ...

    2022

  67. [76]

    **Scene Description**: The physical environment, setting, lighting, background elements, and any visible objects or UI elements (game screens, chat overlays, etc.)

  68. [77]

    **People**: Describe the appearance, clothing, position, and distinctive features of anyone visible in the frame

  69. [78]

    **Actions & Gestures**: Movement, body language, hand gestures, and physical interactions with objects or other people

  70. [79]

    Only include actual spoken dialogue that can be inferred from the context

    **Speech**: What the person says, enclosed in square brackets [like this]. Only include actual spoken dialogue that can be inferred from the context

  71. [80]

    do you have a pattern for the cardigan in the back?

    **Reactions**: How the streamer responds to the chat comment - their emotional tone, engagement level, and any specific actions taken in response Write in present tense. Be specific and descriptive. Describe what you can see in the frame and what actions/movements occur. Place...

  72. [81]

    Um", "uh

    **Sentence Reconstruction (Strict Verbatim):** * **Merge & Split:** Combine fragmented subtitle lines into complete sentences based on punctuation and context. * **Strictly Verbatim:** Do NOT remove filler words (e.g., "Um", "uh"), stammers, or false starts. Preserve the text ...

  73. [82]

    End of sentence A. Start of sentence B

    **Timestamp Logic (Block Approximation):** * **Start Time:** Use the ‘start‘ timestamp of the *first* subtitle block that contains any part of this sentence. * **End Time:** Use the ‘end‘ timestamp of the *last* subtitle block that contains any part of this sentence. * *Note:*...

  74. [83]

    start":

    **Formatting:** * Output strictly valid JSON. * Schema: ‘[{"start": "string", "end": "string", "text": "string"}]‘ **Example:** *Input:* 1 00:00:01,000 --> 00:00:05,000 I ate the apple. Then I 2 00:00:05,000 --> 00:00:08,000 went to sleep. Um, yeah. *Output Logic:* * "I ate th...

  75. [85]

    I bought a dress

    **Identify** the specific chat message (bullet) that triggered the response, if the sentence is classified as ‘reactive_response‘. ### Input Data Structure * ‘target‘: The main sentence to classify. * ‘context_prev‘: The sentence spoken immediately before. * ‘candidates‘: A li...

  76. [88]

    meta": { ... },

    ‘reason‘: A brief explanation of why you chose the label and (if applicable) why you selected that specific candidate. **Example Output:** ‘‘‘json [ { "meta": { ... }, "target": { "text": "Hi Nina, glad you are here!" ... }, "context_prev": "I just started.", "candidates": [ ....

  77. [89]

    **Classify** the ‘target‘ sentence into one of three categories: ‘continuation‘, ‘ reactive_response‘, or ‘proactive_monologue‘

  78. [90]

    I bought a dress

    **Identify** the specific chat message (bullet) that triggered the response, if the sentence is classified as ‘reactive_response‘. ### Input Data Structure * ‘target‘: The main sentence to classify. * ‘context_prev‘: The sentence spoken immediately before. * ‘candidates‘: A li...

  79. [91]

    ‘label‘: The classification (‘continuation‘, ‘reactive_response‘, or ‘ proactive_monologue‘)

  80. [92]

    **Return ‘null**‘ if the label is not ‘reactive_response‘

    ‘trigger_bullet_index‘: The ‘bullet_index‘ of the candidate that triggered the response. **Return ‘null**‘ if the label is not ‘reactive_response‘

  81. [93]

    meta": { ... },

    ‘reason‘: A brief explanation of why you chose the label and (if applicable) why you selected that specific candidate. **Example Output:** ‘‘‘json [ { "meta": { ... }, "target": { "text": "Hi Nina, glad you are here!" ... }, "context_prev": "I just started.", "candidates": [ ....

  82. [94]

    If the primary language spoken is English, flag as ‘true‘

    **Language Detection:** * English Transcript: Evaluate the provided transcript text. If the primary language spoken is English, flag as ‘true‘. If it is in another language, or mixed with English not being the primary language, flag as ‘false‘

  83. [95]

    Explicitly ignore any humans appearing in a Picture-in-Picture (PiP) frame, webcam overlay, or corner streamer camera

    **Human Enumeration:** * Please analyze the video and count the number of humans based on the following strict criteria: * Main Window Only: Only count humans appearing in the primary content area (the main window). Explicitly ignore any humans appearing in a Picture-in-Pictur...

  84. [96]

    If the video is not a single, continuous, uncut shot, flag with ‘scene_cut: true‘

    **Scene & Layout Detection:** * Scene Cuts: Monitor the video for jump cuts, camera angle changes, or abrupt visual transitions to different footage. If the video is not a single, continuous, uncut shot, flag with ‘scene_cut: true‘. * Multiscreen/Split-Screen: Identify structu...

  85. [97]

    DIRECT": The video literally depicts what is being spoken about. The speaker simultaneously conducts an action that is related to their spoken response. (e.g., Bullet:

    **Content-Transcript Resonance:** * Analyze the semantic relationship between the *visuals* and the *transcript*, keeping in mind that the speaker is directly responding to the ‘Chat Bullet‘. * Assign a relevance rating based on these categories: * "DIRECT": The video literall...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.