{"id":"2362d914-d335-4371-9368-f51128c6ca89","arxiv_id":"2504.21718","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"VividListener uses a Diffusion Transformer with a Responsive Interaction Module and Emotional Intensity Tags to generate controllable listener head dynamics over long multi-turn dialogues, trained on the new ListenerX dataset of 1.4M frames.","lead":"The paper introduces a large dataset of 3D dyadic conversations and a diffusion-transformer framework that generates listener head motions conditioned on text descriptions and emotional intensity tags. It claims fine-grained, controllable listener reactions over eight-second sequences, beating prior short-term baselines on its own benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed fine-grained controllability is unsupported: conditioning annotations are extracted from the target motions, and all quantitative metrics compare to those same targets; no independent text-alignment or intensity-control metric is reported.","rationale":"The reader's weakest assumption about annotation trustworthiness is related, but I believe the deeper issue is the evaluation design: even if every text and tag were perfectly accurate, the quantitative results would still not substantiate controllability because the conditions are derived from the very targets the metrics compare against. The 25.21% FD reduction in cross-scenario inference (Tab. 2) is impressive but expected when the model receives a privileged description of the target that baselines lack. The model's value depends on whether a user's arbitrary text and intensity tag actually steer the generated motion; that is only checked qualitatively (Fig. 4) and in a user study that does not ask about condition alignment (Fig. 5). The objective function's emotional loss (Eq. 5) reinforces the same circularity if the predictor is the annotator. These issues are testable with a text-swap and tag-scaling experiment using independent raters or a separate estimator, as described in concrete_test. If the model passes such a test, the controllability claim is credible; if not, the paper's central novelty is unsupported. The absence of released code/data and error bars further limits verification, but the circular evaluation is the load-bearing problem. Therefore I concur with the CONDITIONAL verdict, unchanged, because the paper may still be correct but requires additional evidence.","tokens_in":14390,"tokens_out":5949,"duration_ms":60998,"concrete_test":"Run a text-swap experiment: for fixed speaker inputs, pair two different GT listener motions with different text descriptions; generate outputs with swapped texts while keeping tags fixed; have raters (or a CLIP-based retriever) match outputs to texts. If accuracy is at chance, text is not controlling. Separately, vary intensity tags monotonically over [-1,1] for fixed text and speaker input; estimate V/A of outputs with an independent estimator (not the one used for annotation) and check for a monotonic relationship. Report error bars; if no monotonic trend, EIT control is unsubstantiated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is fine-grained, controllable listener dynamics (Sec. 1). The evidence for it is circular in two ways. (1) Text descriptions are generated from the listener's own facial frames via an action-unit detector and VLM, then manually revised (Sec. 3.2). Intensity tags are continuous valence/arousal values from an automatic estimator applied to the same frames (Sec. 3.2, ref [50]). Thus the conditioning signals are derived from the target motions. The main quantitative metrics—FD, P-FD, MSE—compare the generated motion to that same ground truth (Sec. 5.1, Tab. 2). A model that learns to invert the text/tag-to-motion mapping will trivially score better than baselines without those inputs; this does not demonstrate that a user-specified open-vocabulary description or a requested intensity level is followed. (2) The emotional intensity loss (Eq. 5) uses a pre-trained predictor P_emotional; if this is the same estimator used to create the tags, the model is trained to satisfy that estimator's biases, not human perception. The ablations (Tab. 4) only show that removing text or tags hurts reconstruction; they do not test whether the model uses them correctly. The user study (Fig. 5) rates similarity, interactivity, and diversity, not alignment with input text or tags. The only evidence of actual controllability is qualitative (Fig. 4). This is the load-bearing gap: without an independent measure of semantic alignment and intensity scaling, the headline claim of 'fine-grained controllable' listener dynamics is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses expressive and controllable listener head-dynamics generation in dyadic conversation. It contributes ListenerX, a new dataset of roughly 1.4M frames of 3D dyadic conversations with FLAME parameters, fine-grained textual descriptions (generated with an action-unit detector plus a vision-language model and then manually revised), and continuous valence/arousal emotion-intensity tags. The proposed VividListener system is a diffusion-transformer model conditioned on speaker motion, speaker audio, listener text descriptions, and emotional intensity tags; it includes a Responsive Interaction Module for multimodal fusion and an Emotional Intensity Tagging mechanism for intensity modulation. Experiments on ListenerX and on a cross-scenario DailyX-to-InterviewX setting compare against L2L, RLHG, and DIM, with ablations and a user study, and report state-of-the-art results.","tokens_in":14707,"tokens_out":4641,"duration_ms":49421,"significance":"If the controllability claims were independently verified, the paper would make a solid contribution: ListenerX appears to be the largest long-sequence listener-dynamics dataset with fine-grained text and intensity annotations, and the proposed diffusion-transformer design with adaptive multimodal fusion is reasonable and clearly better than the retrained baselines on the reported metrics. The dataset alone could be valuable to the community. However, the central claim of fine-grained, semantically controllable listener generation is currently supported only by evidence that is partly circular: the conditioning signals are derived from the same ground-truth listener motions used for both training and evaluation, and the quantitative metrics compare against those same targets. Unless the authors add independent controllability evaluations and annotation-quality validation, the headline claim remains plausible but not established.","major_comments":[{"comment":"The conditioning and evaluation loop is circular. The text descriptions are produced from the listener's own facial frames via an AU detector plus VLM, then manually revised, and the intensity tags are computed by an automatic V/A estimator applied to the same frames (Sec. 3.2). All main quantitative metrics (FD, P-FD, MSE) compare generated motion to that same ground-truth motion (Table 2). A model that receives target-derived conditions can therefore improve reconstruction over baselines that do not receive them, without showing that a user-specified open-vocabulary description or a requested intensity level is actually followed. I request independent tests of controllability: (i) a condition-swap experiment in which text or intensity tags are changed while speaker input is fixed, with a check that the output changes in the intended direction; (ii) a text-to-motion alignment metric computed on held-out conditions that are not derived from the target listener motion; and (iii) an intensity-scaling test in which the tag is varied and the predicted V/A of the generated output is measured. Without such tests, the fine-grained control claim is not supported by the current experiments.","section":"Sec. 3.2, Sec. 5.1, Table 2"},{"comment":"The emotional intensity loss in Eq. (5) uses a pre-trained predictor P_emotional on the generated motion. If this predictor is the same estimator used in Sec. 3.2 to create the intensity tags, then the model is trained to reproduce the estimator's own biases, and the quantitative intensity-control evidence becomes self-referential. The paper never identifies P_emotional or states whether it is identical to the annotation model [50], so this needs to be clarified. If the same model is used, the authors should provide human validation of intensity alignment or use an independent predictor. Relatedly, Table 3 refers to an 'Emotional Identity Predictor (EIP)' while Sec. 4.3 and Eq. (5) describe an 'Emotional Intensity Predictor'; this naming inconsistency should be fixed.","section":"Sec. 4.3, Eq. (5), Table 3"},{"comment":"The headline cross-scenario claim is not accurately quantified or statistically supported. The text says the method achieves a 'reduction in FD by 25.21%', but 41.510 - 16.309 = 25.201 is an absolute FD difference, not a percentage reduction (the relative reduction is about 60.7%). Also, no error bars, confidence intervals, or significance tests are provided for any metric, and the cross-scenario experiment uses a single train/test split. Given that the conditions include text and intensity tags extracted from the test-set listener motions, the comparison is potentially unfair to the baselines, which do not receive those target-derived signals. The authors should report variability across seeds, run a significance test, and ideally include a baseline that also receives the same text/tag inputs, to isolate the benefit of the proposed architecture from the benefit of target-derived conditioning information.","section":"Sec. 5.2, Table 2"},{"comment":"The ablations and user study do not test semantic alignment with the input text or intensity levels. Table 4 shows that removing text or tags degrades FD, SID, Var, and rPCC, but those metrics measure reconstruction quality or diversity, not whether the model uses the text semantically or scales intensity appropriately. A model could overfit to arbitrary condition codes and still show such degradation. The user study in Fig. 5 asks raters to judge similarity, interactivity, and diversity; none of the questions asks whether the generated motion matches the supplied text description or the requested emotional intensity. I recommend adding a human evaluation with forced-choice questions such as 'which video better matches the text description' and 'which video has stronger emotional intensity,' and reporting an automated alignment metric (e.g., CLIP-based similarity between the description and rendered generated frames).","section":"Sec. 5.2, Sec. 5.3, Table 4, Fig. 5"}],"minor_comments":[{"comment":"In the related work, the citation '38 ?' appears in an incomplete form; the reference list should be cleaned up.","section":"Sec. 2.2, References"},{"comment":"The header row 'CV P R' and entries such as 'ListenerX(Ours 2025)' are unclear or inconsistently formatted; the table would benefit from a caption defining abbreviations and a cleaner layout.","section":"Table 1"},{"comment":"Please define the transposition notation E_text' explicitly and state the dimensions of W_fuse, E_fused, and E_text; currently the Hadamard product between E_fused and the pooled weight vector is under-specified.","section":"Eq. (2)"},{"comment":"The cross-references are inconsistent: Sec. 5.3 states that Fig. 5 illustrates conditional control results, but Fig. 5 is the user-study figure, whereas Fig. 4 shows conditional control examples. Please correct the references.","section":"Sec. 5.3 and Fig. 4/Fig. 5"},{"comment":"The phrase 'sub-optimal counterparts' should identify which baseline is considered sub-optimal and define how the reduction percentage is computed.","section":"Sec. 5.2, Table 2"},{"comment":"The term 'authoritative emotional intensity tags' is overstated, since the tags come from an automatic single-model estimator; the paper should acknowledge this and provide some validation, such as correlation with human V/A ratings or inter-annotator agreement.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the self-referential conditioning and evaluation loop. If the authors can add independent controllability metrics and annotation-quality validation, the paper could become a strong contribution; the dataset alone is a useful resource. The current form does not yet substantiate the fine-grained semantic control claim, but the issues are addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know about this one: the ListenerX dataset (1.4M frames of paired speaker-listener 3D head motion with text and V/A tags) is the actual contribution. The method—a DiT with a responsive interaction module and intensity tags—is a sensible combination of known pieces, and the paper is clearly written. If you work on listener/social-signal generation, the dataset alone is worth a look.\n\nThe problem is that the evidence for the central claim of \"fine-grained controllable\" generation is weaker than the paper suggests. The text descriptions are generated from the listener's own frames via an action-unit detector + VLM, and the intensity tags come from an automatic V/A estimator on those same frames. So the conditioning signals are functions of the target. All quantitative metrics (FD, P-FD, MSE) compare generated motion to that same ground truth. A model given text and tags extracted from the target will naturally beat baselines that don't get those inputs—that shows the model can compress/invert the labels, not that a user can specify an open-vocabulary description or an intensity level and have it followed. The emotional intensity loss (Eq. 5) uses a pre-trained predictor; if it is the same estimator used to create the tags, it's training to that estimator's biases. The ablations (Tab. 4) only show that removing text or tags hurts reconstruction, which again is consistent with the model fitting annotations rather than using them semantically. The user study rates similarity, interactivity, diversity, not text alignment or intensity scaling. So the actual evidence for controllability is the qualitative examples in Fig. 4. That's a real gap.\n\nThere are also minor issues: no error bars or significance tests, no annotation-quality analysis (e.g., inter-annotator agreement for the manual text revisions), and no released code/data. The cross-scenario FD drop is large (41.5 to 16.3 for RLHG to theirs) and not really explained—it could be a distribution artifact or the retrained baselines being suboptimally tuned.\n\nThe honest summary: the dataset is new and the task framing is useful; the method is plausible; the controllability evidence is not yet there. This deserves review rather than desk reject, because the dataset plus a good controllability evaluation could be a solid paper. For a revision, I'd want code/data release, an independent metric of text-to-motion semantic alignment, a demonstration that intensity tags scale motion amplitude monotonically, and annotation reliability numbers.\n\nVerdict: engage, but expect a major revision.","headline":"A potentially useful dataset and a plausible method, but the headline claim of fine-grained control is supported only by qualitative examples because the conditioning labels come from the target motions.","tokens_in":15228,"tokens_out":1861,"would_cite":false,"duration_ms":18872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VividListener proposes that long-sequence listener head dynamics can be generated with fine-grained control from text descriptions and continuous emotional-intensity tags, and supports this with a new 1.4M-frame dyadic-conversation dataset.","keywords":["listener dynamics","dyadic conversation","3D head motion generation","diffusion transformer","emotion intensity control","multi-modal conditioning","text-guided animation","valence-arousal"],"falsifier":"Take a fixed speaker clip and a fixed text description, then sweep the valence/arousal tags from their minimum to maximum while keeping everything else constant. If a held-out facial behavior recognizer does not show a monotonic increase in measured arousal or expression amplitude across the sweep, the claimed emotional-intensity control is not real. Independently, have two teams of human annotators re-describe a random sample of ListenerX clips and rate their intensity; low inter-annotator agreement on either would undermine the ground truth that both training and evaluation rely on.","tokens_in":14226,"feed_emoji":"🎭","tokens_out":7552,"duration_ms":69589,"temperature":0.7,"pith_summary":"The paper tries to establish that listener head dynamics—the nodding, blinking, smiling, and other non-verbal reactions of the person who is not speaking—can be generated for long, multi-turn dialogues under fine-grained control. To do this, it assembles ListenerX, a dataset of 3D dyadic conversations with more than 1.4 million frames, each annotated with text descriptions of facial expression changes and continuous valence/arousal emotion-intensity tags. It then proposes VividListener, a diffusion-transformer generator that takes the speaker's motion and audio, the listener's text description, and intensity tags as conditions, and outputs an 8-second, 240-frame sequence of listener head motion. If correct, this would let virtual avatars display expressive, responsive reactions that shift with described emotion and with requested intensity, going beyond previous short-term, category-labeled listener models.","feed_headline":"Text and emotion tags now steer 3D avatar listeners","feed_subtitle":"A 1.4M-frame dyadic dataset plus a diffusion model brings fine-grained, intensity-controlled reactions to avatars.","key_machinery":"The load-bearing components are the Responsive Interaction Module (RIM) and the Emotional Intensity Tags (EIT). RIM fuses speaker motion and audio via bidirectional multi-head attention, encodes listener text with a frozen vision-language text encoder, then computes a temporal similarity matrix between text embeddings and speaker features; adaptive pooling of this matrix produces blending weights that modulate the speaker features so the listener motion follows the described expression changes while keeping rhythm with the speaker. EIT injects continuous valence/arousal tags in two places: multiplied with the fused features as input, and used as the query in a cross-attention layer whose output is passed through a convolutional layer and an adaptive instance normalization layer to modulate intermediate listener motion features. Both feed a Diffusion Transformer denoiser trained with a denoising objective, an emotional-intensity prediction loss, and a velocity smoothness loss.","core_discovery":"The central claim is that expressive, controllable listener dynamics for long dialogue sequences can be treated as a multi-modal conditioning problem and solved with a Diffusion Transformer, provided the missing ingredient—paired data with fine-grained annotations—is built at scale. On the collected ListenerX corpus, VividListener reports superior quantitative results across realism, synchrony, diversity, and accuracy metrics, including a 25.21% reduction in Fréchet distance over the best baseline under cross-scenario training. Ablations show that removing the text input, the intensity tags, the temporal-semantic interaction module, or the emotional-control layer degrades performance. The work also demonstrates conditional control with open-vocabulary descriptions and adjustable intensity values, and shows multi-turn role-switching results.","pith_inferences":["One consequence the paper leaves implicit: if released with its text and valence/arousal annotations, ListenerX could serve as a benchmark for other controllable generation tasks, including speaker-side generation, because it provides paired audio and 3D motion for both roles.","A testable extension would be to hold the speaker clip and text description fixed while sweeping the intensity tags across their range, then measure how monotonically the generated facial expression amplitude responds, isolating intensity control from speaker-imitation effects.","The metric design leaves room for a stronger controllability check: the reported rPCC and diversity metrics do not directly verify that the generated expression semantically matches the input text, so a separately trained text-expression alignment measure would make the claim easier to assess."],"forward_implications":["Listener head motion can be generated in 8-second, 240-frame sequences while following text descriptions that describe emotion transitions, not just static labels.","Continuous valence/arousal tags provide a controllable dial for emotional intensity, so the same dialogue can be rendered calmer or more excited by changing tags.","Cross-scenario generalization improves dramatically: trained on casual daily conversations and tested on structured interviews, the method reports a 25.21% lower Fréchet distance than the sub-optimal baseline.","The ListenerX dataset, with 6,683 videos, 1.4M frames, text descriptions, and intensity tags, supports long-sequence multi-turn listener modeling for downstream avatar and interaction systems.","Ablations show that each condition matters: removing text or tags degrades semantic alignment and diversity, and removing the temporal-semantic interaction or emotional-control components lowers realism and control."],"supporting_citations":[{"why":"Supplies the monocular 3D facial capture method used to obtain 3D head parameters for both conversation participants.","marker":"[9]"},{"why":"Provides the action-unit detector whose outputs prompt the vision-language model for accurate text descriptions.","marker":"[4]"},{"why":"The vision-language model used to generate fine-grained facial expression descriptions for listener frames.","marker":"[22]"},{"why":"The facial-affect analysis method that outputs continuous valence and arousal values used as emotional intensity tags.","marker":"[50]"},{"why":"The pre-trained text encoder that turns listener descriptions into sequence-aware semantic embeddings in the framework.","marker":"[37]"},{"why":"The Diffusion Transformer architecture that serves as the generative backbone for listener head motion sequences.","marker":"[31]"},{"why":"Defines the L2L baseline and dataset protocol that VividListener compares against and extends to long sequences.","marker":"[27]"},{"why":"Provides the RLHG baseline for responsive listening head generation, used as a state-of-the-art competitor.","marker":"[59]"},{"why":"Provides the DIM dyadic interaction baseline against which VividListener is compared.","marker":"[51]"},{"why":"Introduces the velocity loss used for motion smoothness supervision in the training objective.","marker":"[57]"}],"fun_headline_variants":["VividListener: control listener dynamics with text and emotion tags","1.4M-frame dyadic dataset powers controllable listener dynamics","Text and emotion tags now steer 3D avatar listener reactions","VividListener: expressive listener dynamics with multimodal control","New dataset and model for controllable avatar listener dynamics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole controllability argument rests on the trustworthiness of the ListenerX annotations: text descriptions produced by a vision-language model prompted with action units and then manually revised, and continuous emotion tags (pleasantness and excitement values) produced by an automatic facial-affect estimator. If those annotations are noisy or inconsistent, the learned mapping from text and tags to motion is compromised, and because the evaluation metrics use those same annotations as conditioning information, they do not independently verify the claimed control.","fun_headline_variants_meta":{"raw":{"variants":["VividListener: control listener dynamics with text and emotion tags","1.4M-frame dyadic dataset powers controllable listener dynamics","Text and emotion tags now steer 3D avatar listener reactions","VividListener: expressive listener dynamics with multimodal control","New dataset and model for controllable avatar listener dynamics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3569,"prompt_tokens":968,"completion_tokens":2601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":2520}},"tokens_in":584,"tokens_out":2601,"duration_ms":21551,"temperature":1.0,"reasoning_tokens":2520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:54:59.048783+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed speaker clip and a fixed text description, then sweep the valence/arousal tags from their minimum to maximum while keeping everything else constant. If a held-out facial behavior recognizer does not show a monotonic increase in measured arousal or expression amplitude across the sweep, the claimed emotional-intensity control is not real. Independently, have two teams of human annotators re-describe a random sample of ListenerX clips and rate their intensity; low inter-annotator agreement on either would undermine the ground truth that both training and evaluation rely on.","supporting_citations":[{"cited_title":"Emoca: Emotion driven monocular face capture and animation","cited_arxiv_id":null,"evidence_quote":"Supplies the monocular 3D facial capture method used to obtain 3D head parameters for both conversation participants."},{"cited_title":"Libreface: An open-source toolkit for deep facial expression analysis","cited_arxiv_id":null,"evidence_quote":"Provides the action-unit detector whose outputs prompt the vision-language model for accurate text descriptions."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"The vision-language model used to generate fine-grained facial expression descriptions for listener frames."},{"cited_title":"Estimation of continuous va- lence and arousal levels from faces in naturalistic conditions","cited_arxiv_id":null,"evidence_quote":"The facial-affect analysis method that outputs continuous valence and arousal values used as emotional intensity tags."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"The pre-trained text encoder that turns listener descriptions into sequence-aware semantic embeddings in the framework."},{"cited_title":"Learning to listen: Modeling non-deterministic dyadic facial motion","cited_arxiv_id":null,"evidence_quote":"Defines the L2L baseline and dataset protocol that VividListener compares against and extends to long sequences."},{"cited_title":"Responsive listening head generation: a benchmark dataset and baseline","cited_arxiv_id":null,"evidence_quote":"Provides the RLHG baseline for responsive listening head generation, used as a state-of-the-art competitor."},{"cited_title":"Dim: Dyadic interaction modeling for social be- havior generation","cited_arxiv_id":null,"evidence_quote":"Provides the DIM dyadic interaction baseline against which VividListener is compared."}],"review_version":1}