{"id":"991300dd-b553-475e-86a9-e222883b8d12","arxiv_id":"2508.16188","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An audio-visual language model that adds full-face visual features to a pre-trained expressive speech model improves emotion recognition and expressive speech generation by a few F1 points over speech-only on synthetic IEMOCAP data.","lead":"The authors feed facial expression and jaw features from video into a speech-language model, alongside audio, and report better emotion recognition and more emotional generated speech than a speech-only model. The gains are a few F1 points, measured on synthetic data with an automated emotion classifier rather than human listeners.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generation gains in Table 7 are confounded with predicted-label conditioning and Step-Audio's synthetic style-to-label mapping; without gold-label/human evaluation, visual cues are not shown to directly improve expressive speech generation.","rationale":"The reader's CONDITIONAL verdict is appropriate. The single most load-bearing hole is not in the emotion-recognition numbers (visual features plausibly help there), but in the generation comparison, which is the part that supports the title's 'Seeing is Believing' claim for speech generation. The paper's protocol inserts the model's own predicted emotion label into the prompt, and the training/evaluation audio is generated by Step-Audio conditioned on the same label. This creates a direct path by which AVLM's better classifier (66.2 vs 61.3) could cause its better generation score (42.49 vs 38.39) without any direct visual-conditioning benefit. The paper's Fig. 5 shows that the label alone has limited controllability in zero-shot inference, so I do not claim the label fully explains the gap; but that very observation means the experiment does not quantify the visual prefix's marginal contribution. The check I propose (gold-label prompts + human evaluation) would settle it. This is consistent with the reader's weakest assumption about the synthetic pipeline; I add the specific mechanism. No verdict change: the paper should remain conditional on this evidence, with data release and human eval as conditions.","tokens_in":18623,"tokens_out":10130,"duration_ms":118691,"concrete_test":"Re-run the Table 7 generation comparison with the gold emotion label inserted into both AVLM and speech-only prompts (so classifier differences are removed), and evaluate the generated audio with human raters rather than Qwen2-Audio. If the AVLM advantage over speech-only disappears or fails a significance test (e.g., bootstrap CI on per-sample F1), the headline claim is not supported: the apparent gain is an artifact of label conditioning and Step-Audio style. This check should be run on the released checkpoints and code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that full-face visual cues improve expressive speech generation (abstract; Table 7, §5.3) rests on an uncontrolled comparison. At inference (Appendix D), the emotion label placed in the generation prompt is the output of each model's own classifier: AVLM (66.2 F1) vs speech-only (61.3 F1). Thus AVLM's generation prompt is conditioned on a better label, so its +4.1 F1 generation advantage (42.49 vs 38.39) may be produced by label accuracy rather than by the visual prefix conditioning the SpeechLM. This is amplified by the synthetic training pipeline (Appendix A): the target response audio was synthesized by Step-Audio-TTS-3B from the same emotion label and a cloned voice, so both training targets and the Qwen2-Audio evaluation encode Step-Audio's label-to-prosody mapping. A model that predicts the label correctly can reproduce that mapping; Qwen2-Audio may then recognize the synthetic style, not natural expressiveness. The paper's own controllability analysis (Fig. 5) shows the emotion label has some effect on generated emotion, and the Limitations section concedes there is no human evaluation and only a few hours of data. Consequently, the evidence does not yet establish that visual information directly improves expressive generation beyond improving emotion-label prediction in this synthetic setup.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Audio-Visual Language Model (AVLM) that integrates full-face visual features into a pre-trained expressive SpeechLM (SpiritLM) via a Q-Former prefix module. Training proceeds in two stages: (i) self-supervised pretraining on LRS3 with next-token prediction over speech, style, and pitch tokens, comparing visual encoders (SMIRK, VGG-Face2, MAGVIT2) and fusion strategies; and (ii) fine-tuning on IEMOCAP-derived synthetic expressive dialogues for emotion classification and expressive speech response generation. The reported results are: lower pretraining perplexity for the chosen prefix model (5.5 vs 5.6), improved AVSR WER under clean/noisy conditions (Table 5), higher emotion-recognition F1 (66.2 vs 61.3 in Table 7), and higher Qwen2-Audio-judged emotional F1 for generated speech (42.49 vs 38.39 in Table 7). The central claim is that visual cues improve both emotion recognition and emotional expressiveness of generated speech.","tokens_in":18974,"tokens_out":4177,"duration_ms":46924,"significance":"If the central claim were established, this would be a useful contribution: most expressive speech LMs are audio-only, and the paper systematically explores full-face visual features rather than lip-only features, with code released and several external benchmarks used for evaluation. The pretraining and AVSR results provide converging evidence that the visual prefix is learned and that it helps under acoustic degradation. The emotion-recognition gain on IEMOCAP is also consistent with prior audio-visual affect literature. However, the headline generation result — the part that would make this a foundation for emotionally intelligent conversational agents — currently rests on a confounded comparison and a synthetic training/evaluation loop. The paper itself notes the lack of human evaluation and the small amount of data. The contribution is therefore defensible in direction but not yet fully supported in its strongest form.","major_comments":[{"comment":"The generation comparison in Table 7 is confounded by the emotion-label conditioning. At inference, Appendix D states that the emotion label in the generation prompt is obtained from each model's own auxiliary classifier. The AVLM classifier is 4.9 F1 points better than the speech-only classifier (66.2 vs 61.3), so the AVLM generation prompt is conditioned on a more accurate label. The +4.1 F1 generation gain (42.49 vs 38.39) can therefore be explained by label accuracy rather than by the visual prefix. This is not a minor issue because Fig. 5 shows that the label influences the output. Please add a controlled experiment in which both models are conditioned on the same gold labels, or on the same fixed predicted labels, and report generation F1 separately. A human listening study is also needed; the Limitations section already concedes that no human evaluation is included.","section":"§5.3 / Appendix D, Table 7"},{"comment":"The synthetic training and evaluation loop makes the generation result hard to interpret. The target response audio is synthesized by Step-Audio-TTS-3B conditioned on the same IEMOCAP emotion label and a cloned speaker voice; these same synthetic responses are used as training targets and are scored by Qwen2-Audio. A model that predicts the emotion label accurately can imitate Step-Audio's label-to-prosody mapping, and Qwen2-Audio may recognize that synthetic style rather than natural expressiveness. Thus the reported gains do not yet show that full-face visual cues improve expressive speech generation beyond improving emotion-label prediction in this synthetic setting. Please evaluate on original IEMOCAP response audio or another natural expressive audio-visual dialogue set, and complement with human perceptual judgments.","section":"Appendix A / §5.3"},{"comment":"The main evaluation has no error bars, confidence intervals, or significance tests, and the test set is 10% of 4,859 synthetic pairs (about 486 utterances). A 4–5 F1 gap on this size can be within sampling noise, especially for four-way macro-F1 of generated speech judged by a third-party audio model. Relatedly, the Qwen2-Audio prompt in Appendix D allows 'Frustrated' as an output label, while the task and ground truth have only four classes (Angry and Frustrated are merged in Appendix A). The paper does not specify how 'Frustrated' predictions are mapped or excluded. Please report multiple seeds or bootstrap confidence intervals and define the label post-processing.","section":"Table 7 / Appendix D"}],"minor_comments":[{"comment":"The abstract says '+5 F1 in emotion recognition', but Table 7 reports 66.2 vs 61.3, which is +4.9. Use consistent rounding.","section":"Abstract / Table 7"},{"comment":"There is a typo: 'style/pitch tokens posii' should likely be 'positions'.","section":"§5.3"},{"comment":"The decoding procedure is described in prose. A short pseudocode or explicit transition table would make the constrained decoding reproducible.","section":"Appendix D"},{"comment":"The text says clips with average yaw angle exceeding '30%' are filtered; this should be '30 degrees'.","section":"Appendix C"},{"comment":"DIRECT CONCAT yields PPL 132.0, which is dramatically worse than the baselines; the paper attributes this to adaptation difficulty. A brief explanation of whether this is convergence failure or a structural issue would be helpful.","section":"§5.1, Table 3"},{"comment":"For Step-Audio synthesis, the exact style-control prompt is '[Emotion] [Text Response]'. It would be useful to state whether any additional style parameters (e.g., speed, pitch) were used, since the synthetic corpus is the basis for all fine-tuning.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a speech/audio-visual language modeling venue, and I see no citation-integrity concern beyond the unspecified handling of 'Frustrated' outputs in the Qwen2-Audio evaluation. The central direction is promising and the AVSR/recognition evidence gives some support for the visual-prefix design. However, the main generation claim is currently supported only by a confounded comparison. I would encourage a revision with (i) a gold-label-conditioned generation comparison, (ii) evaluation on natural audio-visual dialogues, and (iii) human listening results. With those additions, the paper could become a solid accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a solid empirical exploration of adding full-face visual features to a pre-trained expressive speech LM. The genuinely new piece is the systematic comparison of fusion strategies (direct concat, Q-Former infill, Q-Former prefix) and visual encoders for this task, plus the AVSR and emotion-recognition fine-tuning. The AVSR gains (3.50 vs 4.43 WER clean, with consistent improvement under noise and masking) are credible and suggest the pre-training truly aligns visual and speech representations. The emotion-recognition result in Table 1 (88.5 vs 73.1 F1 for speech-only) is a strong, independent motivation for the whole approach.\n\nThe soft spot is the headline generation claim in Table 7. At inference, the emotion label in the prompt is each model's own classifier output. The AVLM's classifier is about 5 F1 points better, so the generation comparison (42.5 vs 38.4) is not a controlled test of visual conditioning. Some or most of that gain could come from the better label, not the visual prefix. The paper's own controllability analysis shows the label affects generation, especially with in-context learning, so the label path is real. The right experiment is to condition both models on the same predicted or gold label, and ideally to add a human listening test. The synthetic Step-Audio pipeline also means Qwen2-Audio's emotion judgments partly reflect how well the model imitates Step-Audio's style-to-label mapping.\n\nThis is not fatal—the broader point that visual cues help emotion recognition is well supported—but the abstract overstates what the generation result shows. Minor issues: the main fine-tuning numbers come from a 486-pair test set with no error bars, and the pretraining PPL difference (5.5 vs 5.6) is negligible, though the masking robustness results are more informative. The authors are transparent about limitations, including no human evaluation and small data, and they release code.\n\nThis deserves peer review, not desk rejection. The AVSR and emotion-recognition results are worth having, and the generation claim is fixable with a controlled inference setup and human evaluation. I'd bring it to a reading group to discuss evaluation design in generative speech modeling.","headline":"The AVSR and emotion-recognition results are solid, but the headline generation claim is confounded by each model using its own predicted emotion label; the fix is a controlled comparison.","tokens_in":19455,"tokens_out":2376,"would_cite":true,"duration_ms":25436,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Full-face video cues make emotion-aware speech generation measurably better than audio alone.","keywords":["audio-visual language model","expressive speech generation","emotion recognition","multimodal fusion","Q-Former","facial expression features","speech language model","IEMOCAP"],"falsifier":"Fine-tune the same two models on real, non-synthetic expressive video dialogue and have human listeners label the emotion of the generated responses; if the audio-visual model does not beat the speech-only model on human-rated emotion F1, the central claim is not supported. A cheaper check: replace the video with static frames at evaluation; if the emotion-recognition F1 gain persists, the model is not using facial dynamics.","tokens_in":18536,"feed_emoji":"🎭","tokens_out":7907,"duration_ms":80816,"temperature":0.7,"pith_summary":"The paper tries to establish that a speech language model can use full-face visual cues to become more emotionally accurate, both at recognizing emotion and at generating spoken responses that carry the intended emotion. It builds on a pre-trained expressive speech model, injects face-derived features through a learned fusion module during pre-training, then fine-tunes on emotion recognition and expressive dialogue. Across both tasks, the audio-visual model beats the same model without vision, with the central reported gains being +4.9 macro F1 in emotion recognition and +4.1 macro F1 in the emotion of generated speech as judged by an automated audio model. If true, the result means vision is not a redundant channel for expressive speech: facial expression supplies information the voice alone does not.","feed_headline":"Face cues lift emotion-aware speech generation by 4 F1","feed_subtitle":"Audio-visual model beats speech-only baselines on emotion recognition and on emotion carried by generated speech.","key_machinery":"The central mechanism is the Q-Former Prefix fusion: query latents attend to visual features through cross-attention and are prepended to the SpeechLM's speech-token sequence, so every generation step can condition on compressed visual information. The visual stream is encoded by SMIRK, a lightweight 3D face-reconstruction model whose expressive and jaw parameters are used instead of raw pixels or lip crops. The speech stream is tokenized into interleaved semantic, style, and pitch tokens by the base expressive SpeechLM's tokenizer, and LoRA updates adapt the model during pre-training and fine-tuning. An auxiliary emotion classifier over visual-query and style/pitch hidden states provides th","core_discovery":"The paper claims that conditioning a pre-trained expressive speech language model on full-face visual features, and not just lip-region video, improves both the model's ability to identify a speaker's emotion and its ability to produce spoken responses whose emotion matches the context. The recipe is two-stage: first pre-train a Q-Former-style fusion module that compresses visual frames into query latents prepended to speech tokens, using next-token prediction on unlabeled video; then fine-tune on emotion classification and expressive dialogue built from IEMOCAP, with an auxiliary emotion classifier that supplies the generation prompt's emotion label at inference. In the head-to-head compari","pith_inferences":["A human listening test on held-out real dialogue would be the sharper check on the expressiveness gain; a single automated audio judge could share systematic biases with the TTS used for training data.","Because the visual features are full-face rather than lip-cropped, the gain could come from expression reading rather than articulation; replacing video with static frames during evaluation would separate those two sources.","The weak zero-shot controllability suggests the model anchors emotion to the input speaker's face and voice; training data that decouples input and response emotion could make label-based style control work without in-context examples.","If the visual benefit transfers to noisy or bandwidth-limited audio, the same architecture could improve emotion-aware speech systems where the audio channel is degraded but a webcam feed is available."],"forward_implications":["Emotion-aware conversational agents can condition spoken responses on the speaker's face, not just the voice, improving emotional alignment without retraining a speech model from scratch.","Video-conditioned pre-training also helps audio-visual speech recognition, lowering word error rate under noise and speech-token masking compared with the speech-only model.","The Q-Former Prefix with 3D facial-expression features is a transferable recipe for adding vision to an existing speech language model while keeping next-token prediction as the objective.","The emotion label alone is a weak control: with in-context demonstrations the model shifts generated emotion, but without them it follows the audio-visual input.","Expressive dialogue data can be synthetically expanded from small acted corpora by rewriting short turns and synthesizing longer responses with voice cloning."],"supporting_citations":[{"why":"Supplies the pre-trained expressive SpeechLM and its semantic/style/pitch tokenizer that the AVLM extends.","marker":"Nguyen et al., 2024"},{"why":"Supplies IEMOCAP, the acted dyadic video dataset whose emotion labels and conversations anchor the synthetic fine-tuning data.","marker":"Busso et al., 2008"},{"why":"Supplies LRS3, the large unlabeled audio-visual corpus used for AVLM pre-training and for the AVSR fine-tuning evaluation.","marker":"Afouras et al., 2018"},{"why":"Supplies SMIRK, the 3D face-reconstruction encoder whose expressive and jaw features carry the visual signal.","marker":"Retsinas et al., 2024"},{"why":"Supplies Step-Audio, the expressive TTS with voice cloning used to synthesize response audio for fine-tuning.","marker":"Huang et al., 2025"},{"why":"Supplies the language model used to rewrite short IEMOCAP responses into longer dialogue turns.","marker":"OpenAI et al., 2024"},{"why":"Supplies the Q-Former cross-attention query-latent design adopted for visual-to-speech fusion.","marker":"Li et al., 2023"},{"why":"Supplies the automated audio model used to evaluate the emotion of generated speech in the head-to-head comparison.","marker":"Chu et al., 2024"},{"why":"Supplies the audio-visual masked-prediction motivation behind the infill-style fusion variant.","marker":"Shi et al., 2022"},{"why":"Supplies the IEMOCAP data-split protocol and audio-encoder recipe for the motivating emotion-recognition comparison.","marker":"Ma et al., 2024b"}],"fun_headline_variants":["Full-face video boosts emotion-aware speech generation by 5 F1","Seeing emotion: Adding face video improves speech generation by 5 F1","Visual context: Full-face frames help speech AI match emotion, +5 F1","Beyond lip-reading: Full-face cues lift expressive speech AI by 5 F1","Emotion-aware speech gets a face lift: +5 F1 with full-face video"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported gains rest on the premise that the synthetic IEMOCAP-derived dialogues and the automated emotion judgments on generated audio behave like real expressive human conversation; if the model is only mimicking the TTS voice it was trained on, the visual advantage may not appear with real speakers.","fun_headline_variants_meta":{"raw":{"variants":["Full-face video boosts emotion-aware speech generation by 5 F1","Seeing emotion: Adding face video improves speech generation by 5 F1","Visual context: Full-face frames help speech AI match emotion, +5 F1","Beyond lip-reading: Full-face cues lift expressive speech AI by 5 F1","Emotion-aware speech gets a face lift: +5 F1 with full-face video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000977,"raw_usage":{"total_tokens":3914,"prompt_tokens":602,"completion_tokens":3312,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":346,"completion_tokens_details":{"reasoning_tokens":3210}},"tokens_in":346,"tokens_out":3312,"duration_ms":24691,"temperature":1.0,"reasoning_tokens":3210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:27:54.009290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same two models on real, non-synthetic expressive video dialogue and have human listeners label the emotion of the generated responses; if the audio-visual model does not beat the speech-only model on human-rated emotion F1, the central claim is not supported. A cheaper check: replace the video with static frames at evaluation; if the emotion-recognition F1 gain persists, the model is not using facial dynamics.","supporting_citations":[{"cited_title":"Chang, Sungbok Lee, and Shrikanth S","cited_arxiv_id":null,"evidence_quote":"Supplies IEMOCAP, the acted dyadic video dataset whose emotion labels and conversations anchor the synthetic fine-tuning data."},{"cited_title":"Filntisis, Radek Danecek, Victoria F","cited_arxiv_id":null,"evidence_quote":"Supplies SMIRK, the 3D face-reconstruction encoder whose expressive and jaw features carry the visual signal."}],"review_version":1}