{"id":"74abe2db-cab3-4d38-ac98-453a3e1dc162","arxiv_id":"2501.10937","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Listen, Perceive, Express (LPE) uses two-stage ASR/SER training plus chain-of-thought prompting to let a frozen LLM generate empathetic responses from speech without question-answer fine-tuning data.","lead":"This paper trains a speech-language model to transcribe and read emotion from speech, then uses chain-of-thought prompting to make it respond empathetically, without any question-answer training data. The method, called LPE, may reduce the cost of building empathetic voice assistants.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoT prompt was selected by iterative refinement on the test set; since the empathetic response capability is entirely prompt-dependent, the headline 'outperforms' result may be an artifact of test-set prompt tuning.","rationale":"The reader's weakest_assumption focused on whether a frozen LLM trained only on transcription/emotion templates would switch to empathetic response generation under a CoT prompt, plus the GPT-4o scoring/reference issue. Both are real. However, the most load-bearing and falsifiable concern is that the CoT prompt itself was selected by iterative refinement on the test set (Section II-C), and the entire expressing capability is prompt-dependent (Table IV shows Non-CoT essentially fails). This introduces selection bias into the headline comparison, and it is unique to LPE because the cascade baseline is forced to use the same prompt. The concern is concrete, precisely located, and directly testable by a held-out prompt-validation split. It does not invalidate the engineering contribution, but it means the reported empirical superiority is not yet established; the reader's conditional verdict already captures this, so I recommend no change. Agreement is partial because the reader mentioned prompt tuning in the rationale but did not make it the primary weakest assumption. The concrete test I propose would settle whether the concern lands: it reuses the paper's own prompt-selection protocol but confines it to a disjoint subset and re-measures all comparisons on untouched data.","tokens_in":9141,"tokens_out":3051,"duration_ms":35206,"concrete_test":"Split each test set (IEMOCAP session 5 and AlpacaTTS) into a prompt-development half and a held-out evaluation half. Perform the iterative prompt refinement of Section II-C using only the prompt-development half; freeze the selected prompt; then recompute Table II/III/IV on the held-out evaluation half for LPE, SALMONN, Qwen-Audio-Chat, and the cascade (giving the cascade its own prompt optimized on the prompt-development half, not LPE's prompt). If LPE's advantage over baselines shrinks or reverses on the held-out half, the reported superiority was an artifact of test-set prompt selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LPE outperforms other speech-based LLMs rests on the expressing capability elicited by a hand-crafted CoT prompt. Section II-C states: 'We evaluate each prompt on several samples of test set using our two-stage trained model. By iteratively refining the prompts, we identify the most effective prompt for each CoT type, leading to the results in Section IV-C.' The final prompt chosen by this test-set refinement is then used for the headline comparisons in Table III and Table IV. Table IV itself shows the model without CoT emits transcriptions or emotion labels (Content 2.14, Empathy 1.35), so the entire empathetic expression behavior is a function of the prompt. Selecting the prompt on the same test set used to report the superiority comparison means the reported advantage (e.g., LPE 3.99 vs SALMONN 2.12 in Content) may reflect test-set overfitting in prompt selection rather than a general capability of the method. This is compounded by Section III.C, which gives the cascade baseline the same CoT prompt optimized on LPE, rather than a prompt optimized for the cascade, further biasing the comparison. Thus the claim that LPE 'outperforms all other speech-based LLMs in all three scores' is not yet established without held-out prompt validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LPE (Listen, Perceive, and Express), a framework for empathetic spoken dialogue that avoids the need for spoken question-answering (QA) training data. LPE uses a WavLM speech encoder with two adapters feeding a frozen LLaMA2-7B-chat decoder. A two-stage training procedure first aligns speech content with text via ASR data, then aligns emotional information via SER data using multitask learning and an emotion classification loss. At inference, a manually designed Chain-of-Thought (CoT) prompt guides the model to first transcribe, then identify emotion, and finally generate an empathetic response. The authors construct test sets from IEMOCAP session 5 and a TTS-augmented Alpaca set, evaluate ASR/SER performance, and use BLEU, BERTScore, and GPT-4o-based subjective scoring to claim that LPE outperforms other speech-based LLMs. Ablations show that stage 2 and the CoT prompt steps are important for the reported gains.","tokens_in":9365,"tokens_out":3144,"duration_ms":35181,"significance":"If validated, the LPE approach is a practically valuable contribution: it shows a frozen text LLM can be adapted to empathetic spoken dialogue using only ASR and SER supervision, which are more readily available than spoken QA datasets. The paper's strengths include the use of natural speech data, a small number of trainable parameters (79.7M, 1.1% of the model), and a clear decomposition of listening, perceiving, and expressing via CoT. The ablations are informative and consistent with the proposed mechanism. However, the significance is tempered by the evaluation design: the headline empathy results rely on GPT-4o as both reference generator and scorer, the CoT prompt is selected on the test set, and no human evaluation is reported. These issues directly affect the central claim that LPE outperforms other speech-based LLMs in empathetic response quality.","major_comments":[{"comment":"The final CoT prompt is selected by iterative refinement on the test set. Section II.C states: 'We evaluate each prompt on several samples of test set using our two-stage trained model. By iteratively refining the prompts, we identify the most effective prompt for each CoT type, leading to the results in Section IV-C.' This same prompt is then used for the headline comparisons in Table III and the objective results in Table II. Since Table IV shows that without the selected prompt the model largely emits transcriptions or emotion labels (No CoT: Content 2.14, Empathy 1.35), the entire empathetic-expression capability is carried by the prompt. Selecting the prompt on the evaluation test set therefore means the reported advantage may reflect test-set overfitting in prompt selection rather than a general capability of LPE. Please validate the prompt on a held-out split or otherwise demonstrate that the chosen prompt generalizes to unseen data.","section":"II.C and IV.C"},{"comment":"The ground truth empathetic responses are generated by GPT-4o (Section III.B) and the subjective quality scores are also produced by GPT-4o (Section III.D), with no human evaluation reported. This creates a circularity: the references and the scoring model share the same model, so the objective metrics (BLEU, BERTScore) and the subjective scores may systematically favor GPT-4o-like responses. The statement 'we manually verify all responses to ensure the quality of GT responses' verifies the references, not the generated responses. Please add a human evaluation on a sample of generated responses, or at minimum report the correlation between GPT-4o scores and human judgments on this task.","section":"III.B and III.D"},{"comment":"The cascaded baseline is given the same CoT prompt optimized for LPE, rather than a prompt tuned separately for the cascade. Section III.C states: 'As for the cascade model, we use the same CoT prompt as our LPE.' Since the paper itself demonstrates that prompt choice strongly affects response quality (Table IV), this design biases the comparison in Table III and Fig. 2: the LPE prompt was selected for LPE's decoding behavior, not for the cascade's. Please tune the cascade's prompt separately, or report the cascade with its official recommended prompt as well as with the LPE prompt.","section":"III.C and Table III"}],"minor_comments":[{"comment":"There are several grammatical errors that should be corrected, including 'listen the content' and 'an novel' in the Abstract and Introduction.","section":"Abstract and Section I"},{"comment":"The bracket notation used to denote transcriptions, emotions, and CoT steps is only described in the caption; please define it in the main text as well.","section":"Fig. 1 caption"},{"comment":"The row label '-w/o stage 2' has inconsistent spacing; also consider reporting confidence intervals or significance tests for the objective metrics, since the differences in BERTScore are small.","section":"Table II"},{"comment":"The prompt-ablation results in Table IV are reported without standard deviations, unlike Table III; adding them would clarify whether the differences between prompt variants are meaningful.","section":"Table IV"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the two-stage training results are internally consistent, but the evaluation paradigm is the main obstacle. Before this paper can be accepted, the authors need to address the test-set prompt selection and provide some form of human evaluation or at least validation of the GPT-4o-based scoring against human judgments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: LPE is a plausible engineering recipe—align a frozen LLaMA into speech content and emotion using ASR/SER supervision, then use a hand-crafted CoT prompt to switch the decoder from transcription/emotion-labeling into empathetic response mode. That combination is new, and the two-stage story is coherent. The ASR/SER numbers are concrete, and the ablation study is useful: stage 2 clearly matters, and the authors honestly report that without CoT the model just transcribes or labels emotion.\n\nThe soft spots are real, though not fatal to the core idea. The stress-test note is correct. Section II-C states the CoT prompt was iteratively refined on the test set, and the model's entire empathetic expression depends on the prompt (Table IV: No CoT gets Content 2.14, Empathy 1.35). So the headline 'outperforms' comparison in Tables II and III partly compares a test-set-tuned prompt against baselines using their official prompts or a prompt tuned for LPE. The cascade baseline gets the same CoT prompt, which is likely suboptimal for it. That biases the comparison.\n\nCompounding this, GPT-4o plays three roles: generating response templates, generating ground-truth empathetic responses, and scoring the outputs. No human evaluation is reported. For an empathy claim, that is a load-bearing gap. The objective metrics (BLEU, BERTScore) are weak for open-ended dialogue and not decisive.\n\nThe novelty claim should be narrowed to empathetic spoken dialogue rather than speech-based dialogue generally, since prior speech CoT work (CoT-ST, SeamlessExpressiveLM) exists.\n\nThis is not a desk-reject. The idea is worth refereeing. A serious reviewer should ask for held-out prompt validation, human ratings, and released code/data. With those, the claim could be convincing.\n\nFor me: I would send it to review, but I would not cite it yet.\n\nBest.","headline":"A sensible two-stage alignment pipeline plus CoT prompting for speech-conditioned empathetic responses, but the headline superiority is undercut by test-set prompt tuning and GPT-4o-only evaluation.","tokens_in":9955,"tokens_out":2127,"would_cite":false,"duration_ms":23104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen text LLM, trained only on ASR transcripts and emotion labels through two small adapters, can be prompted to generate empathetic spoken-dialogue responses with no question-answer supervision.","keywords":["empathetic dialogue","spoken dialogue system","chain-of-thought prompting","speech emotion recognition","automatic speech recognition","multitask learning","frozen language model","speech-text alignment"],"falsifier":"Corrupt the emotion embedding at inference while keeping the transcript identical; if generated responses and their empathy scores stay unchanged, the perceive stage is not load-bearing and the claimed role of emotion perception is unsupported.","tokens_in":8883,"feed_emoji":"💬","tokens_out":8528,"duration_ms":83889,"temperature":0.7,"pith_summary":"The paper sets out to show that empathetic spoken dialogue does not require spoken question-answer training data. The proposed LPE method decomposes the task into listening (ASR), perceiving (SER), and expressing (empathetic response), trains a frozen LLaMA2 decoder on existing speech transcription and emotion datasets, and then uses a hand-written Chain-of-Thought prompt to activate the expression step. If correct, this removes the main data bottleneck for empathetic spoken dialogue and gives a low-cost recipe: two adapters plus prompting on top of a frozen text LLM. The experiments report competitive ASR and SER accuracy, better generated-response scores than SALMONN and Qwen-Audio-Chat, and win rates above a cascaded Whisper + emotion2vec + LLaMA2 baseline.","feed_headline":"Skip QA data: prompt a frozen LLM to answer speech with empathy","feed_subtitle":"Existing ASR and emotion data plus a step-by-step prompt let a frozen LLM answer speech empathetically.","key_machinery":"The central mechanism is the LPE (Listen, Perceive, Express) framework: a frozen LLaMA2-7B-chat decoder is fed speech content embeddings from a WavLM encoder through a subsampler adapter, plus a pooled emotion embedding from a multiscale adapter. Two-stage training first aligns content with ASR transcripts, then aligns emotion labels through multitask ASR/SER training. At inference, a hand-crafted zero-shot Chain-of-Thought prompt explicitly lists the steps transcribe, identify emotion, and respond empathetically, and this step decomposition is what unlocks response generation without QA supervision.","core_discovery":"The paper claims that a spoken dialogue system can generate empathetic text responses without ever being trained on question-answer pairs. A frozen LLaMA2-7B-chat decoder is aligned to speech content through a subsampler adapter trained on ASR data, then aligned to emotional cues through a multiscale adapter trained with multitask ASR and SER data. At inference, a manually designed zero-shot Chain-of-Thought prompt instructs the model to first transcribe the speech, then identify the emotion, and finally produce an empathetic response. The authors report that without this prompt the model merely outputs transcriptions or emotion labels, and that with it LPE outperforms existing speech-based LLMs on objective and GPT-4o-based subjective metrics while matching a state-of-the-art cascaded pipeline.","pith_inferences":["If the CoT prompt is the true switch that turns transcription behavior into response behavior, the same two-stage recipe may transfer to other modalities where paired instruction data is scarce, such as visual emotion perception; the bottleneck becomes designing intermediate steps rather than collecting new supervised pairs.","The poor few-shot CoT results suggest the frozen LLM may be copying the style of the provided rationale instead of conditioning on the speech input; a follow-up could measure n-gram overlap between the generated response and the few-shot example to test this copying hypothesis.","Because the current evaluation uses GPT-4o-generated references and GPT-4o scoring, a human listening study would test whether the reported empathy win rates hold when people, rather than an automated judge, rate the responses."],"forward_implications":["Spoken empathetic dialogue can be built from a frozen text LLM plus two trainable speech adapters using only existing ASR and SER datasets, so the need for expensive spoken question-answer corpora disappears.","Zero-shot CoT with manually defined listen-perceive-express steps is the prompt configuration that unlocks empathetic generation, while longer reasoning prompts and few-shot rationales hurt content quality in this setup.","The second training stage is essential: without emotion alignment via multitask ASR/SER training, the model falls back to transcribing or labeling instead of responding.","The single-model LPE matches or slightly exceeds a cascaded Whisper + emotion2vec + LLaMA2 system on content and empathy scores, suggesting the end-to-end adapter approach does not sacrifice quality for lower latency and simplicity."],"supporting_citations":[{"why":"Supplies the Chain-of-Thought prompting technique that LPE uses to decompose empathetic dialogue into listening, perceiving, and expressing steps.","marker":"[17]"},{"why":"Provides the WavLM speech encoder whose layer outputs give both content and emotion embeddings.","marker":"[22]"},{"why":"Supplies the frozen LLaMA2-7B-chat decoder that generates all responses without parameter updates.","marker":"[4]"},{"why":"Provides LibriSpeech ASR data used in stage 1 to align speech content with text.","marker":"[27]"},{"why":"Supplies IEMOCAP emotion data for stage 2 emotion alignment and for the held-out session 5 test set.","marker":"[28]"},{"why":"Supplies MEAD emotion data used alongside IEMOCAP in stage 2.","marker":"[29]"},{"why":"Provides zero-shot CoT prompting, which the paper extends by adding manually defined inferential steps.","marker":"[24]"},{"why":"Serves as an end-to-end speech LLM baseline (Qwen-Audio-Chat) that LPE is compared against.","marker":"[10]"},{"why":"Serves as an end-to-end speech LLM baseline (SALMONN) that LPE is compared against.","marker":"[11]"}],"fun_headline_variants":["Frozen LLM speaks empathy via chain-of-thought, no QA data","Prompt, don't train: empathetic speech answers from a frozen LLM","Chain-of-thought unlocks empathy in speech-based LLM without QA pairs","No QA data needed: CoT prompt makes speech LLM empathetic","Listen, perceive, express: zero-shot empathy in spoken dialogue"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a hand-written step-by-step prompt can flip a language model with locked weights from repeating the transcript or emotion label it was trained to emit into producing a new empathetic response, even though the model never saw an example of that final response during training.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLM speaks empathy via chain-of-thought, no QA data","Prompt, don't train: empathetic speech answers from a frozen LLM","Chain-of-thought unlocks empathy in speech-based LLM without QA pairs","No QA data needed: CoT prompt makes speech LLM empathetic","Listen, perceive, express: zero-shot empathy in spoken dialogue"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000755,"raw_usage":{"total_tokens":3352,"prompt_tokens":934,"completion_tokens":2418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2324}},"tokens_in":550,"tokens_out":2418,"duration_ms":16835,"temperature":1.0,"reasoning_tokens":2324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:48:15.337064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Corrupt the emotion embedding at inference while keeping the transcript identical; if generated responses and their empathy scores stay unchanged, the perceive stage is not load-bearing and the claimed role of emotion perception is unsupported.","supporting_citations":[{"cited_title":"Mead: A large-scale audio-visual dataset for emotional talking-face generation,","cited_arxiv_id":null,"evidence_quote":"Supplies MEAD emotion data used alongside IEMOCAP in stage 2."}],"review_version":1}