{"id":"086e4144-ef3e-4ec1-9e12-aa2515b3648d","arxiv_id":"2608.06409","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A diagnostic ladder shows that speech LMs lose emotion accuracy both at the decision rule over option logits and at the readout from hidden states, and that hidden-state emotion information exists but is rarely used in answers.","lead":"The paper introduces a diagnostic ladder that separates where speech language models lose emotion information: at the decision rule over option scores, or at the readout from hidden states. Across five models and two emotion corpora, hidden states support far better emotion decoding than the models' generated answers, but the extra information rarely changes what the models say.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The availability-use distinction rests on an L*-layer projection of the final head; this is a diagnostic approximation, not the native readout, so the readout-external label and null causal effects are not directly anchored to the answer-position coverage gap.","rationale":"The reader's weakest_assumption identifies precisely this point: the L* projection onto V4 and its complement is load-bearing for the readout-external analysis, and if L* is misaligned with the final head, the readout-external directions could be mislabeled and the null causal effects would not support the availability-use distinction. I agree with that assessment, and I have added a concrete final-layer experiment that would settle it. The paper is otherwise careful: the ladder decomposition is exact by telescoping, the endpoint audit verifies that the saved state reproduces the stored logits at the final layer, the matched-budget refit and MLP-on-contrasts robustness checks strengthen the coverage-gap interpretation, and the acoustic controls are thorough. The transductive logit correction is disclosed and discussed. Because the concern is already recognized in the reader's conditional verdict and the proposed test is an additional validation rather than a demonstrated contradiction, the verdict should remain CONDITIONAL: acceptance should require either the final-layer analysis or an explicit validation that the L*-space decomposition corresponds to the actual readout geometry. The paper's central descriptive claims (positive gaps in all conditions, logit-correction gains, limited causal influence of the selected L* external directions) are internally consistent and well supported given the stated caveats; the missing piece is the bridge from the L* subspace analysis back to the answer-position coverage gap.","tokens_in":31347,"tokens_out":6042,"duration_ms":65895,"concrete_test":"Repeat the rank-three held-out decoding and the rank-four minimal-pair replacement at the final answer layer (the layer whose state is fed to the output head), using V3 and V4 defined from the final head. Concretely: project final-layer states onto V4⊥, fit a logistic model on training speakers, take the three leading singular directions of its coefficient matrix as S_decoding, evaluate on held-out speakers against 20 random rank-matched subspaces; then repeat the S_intervention replacement at the final layer and measure donor-following against the random arm. If at the final layer S_decoding does not beat random subspaces, the readout-coverage gap is not attributable to information outside the native readout.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central availability-use claim depends on identifying 'readout-external' directions that are available but not causally used. The final-head decomposition V3/V4 is exact only at the final answer position (Section III-D), yet all rank-three decoding (Fig. 4a) and rank-four causal replacement (Fig. 4b,c) are performed at an intermediate layer L* selected by logit-lens accuracy. At L*, applying the final output head is a linear diagnostic approximation; the true mapping from h_L* to the final logits is a nonlinear stack of attention, MLP, and residual blocks. A direction classified as readout-external (in V4⊥) could therefore influence the final logits through later layers, and a direction inside V4 need not be the operative readout direction at L*. The paper acknowledges that 'At L*, V3 is therefore a reference aligned with the final readout, not an exact decomposition of the final logits,' but it still uses the V4-complement to draw conclusions about readout-coverage. The coverage gap Δ_coverage is measured at the final layer, while the external-information analysis is at L*, and no experiment validates that the L*-projection into V4/V4⊥ represents the native readout geometry at the answer position. If the final layer's V3⊥ contains little decodable emotion information, then the large Δ_coverage values in Table I could reflect the full-state reader's larger input space or regularization differences rather than information outside the native readout, undermining the availability-use distinction as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a generation-aligned diagnostic ladder for speech language models, comparing four levels at the same first-answer-token event: generated accuracy (A_gen), the option-logit argmax (A_opt), an affine reader over option-logit contrasts (A_aff), and a regularized affine reader over the full hidden state (A_state). Because the levels are nested, the differences telescope into an endpoint-validity term, a decision-rule gap, and a readout-coverage gap (Eq. 3). Across five systems and two emotion corpora (ten conditions), the paper reports A_state - A_gen = 0.278 on average, with both explanatory gaps positive in every condition. A label-free logit correction improves generation in all ten conditions. In rank-matched subspace analyses at an intermediate layer L*, emotion information outside the native option-readout span generalizes to held-out speakers and survives acoustic controls, but minimal-pair replacement of the selected readout-external directions rarely changes the emitted answer. The authors conclude that decision-rule misalignment and readout-coverage limitations are separate, and that decodable information need not be causally used.","tokens_in":31687,"tokens_out":10401,"duration_ms":106172,"significance":"The diagnostic ladder is simple and the telescoping identity is exact; the empirical protocol is unusually careful, with speaker-disjoint evaluation, paired speaker-clustered bootstrap intervals, matched-budget refits, delete-one-speaker jackknives, endpoint audits, perturbation-magnitude diagnostics, and a batch-calibration procedure whose predictions were fixed before the generation pass. The paper also states its limitations explicitly. If the L*-based availability/use analysis is anchored to the final answer position, the contribution would be a useful, falsifiable decomposition that separates behavioral failures from representational ones. The main weakness is that the availability/use claims are established in L* geometry rather than at the answer-position readout, which weakens the central availability-use distinction. With added final-layer validation, this would be a strong paper.","major_comments":[{"comment":"The availability/use analysis is carried out at an intermediate layer L* using the final head's option-row geometry, while the coverage gap Δ_coverage is defined at the final answer position. The paper acknowledges (Section III-D) that at L*, V3 is \"a reference aligned with the final readout, not an exact decomposition of the final logits,\" but Section V-D nonetheless concludes that \"readout-external emotion information has limited causal access at the answer.\" Because later attention, MLP, and residual blocks can mix a V4⊥ component into the final option logits, a direction that is readout-external in the L* geometry need not be readout-external in the geometry that determines the emitted answer. The one-condition propagation analysis (Supplementary S7-F) shows that in Qwen2-Audio×CREMA-D an injected layer-20 external component reaches the final logits, but no analogous evidence is provided for the other nine conditions. I ask for either (a) a repeat of the key decodability and minimal-pair causal analyses at the final answer-position state, where the V3/V4 decomposition is exact, or (b) a direct validation that the L* projection behaves like the final readout for these subspaces (e.g., compare V4⊥ decodability and injected-component propagation at L* and at the final layer across all conditions). Without this, the readout-external label and the null causal effects are not anchored to the answer-position readout.","section":"III-D and V-D"},{"comment":"The readout-coverage gap is a final-layer performance gap between a three-contrast reader and a full-state reader, and the paper is careful to call it a performance gap. However, the localization of that gap to information \"outside the native readout\" is based on rank-matched subspace decoding at L*, not at the answer position. If the final-layer V3⊥ complement contains little decodable emotion information, the large Δ_coverage values in Table I could be dominated by the full-state reader's larger input space rather than by information that the native readout fails to expose. The matched-budget refit (Table S6) rules out regularization differences but not the input-space dimension. I would like to see the final-layer analogue: held-out decoding from P_{V3} and P_{V3⊥} at the answer position, with matched ranks and budgets, so that Δ_coverage is directly connected to the availability claim.","section":"III-D and Table I"}],"minor_comments":[{"comment":"The definition of the offset b_k is visually ambiguous: the centering term should be written as (1/4) Σ_{j=1}^4 log p̂_j, and the text should state explicitly that the sum runs over the four prompted options.","section":"Section III-C, Eq. (4)"},{"comment":"The causal intervention uses one fixed rand4 draw as the control arm; because individual random subspace draws vary in the decoding analyses (Table S9), a sensitivity check over several rand4 draws would help establish that the small readout-external effects are not an artifact of the particular random control.","section":"Section IV-B and Fig. 4"},{"comment":"The continuous chance line at 0.25 in Fig. 4(a) should be accompanied by the empirical majority-class rates (25.94% for CREMA-D and 25.01% for VESUS, Table S2) so the reader can interpret performance relative to the actual class distribution.","section":"Section V-C and Fig. 4(a)"},{"comment":"The term \"readout-external\" is used both for the exact final-position decomposition in Eq. (6) and for the L*-diagnostic geometry; introducing a distinct label for the latter would avoid treating the L* projection as exact.","section":"Section III-D"},{"comment":"Reference [4] (Kimi-Audio technical report) lacks a stable identifier, and reference [38] should include the full Interspeech citation details; both are easy to resolve.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical core (the ladder and the correction result) is sound and well reported. The main open question is whether the availability/use distinction can be anchored to the final answer position; the authors have the data and infrastructure to run the final-layer check, so the revision seems feasible. The paper's honest treatment of VESUS's ten-speaker limitation and the thorough supplementary tables are strengths. If the final-layer anchoring is provided, I would be comfortable supporting publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time if you work on evaluation or interpretability of speech LMs. The core idea is simple but genuinely useful: align the emitted token, the option logits, and the hidden state at the same answer event, then telescope the accuracy differences into endpoint, decision-rule, and readout-coverage gaps. The identity is exact by definition, but the empirical payoff is real. Both gaps are positive in all ten conditions, the label-free logit correction helps in every condition, and the rank-matched decoding plus causal replacement gives a clean separation of availability from use. Across five systems and two corpora, with speaker-disjoint splits, paired bootstrap intervals, matched-budget refits, and acoustic controls, this is careful, reproducible-minded work. The authors are also unusually explicit about what their measurements do and do not show.\n\nI looked hard at the stress-test worry: the readout-external analysis happens at an intermediate layer L* chosen by logit-lens accuracy, not at the final answer position. That is a real caveat. The final-head decomposition V3/V4 is exact only at the end, and applying the head at L* is a linear approximation of a nonlinear stack. But the paper explicitly calls V3 at L* a reference, not an exact decomposition, and it restricts the strong claims accordingly. The coverage gap itself is measured at the final layer; the L* analysis supports the availability/use distinction rather than carrying it alone. Combined with the full-state replacement control and the depth scan, the interpretation that readout-external information has limited causal influence is reasonably anchored. I would have liked a validation that the L*-projection geometry is stable across nearby layers, but the authors already provide a partial check with the depth profile.\n\nMy biggest practical objection is the lack of released code and artifacts. The paper has enough detail to reproduce most of it, but for a study this quantitative, independent verification of the exact numbers would settle the remaining doubts. The VESUS corpus has only 10 speakers, which the authors flag; the transductive logit correction is also a limitation they acknowledge, and it is not fatal.\n\nThe central conclusion holds up: behavioral accuracy conflates distinct failure locations, and this framework localizes them in a way that suggests different interventions. I would send this to peer review, and I would cite it if I were writing on speech LM evaluation or probing. Serious thinker: yes.","headline":"A well-executed diagnostic framework that separates decision-rule from readout-coverage gaps in speech LMs; the L* approximation is a real caveat but the authors handle it honestly, and the paper deserves a serious referee.","tokens_in":32190,"tokens_out":1631,"would_cite":true,"duration_ms":19394,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diagnostic ladder separates decision-rule from readout-coverage failures in speech emotion answers.","keywords":["speech language models","paralinguistic evaluation","emotion recognition","diagnostic ladder","logit correction","readout-coverage gap","probing classifiers","activation patching"],"falsifier":"Repeat the ladder with the full-state reader restricted to a three-dimensional projection of the hidden state chosen without emotion labels, such as the top three principal components of the training states. If its accuracy no longer exceeds the affine reader over the three option-logit contrasts, the readout-coverage gap would be exposed as an artifact of the state reader's larger input space rather than a property of the model's readout geometry.","tokens_in":31157,"feed_emoji":"🎙️","tokens_out":11220,"duration_ms":99059,"temperature":0.7,"pith_summary":"The paper tries to establish that a speech language model's poor answer on a prompted emotion question does not mean the emotion information never reached the model. It builds a generation-aligned ladder with four rungs — the emitted answer, the option-logit choice, an optimized affine reader of those logits, and a linear reader of the hidden state at the same answer token — so that successive accuracy differences decompose the total gap exactly into an endpoint term, a decision-rule gap, and a readout-coverage gap. Across five systems and two emotion corpora, both explanatory gaps are positive in every one of the ten conditions, and full-state decoding beats generation by 27.8 accuracy points on average. A label-free logit correction improves the emitted answer in all ten conditions, while readout-external directions that remain decodable rarely change the emitted answer when replaced, separating information availability from behavioral use.","feed_headline":"Speech-model answers lag their own hidden states by 27.8 points","feed_subtitle":"A four-rung ladder separates decision-rule losses from readout-coverage losses in every condition tested.","key_machinery":"The central object is the generation-aligned diagnostic ladder, anchored at one answer token. Let $x_t$ be the post-normalization hidden state, $W$ the output head, and $z_t = W x_t$ the full-vocabulary logits; the option-contrast vector $c_t \\in \\mathbb{R}^3$ contains the three option-logit differences relative to a reference option. The ladder compares greedy generation over the full vocabulary, the argmax over the four option logits, an affine reader fitted on $c_t$, and an affine reader fitted on $x_t$, all scored on the same rows. The exact identity $A_{\\mathrm{state}}-A_{\\mathrm{gen}} = (A_{\\mathrm{opt}}-A_{\\mathrm{gen}})+(A_{\\mathrm{aff}}-A_{\\mathrm{opt}})+(A_{\\mathrm{state}}-A_{\\mathrm{aff}})$ carries the argument, splitting the total gap into endpoint, decision-rule, and readout-coverage terms. To localize the coverage gap, $x_t$ is decomposed as $x_t = P_{V_3}x_t + P_{V_3^\\perp}x_t$, where $V_3$ is the row space of the option-contrast matrix $C$ (the answer-readout space); decoding from $V_3$ is compared with supervised and random subspaces of $V_4^\\perp \\subset V_3^\\perp$ at a training-selected layer $L^*$. Minimal-pair activation replacement, $h_r + P_S(h_d - h_r)$, tests whether the selected readout-external directions actually steer the emitted answer.","core_discovery":"The paper's central discovery is that the gap between what a speech language model emits and what its answer-position hidden state supports has a precise internal structure. At the first answer token, the model's hidden state $x_t$ is passed through the output head $W$ to produce full-vocabulary logits; restricting those logits to the four prompted options gives the native option choice. The paper defines four accuracy levels on identical rows — $A_{\\mathrm{gen}}$ (greedy generation), $A_{\\mathrm{opt}}$ (option-logit argmax), $A_{\\mathrm{aff}}$ (a fitted affine reader over the three option-logit contrasts), and $A_{\\mathrm{state}}$ (a regularized affine reader over the full state) — and shows by telescoping that $A_{\\mathrm{state}}-A_{\\mathrm{gen}}$ equals the sum of an endpoint term, a decision-rule gap $A_{\\mathrm{aff}}-A_{\\mathrm{opt}}$, and a readout-coverage gap $A_{\\mathrm{state}}-A_{\\mathrm{aff}}$. In all ten system–corpus conditions both explanatory gaps are positive and their confidence intervals exclude zero, with $A_{\\mathrm{state}}$ exceeding $A_{\\mathrm{gen}}$ by 27.8 points on average. The readout-coverage gap is not pure readout geometry: the paper treats it as a performance gap and leans on rank-matched subspace comparisons. A label-free logit correction improves the generated answer in every condition, while replacing selected readout-external directions rarely changes the answer, so the paper's sharper claim is that emotion information can be available at the answer position without being used.","pith_inferences":["A testable extension: the same three-view ladder can be applied to any single-token prompted classification task; the ratio of the two gaps may predict whether logit calibration or readout adaptation will help.","An implicit consequence: if readout-external emotion directions rarely change the answer, then auxiliary supervision that pushes emotion information into the option-token span should shrink the coverage gap; donor-following rates would rise if that routing story is right.","A stress test of the correction: estimate the option offsets on one corpus and apply them to another; if the gains vanish, the decision-rule gap is dominated by corpus-specific option priors rather than a stable system bias.","An editorial caution: the acoustic controls cover twenty measured descriptors and loudness, not every signal property; resynthesis-based controls could settle whether the residual readout-external decodability is genuinely emotion content or additional unmeasured cues."],"forward_implications":["A single generation-accuracy score understates what a speech model knows: in every evaluated condition, the answer-position state supports substantially better emotion decoding than the model's emitted answer.","A positive decision-rule gap means logit-side fixes — calibrating option preferences or fitting a better rule over existing contrasts — can recover accuracy without changing hidden representations.","A positive readout-coverage gap means reweighting option logits is not enough; improving answers requires changing how hidden-state information reaches the output, such as a readout adapter or auxiliary supervision.","Evaluation of paralinguistic language models should report generated answers, option logits, and hidden-state decodability at the same answer event, with speaker-disjoint splits and dimension-matched controls.","Probing results alone are not mechanism evidence: matched replacement shows that decodable readout-external information can have only limited causal influence on the answer."],"supporting_citations":[{"why":"Provides the CREMA-D emotion corpus whose 4,900 clips anchor four of the ten conditions.","marker":"[11]"},{"why":"Provides the VESUS emotion corpus, adding a second speaker set and five more conditions.","marker":"[38]"},{"why":"Source of label-free calibration before use, adapted by the paper into a generation-time logit correction.","marker":"[12]"},{"why":"The unlabeled-batch marginal estimator that the correction adopts as its offset.","marker":"[13]"},{"why":"Documents option-position and token biases in language models, which the decision-rule gap is designed to capture.","marker":"[18]"},{"why":"Warns that probing decodability does not imply use, motivating the distinction between availability and behavioral influence.","marker":"[28]"},{"why":"Shows supervised subspace selection can confound decodability with mechanism, motivating the rank-matched random controls.","marker":"[34]"},{"why":"Establishes a related knowledge-prediction gap in text-only multiple-choice questions that the speech ladder extends.","marker":"[35]"}],"fun_headline_variants":["Speech-model answers trail hidden states by 27.8 accuracy points","New ladder splits emotion answer gaps into decision and readout losses","Logit correction fixes emotion answers in every condition tested","Hidden-state info beats emitted answers by 27.8 points across 10 runs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole readout-coverage story rests on the assumption that the intermediate layer chosen for inspection reflects the model's real answer pathway; if that layer does not line up with the final output head, directions called 'readout-external' are mislabeled, and the weak causal effects no longer show that the model has emotion information it fails to use.","fun_headline_variants_meta":{"raw":{"variants":["Speech-model answers trail hidden states by 27.8 accuracy points","New ladder splits emotion answer gaps into decision and readout losses","Logit correction fixes emotion answers in every condition tested","Hidden-state info beats emitted answers by 27.8 points across 10 runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1710,"prompt_tokens":1092,"completion_tokens":618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":708,"tokens_out":618,"duration_ms":6083,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:26:45.764117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the ladder with the full-state reader restricted to a three-dimensional projection of the hidden state chosen without emotion labels, such as the top three principal components of the training states. If its accuracy no longer exceeds the affine reader over the three option-logit contrasts, the readout-coverage gap would be exposed as an artifact of the state reader's larger input space rather than a property of the model's readout geometry.","supporting_citations":[{"cited_title":"VESUS: A crowd-annotated database to study emotion production and perception in spoken English,","cited_arxiv_id":null,"evidence_quote":"Provides the VESUS emotion corpus, adding a second speaker set and five more conditions."},{"cited_title":"Calibrate before use: Improving few-shot performance of language models,","cited_arxiv_id":null,"evidence_quote":"Source of label-free calibration before use, adapted by the paper into a generation-time logit correction."}],"review_version":1}