{"id":"209ca10f-08e8-4457-813c-372179cff19a","arxiv_id":"2505.13237","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SAKURA shows large audio-language models struggle with multi-hop reasoning from speech and audio even when they correctly perceive the needed attribute.","lead":"This paper introduces SAKURA, a benchmark of 4,000 multiple-choice questions that tests whether audio-language models can reason across several pieces of information heard in speech and sound. It finds these models often fail to combine what they hear with general knowledge, even when they have recognized the sound correctly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle-caption text control, combined with single-hop-correct filtering that admits lucky guesses, leaves the S/A-vs-Text gap ambiguous as evidence for a modality-integration deficit.","rationale":"The reader's weakest assumption correctly identifies the oracle-caption control as the vulnerable point. Our stress-test sharpens it: the filter on single-hop correctness is not sufficient to ensure genuine extraction because MCQ guessing allows lucky correct answers, and for low-accuracy models the filtered subset is near-noise. This directly undermines the central claim's premise. We also note that Table 3's reported 95% confidence intervals appear to ignore the per-model variable subset sizes (e.g., GAMA-IT Language, about 28 instances, would have a CI of roughly ±16 percentage points, not ±4.0), which further weakens the quantitative support. These issues do not refute the benchmark's value or the qualitative finding that audio-based reasoning lags text-based reasoning for many models; they do mean the strong interpretation—a genuine lack of multimodal integration—requires additional controls. The reader's CONDITIONAL verdict is appropriate, and we recommend keeping it.","tokens_in":9655,"tokens_out":11276,"duration_ms":107952,"concrete_test":"For each model and track, isolate the subset of instances where the model answers the single-hop question correctly under at least two of three independently paraphrased prompts (or where a high-confidence reference system confirms the attribute is clearly present in the audio). Recompute Table 3's S/A and Text accuracies on this high-confidence subset only. If the accuracy gap persists there, the integration-deficit claim is supported; if it collapses, the original gap is an artifact of lucky guesses and oracle captions. As a complementary check, replace the oracle caption in the text condition with the model's own single-hop answer; if Text accuracy drops to S/A levels, the gap reflects the quality of the extracted attribute rather than modality integration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 3 is the sole evidence for the central claim that LALMs fail to integrate speech/audio representations into multi-hop reasoning despite accurate extraction. The filtering step (Section 4.2) retains only instances where the model answered the corresponding single-hop question correctly. For models with near-chance single-hop accuracy, this subset is dominated by lucky guesses: GAMA-IT's Language single-hop accuracy is 5.6% against a 25% chance level, so most of its ~28 'correct' items are not genuine extractions. Even for high-accuracy models, multiple-choice guessing contaminates a non-trivial fraction of correct answers. The text condition, by contrast, always supplies an oracle caption containing the true attribute (e.g., 'The animal making the sound is cat.'), so it is not true that the only difference is modality: text guarantees the attribute is present in a clean, usable form, while the audio condition's 'correct' items may be correct by chance. Consequently, the large S/A-vs-Text gap may largely reflect perception failures and lucky-guess contamination rather than a failure to incorporate latent audio representations. The paper's assertion that 'even when extracting accurate information, LALMs fail to incorporate latent speech/audio representations' is therefore not established by the current analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SAKURA, a benchmark of 4000 human-verified multiple-choice questions for evaluating multi-hop reasoning in large audio-language models (LALMs) based on speech and audio. It comprises four attribute tracks (speaker gender, spoken language, speaker emotion, animal sound), each with single-hop perception and multi-hop reasoning sub-tracks that share the same audio inputs. The authors evaluate eight LALMs and two cascaded systems, report single-hop and multi-hop accuracies, and analyze a filtered subset where each model answered the corresponding single-hop question correctly, comparing speech/audio-based versus text-caption-based multi-hop reasoning. They conclude that LALMs fail to integrate extracted speech/audio information into reasoning even when extraction appears to succeed.","tokens_in":9857,"tokens_out":8879,"duration_ms":89292,"significance":"If the central claim holds, SAKURA is a valuable resource for the community: the dataset is human-verified with high agreement, audio sources are diverse and balanced, and the single-hop/multi-hop split is clean. The evaluation uses an LLM judge that was validated against human judgments on 200 samples with 99.5% agreement, and the benchmark and evaluation code are publicly released. The finding that current LALMs lag behind cascaded systems and behind their own text-based reasoning is a falsifiable, important observation that challenges the end-to-end design of LALMs and should spur research on multimodal integration. The comparison across open-source, proprietary, and cascaded systems also provides a useful reference point.","major_comments":[{"comment":"The claim that 'even when extracting accurate information, LALMs fail to incorporate latent speech/audio representations into the reasoning process' is not fully supported by the experimental design. The text-based control condition provides an oracle caption (e.g., 'The animal making the sound is cat.'), which guarantees the attribute is available in clean propositional form, whereas the speech/audio condition's filtered subset is based only on the model's multiple-choice single-hop answer. For models with near-chance single-hop accuracy, such as GAMA-IT on Language (5.6% versus 25% chance), many 'correct' single-hop items are likely due to guessing or systematic biases, so the filtered S/A subset does not reliably isolate genuine extraction. Consequently, the S/A-versus-Text gap may partly reflect residual perception failures rather than a modality-integration deficit. The authors should either add a stricter extraction verification (e.g., open-ended single-hop questions, model-generated captions, or confidence-based filtering) or substantially soften the claim and limit it to models with high single-hop accuracy.","section":"Section 4.2, Table 3"},{"comment":"The 95% confidence intervals in Table 3 appear to be computed over the full 500-item track rather than over the smaller filtered subset. For example, GAMA-IT's Language single-hop accuracy is 5.6%, giving roughly 28 filtered instances, but the table reports a CI of ±4.0 for the S/A multi-hop accuracy, which is implausible for n=28 (a binomial 95% CI would be approximately ±17). This overstates the precision of the estimates and may mislead readers about the statistical significance of the observed gaps. The authors should recompute all Table 3 CIs using the actual filtered sample sizes and report the per-cell n in the table or in a supplementary table.","section":"Table 3"},{"comment":"The statement that 'the only difference between these conditions is the modality used to present attribute information' is imprecise. The text captions differ not only in modality but also in information form: they are clean, propositional, and directly relevant, whereas the audio contains acoustic ambiguity and requires the model to internally derive the same proposition. Even when single-hop is answered correctly, the latent speech/audio representation may be less accessible to the reasoning modules than an explicit textual proposition. This does not invalidate the comparison, but it means the observed gap is an upper bound on a pure integration deficit rather than a direct measure of it. The paper should acknowledge this confound in the interpretation.","section":"Section 4.2"}],"minor_comments":[{"comment":"The omission of GPT-4o Audio's Gender track results is noted only in the main text; the table caption should also state that these entries are omitted due to the model's refusal to answer gender-related questions.","section":"Table 2, caption"},{"comment":"The number of answer choices per track is not stated explicitly. The chance-level row implies four choices for Language, Emotion, and Animal, and two for Gender, but the paper describes eight languages in the Language track and five emotions in the Emotion track. Please clarify how choices are sampled and whether the number of choices is constant within each track.","section":"Section 2.1 and Table 2"},{"comment":"The LLM-as-a-judge evaluation would benefit from more detail: include the exact prompt or a reference to the appendix, and describe how the 200 samples for human verification were selected and how the 99.5% agreement was computed (e.g., Cohen's kappa or simple agreement).","section":"Section 3.1"},{"comment":"Please specify how the Average column is computed, in particular whether it is an unweighted mean of the four track accuracies or a pooled accuracy across tracks. This affects readability and comparability.","section":"Table 3, Average column"},{"comment":"There are minor typographical and formatting issues: 'benchmar k' in the SAKURA expansion, inconsistent use of 'GPT-4o Audio' versus 'gpt-4o-audio-preview', and the unusual rendering of 'S ¸as ¸maz' in the references. These should be cleaned up.","section":"Throughout"},{"comment":"The claim of being the 'first benchmark' for this purpose should be qualified as 'to our knowledge' and should be checked against concurrent or recent benchmarks such as MMAU, which is cited but may overlap in reasoning-focused tasks.","section":"Section 1 and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a well-constructed benchmark, but the central claim is somewhat stronger than the evidence supports. The oracle-caption control and the filtering issue are real confounds that need to be either addressed with additional analyses or reflected in a more cautious conclusion. The confidence-interval reporting error in Table 3 is a serious but fixable statistical issue. I believe the paper can be made acceptable after a major revision that tightens the interpretation and adds supporting analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nSAKURA is a genuinely useful benchmark, and the paper is worth engaging, but the marquee claim is stronger than the evidence. The authors show that LALMs do poorly on multi-hop questions grounded in speech/audio, and that they do much better when the same attribute is supplied as a caption. The first half of that is solid. The second half is what they use to conclude that LALMs fail to integrate audio representations into reasoning rather than failing to perceive the attribute. That inference is undercut by a control that is not as clean as it looks.\n\nWhat's new: the first benchmark designed specifically for multi-hop reasoning in LALMs, with paired single-hop and multi-hop sub-tracks sharing the same audio. The construction is careful: four attribute tracks, human-verified choice pools, balanced option ordering, LLM-as-judge validated at 99.5% agreement on 200 samples, and confidence intervals throughout. The dataset and code are promised on GitHub. That is a real resource for the field.\n\nThe core empirical findings in Table 2 are convincing: multi-hop accuracy is far below single-hop for nearly every model, and perception blind spots are real and model-specific. The cascaded ASR+AAC+LLM baseline beating most end-to-end LALMs is a useful data point too.\n\nThe soft spot is Section 4.2. The text condition feeds the model an oracle caption that states the attribute outright (e.g., \"The animal making the sound is cat.\"). The audio condition feeds the raw audio, and the analysis filters to instances where the model got the single-hop question right. But being right on a multiple-choice question does not mean the model has a reliable, usable representation of the attribute. For models near or below chance on a track—GAMA-IT is at 5.6% on Language vs. 25% chance—the 'correct' subset is dominated by lucky guesses. So the S/A-vs-Text gap in Table 3 plausibly reflects perception and guess contamination, not purely a failure to integrate a correctly extracted latent representation. The paper's claim that 'the only difference between these conditions is the modality used to present attribute information' is false on that reading.\n\nThis doesn't sink the paper. The benchmark stands on its own, and the general conclusion that LALMs are weak at audio-grounded multi-hop reasoning is likely right. But the specific conclusion about 'true multimodal integration' needs a stronger control: e.g., feeding the model's own single-hop answer (not the oracle) as the text condition, or filtering on confidence, or running an experiment where the audio attribute is unambiguously verified by multiple correct single-hop questions.\n\nWho it's for: people building or evaluating LALMs; benchmark designers. A serious referee should see it. I'd suggest a conditional accept with a request to fix or soften the Section 4.2 claim.","headline":"Useful new benchmark, but the headline claim about a modality-integration deficit is undercut by an oracle-caption control that doesn't isolate reasoning from perception.","tokens_in":10413,"tokens_out":3942,"would_cite":true,"duration_ms":36911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even when large audio-language models accurately extract speech or audio attributes, they fail to carry those latent representations into multi-hop reasoning, a new benchmark shows.","keywords":["large audio-language model","multi-hop reasoning","speech understanding","audio understanding","multimodal reasoning","benchmark","speech attributes","latent representations"],"falsifier":"Take any LALM that answers a SAKURA single-hop question correctly, then before the multi-hop question ask the model to state the attribute it heard (e.g., 'What animal is in the sound?'); if multi-hop accuracy on instances with correct self-reported attributes reaches the text-condition level, the integration deficit is in making the latent representation available to reasoning, not in the reasoning step itself. Conversely, if the model still fails despite correctly verbalizing the attribute, the deficit lies deeper in the reasoning module.","tokens_in":9444,"feed_emoji":"🎧","tokens_out":5902,"duration_ms":48327,"temperature":0.7,"pith_summary":"The paper introduces SAKURA, a benchmark of 4,000 human-verified multiple-choice questions that tests large audio-language models (LALMs) on single-hop perception and multi-hop reasoning across four speech/audio attributes: speaker gender, spoken language, speaker emotion, and animal sound. Its central finding is that LALMs often answer single-hop attribute questions correctly but then fail on multi-hop questions that require reasoning over that same information. When the same attribute information is supplied as text instead of speech or audio, the same models reason much better. The paper interprets this gap as a lack of genuine multimodal integration: the models' reasoning remains text-driven even in end-to-end audio-language architectures.","feed_headline":"Audio models fail at multi-hop reasoning even when they hear correctly","feed_subtitle":"SAKURA benchmark shows text-based reasoning beats speech/audio-based reasoning on the same correctly heard facts.","key_machinery":"The central mechanism is the controlled single-hop/multi-hop comparison built into SAKURA. Each of the four tracks shares the same speech/audio inputs between single-hop and multi-hop sub-tracks, so multi-hop questions can be conditioned on whether the model perceived the attribute correctly. The decisive instrument is the text-based control condition: replacing the audio input with an oracle caption of the attribute isolates the reasoning step from perception, and the gap between speech/audio-based and text-based multi-hop accuracy on the same correctly-perceived instances is what carries the paper's claim.","core_discovery":"The paper claims that current LALMs exhibit a modality-integration deficit rather than a pure perception deficit. In Section 4.2, the authors compare multi-hop accuracy on instances where each model already answered the corresponding single-hop question correctly, under two conditions: raw speech/audio input, and a textual caption stating the relevant attribute (e.g., 'The animal making the sound is cat.'). Across models, accuracy is consistently and often dramatically higher in the text condition; for instance, DeSTA2 rises from 65.8% to 93.3% average multi-hop accuracy, and Gemini-1.5-pro from 53.7% to 84.4%. Because the only intended difference is modality, the authors conclude that even when LALMs extract accurate information, they fail to incorporate latent speech/audio representations into the reasoning process, contradicting the expectation that end-to-end models unify understanding with reasoning.","pith_inferences":["If the integration deficit is real, a natural fix is to train LALMs on objectives that require reasoning over the audio encoder's latent state, not just the text decoder's output; this could be tested by fine-tuning with multi-hop audio examples and measuring whether the text-vs-audio gap shrinks.","A sharper test would compare not only whole-model accuracy but whether the attribute is linearly decodable from the model's internal representations at the multi-hop step; a decodability failure would mean the perception is not actually available to the reasoning module, while decodability with continued failure would pinpoint a reasoning-module limitation.","Because SAKURA covers only four attributes and relatively clean audio, extending it to noisy or overlapping acoustic conditions could reveal whether integration failures worsen when the extracted representation is weaker or noisier.","The cascade comparison hints that the text interface remains the most reliable carrier of attribute information; one could extend that by feeding ASR transcripts or captions from a stronger AAC model into LALMs to see whether the integration gap tracks the quality of the external text."],"forward_implications":["Improving LALM perception alone will not fix multi-hop reasoning; the integration of extracted attributes into the reasoning path must be addressed separately.","Current LALMs fall short of cascaded systems such as ASR plus audio captioning plus an LLM, which outperforms most end-to-end LALMs on this benchmark.","Benchmark designers should treat single-hop accuracy as insufficient evidence that a model can use the perceived attribute in downstream reasoning.","The finding predicts that probing or prompting LALMs to verbalize the extracted attribute before answering a multi-hop question should close part of the gap, if the deficit is in integrating latent representations."],"supporting_citations":[{"why":"Provides the Common Voice testing samples used for the Gender and Language tracks.","marker":"[31]"},{"why":"Provides CREMA-D speech samples used for the Emotion track.","marker":"[32]"},{"why":"Provides MELD speech samples used for the Emotion track.","marker":"[33]"},{"why":"Provides ESC-50 animal sound samples used for the Animal track.","marker":"[35]"},{"why":"Provides additional animal sound samples augmenting ESC-50 in the Animal track.","marker":"[36]"},{"why":"Supplies the LLM-as-a-judge evaluation method used to score model responses.","marker":"[37]"},{"why":"Provides the Whisper-large-v3 ASR model used in the cascaded baseline.","marker":"[38]"},{"why":"DeSTA2 is the evaluated LALM showing the largest text-vs-audio multi-hop gap, exemplifying the central claim.","marker":"[11]"}],"fun_headline_variants":["Hearing isn't understanding: audio LLMs stumble on multi-hop logic","Correct audio, wrong reasoning: why LALMs fail multi-hop","Perception without integration: audio models can't reason over what they hear","Text beats audio for multi-hop reasoning, even when audio is correctly heard"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that replacing the audio with a textual caption changes only the input modality, and that a correct single-hop answer guarantees the model actually has the attribute in a usable form when it tackles the multi-hop question; if either assumption fails, the speech-versus-text gap could overstate a lack of true multimodal integration.","fun_headline_variants_meta":{"raw":{"variants":["Hearing isn't understanding: audio LLMs stumble on multi-hop logic","Correct audio, wrong reasoning: why LALMs fail multi-hop","Perception without integration: audio models can't reason over what they hear","Text beats audio for multi-hop reasoning, even when audio is correctly heard"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3265,"prompt_tokens":876,"completion_tokens":2389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":2311}},"tokens_in":492,"tokens_out":2389,"duration_ms":16781,"temperature":1.0,"reasoning_tokens":2311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:20.864808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any LALM that answers a SAKURA single-hop question correctly, then before the multi-hop question ask the model to state the attribute it heard (e.g., 'What animal is in the sound?'); if multi-hop accuracy on instances with correct self-reported attributes reaches the text-condition level, the integration deficit is in making the latent representation available to reasoning, not in the reasoning step itself. Conversely, if the model still fails despite correctly verbalizing the attribute, the deficit lies deeper in the reasoning module.","supporting_citations":[{"cited_title":"Hopping too late: Exploring the limitations of large language models on multi-hop queries,","cited_arxiv_id":null,"evidence_quote":"Provides CREMA-D speech samples used for the Emotion track."},{"cited_title":"Investigating multi-hop factual shortcuts in knowl- edge editing of large language models,","cited_arxiv_id":null,"evidence_quote":"Provides MELD speech samples used for the Emotion track."},{"cited_title":"AIR-bench: Benchmarking large audio-language models via generative comprehension,","cited_arxiv_id":null,"evidence_quote":"Provides ESC-50 animal sound samples used for the Animal track."},{"cited_title":"Advancing large lan- guage models to capture varied speaking styles and respond prop- erly in spoken conversations,","cited_arxiv_id":null,"evidence_quote":"Provides additional animal sound samples augmenting ESC-50 in the Animal track."},{"cited_title":"Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-as-a-judge evaluation method used to score model responses."},{"cited_title":"Listen and speak fairly: a study on semantic gender bias in speech integrated large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the Whisper-large-v3 ASR model used in the cascaded baseline."},{"cited_title":"Do large language models latently perform multi- hop reasoning?","cited_arxiv_id":null,"evidence_quote":"Provides the Common Voice testing samples used for the Gender and Language tracks."},{"cited_title":"Our findings show that LALMs struggle to recognize certain speech and audio attributes, exhibiting perception blind spots","cited_arxiv_id":null,"evidence_quote":"DeSTA2 is the evaluated LALM showing the largest text-vs-audio multi-hop gap, exemplifying the central claim."}],"review_version":1}