{"id":"0641fc8a-7a6d-432f-a5bd-0ce01a41f60f","arxiv_id":"2411.15685","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A fully state-space audio-language model matches transformer counterparts on audio classification and captioning using a 2.8B parameter backbone.","lead":"This paper builds the first audio-language model that uses state-space networks instead of transformers for both the audio analyzer and the language model. It reports competitive performance on audio classification and captioning while using far fewer parameters, which could make such systems cheaper to run.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The close-ended evaluation embeds LALM outputs and labels with a text encoder and chooses labels by cosine similarity; if this protocol rewards terse, label-like outputs, the competitive claim could be an artifact of evaluation rather than audio understanding.","rationale":"The reader's weakest assumption is exactly the evaluation protocol: that cosine similarity between generated text and label embeddings is a valid measure of audio reasoning. I agree, and I sharpen it with a concrete mechanism: the protocol can reward terse, label-like outputs, which the non-instruction-tuned Mamba in ssLALM may produce, while penalizing the verbose, instruction-tuned transformer baselines. This is a load-bearing concern because the central claim is a comparative one; if the comparison metric is biased, the reported numbers do not support the conclusion. The paper does have genuine strengths: the hybrid-LALM vs. ssLALM ablation controls the audio encoder, the DASS encoder is a reasonable state-space replacement, and the open-ended samples are plausible. However, the evaluation pipeline is taken from LTU and the baseline numbers for SALMONN, Pengi, and AudioGPT are borrowed from GAMA, so any mismatch in generation settings or text encoder across papers would also affect comparability. The concrete test would settle whether the observed competitiveness survives a less verbosity-sensitive evaluation. Since the reader already issued a CONDITIONAL verdict, my concern reinforces that condition rather than demanding rejection; the claim is plausible but not yet robustly established.","tokens_in":8623,"tokens_out":11445,"duration_ms":117614,"concrete_test":"Run the classification evaluation on ESC50 and VGGSound while recording the average token length of generated responses for each model under the fixed prompt 'write an audio caption describing the sound'. Then re-score the same outputs using (i) a different text encoder (e.g., a sentence-transformer with mean pooling) and (ii) a substring-matching metric (does the gold label string appear in the generated output, modulo stemming). If the ssLALM responses are substantially shorter than LTU's and its relative scores drop under these alternative metrics, the cosine-similarity protocol is load-bearing and the competitive claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends entirely on Table I, which is produced by the Section III.E protocol: encode the LALM-generated text and each class label with an unspecified text encoder (presumably CLIP), then compute cosine similarity. For single-label tasks, accuracy is the fraction of times the highest-similarity label is correct; for multi-label tasks, the cosine similarity itself is the prediction score. This protocol is known to be sensitive to output verbosity and phrasing: the CLIP text encoder truncates at 77 tokens and its final-token representation can be dominated by sentence structure rather than semantic content. A model that outputs short, label-like phrases (e.g., 'dog barking') will naturally achieve higher cosine similarity to the label 'dog barking' than a model that outputs a full descriptive sentence, even if both correspond to the same audio. The ssLALM uses a non-instruction-tuned Mamba-2.8B, whereas the transformer baselines (LTU, SALMONN, Pengi) use instruction-tuned LLaMA-7B; instruction-tuned models are biased toward verbose, conversational answers. Thus the evaluation may systematically deflate the transformer baselines and inflate the state-space model, making the 'competitive' result an artifact. The paper provides no control for output length or phrasing, and no error bars, so this is not ruled out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the first state-space-based large audio language model (LALM). The audio encoder is DASS, a distilled audio state-space model, and the language model is either a frozen LLaMA/Vicuna or a frozen state-space LLM (Mamba-2.8B), with LoRA adapters and a trainable downsampling projection. Models are trained on the OpenAQA dataset with next-token prediction. Evaluation covers eight audio classification benchmarks and two audio captioning benchmarks, reporting that the state-space models are competitive with transformer-based LALMs such as LTU while using fewer total parameters. The central claim is that both the audio perception and language generation components of a LALM can be replaced by state-space models without a major accuracy loss.","tokens_in":8906,"tokens_out":3471,"duration_ms":34228,"significance":"If validated, this result is useful: it demonstrates that state-space components can substitute for transformers in LALMs, with potential gains in memory and time efficiency for long audio and resource-constrained deployment. The paper is explicit about training details, LoRA configurations, and the evaluation protocol, and it provides a comparison table across 10 benchmarks. However, the validation is currently incomplete because the evaluation protocol is not fully controlled and the reported differences are small relative to the absence of error bars or significance tests. The strength of the central claim therefore depends on additional experiments that the authors can reasonably perform.","major_comments":[{"comment":"The classification protocol computes cosine similarity between a text-encoder embedding of the LALM output and embeddings of class labels, but the text encoder is never specified and no control is provided for output phrasing or length. The state-space LLM used in ssLALM is not instruction-tuned, while the transformer baselines (LTU, SALMONN, Pengi) use instruction-tuned LLaMA-7B; instruction-tuned models tend to produce longer, more conversational outputs, and a cosine-similarity scorer can systematically favor terse, label-like outputs. This confound could inflate the ssLALM results relative to the baselines. Please specify the text encoder, report results under an alternative scoring scheme (for example, exact-label matching or length-normalized embeddings), and, if possible, rerun the transformer baselines under the identical protocol with identical prompts and decoding settings.","section":"§III.E, Table I"},{"comment":"No error bars, multiple-seed results, or significance tests are reported, and many of the headline differences are very small: the classification averages are 50.2 and 49.8 for the hybrid models, 50.0 and 51.4 for the ssLALMs, and 50.3 for LTU. Differences of this size cannot be distinguished from run-to-run or decoding stochasticity. Please provide variance estimates (for example, across at least three decoding runs or training seeds), or at least report per-item bootstrap intervals for the aggregate averages. This is load-bearing because the central claim is that the state-space models are 'competitive,' not merely that they are in the same broad range.","section":"Table I"},{"comment":"For SALMONN, Pengi, and AudioGPT, the results are taken from the GAMA paper, and the footnotes indicate differences in evaluation conditions (for example, ZS- weak zero-shot, and a higher sampling rate for some methods). If the evaluation protocol, prompt template, text encoder, or decoding settings used to produce those numbers differ from those used for the proposed models, the comparison is not controlled. The authors should either rerun all baselines with the same evaluation pipeline or explicitly state which numbers were produced under which conditions and discuss the limitations. The mAP columns marked with the dagger are also interpreted with a post-hoc explanation that LALMs under-predict non-prominent classes; this is a reasonable hypothesis, but it should not be used to discount unfavorable multi-label results without supporting evidence.","section":"§III.E and Table I footnotes"}],"minor_comments":[{"comment":"The abstract says 'space-based LALM' where 'state-space-based LALM' is intended, and Section II contains the typo 'wed-scale data' instead of 'web-scale data.'","section":"Abstract and Section II"},{"comment":"The aggregate columns 'Classif. Avg.' and 'Cap. Avg.' appear to be simple arithmetic means over datasets of very different difficulty and metric types (accuracy, F1, mAP, SPICE). Please clarify the averaging procedure and consider reporting per-dataset results with a defined aggregation rule.","section":"Table I"},{"comment":"The decoding settings (temperature=0.1, top-k=500, top-p=0.95, repetition penalty=1.1) are given for the proposed models, but it is not stated whether the same settings were used for all baselines in Table I. Please state this explicitly.","section":"§III.E"},{"comment":"For the state-space LLM, the LoRA adapters are said to be added to 'the input projection layers of the state-space block'; specifying the exact matrices (for example, the in-projection of the Mamba block) and the rank/alpha choices for both LLMs would improve reproducibility.","section":"§III.C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable systems contribution, and the central claim is plausible, but the evaluation protocol is currently the main risk. The absence of a specified text encoder and the lack of length/phrasing control in the cosine-similarity classifier, together with the very small margins in Table I, mean the 'competitive' claim is not yet established. I would be willing to reconsider after the authors provide a controlled comparison, ideally with the baselines rerun under the same protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a straightforward architecture swap: it takes LTU's LALM recipe, replaces the AST encoder with DASS and the Vicuna-7B with Mamba-2.8B, and shows the resulting state-space system lands within a whisker of the transformer baselines on close-ended audio classification and captioning, at roughly 40% of the parameter count. If you work on audio-language models or efficient multimodal LLMs, that is worth knowing.\n\nThe genuinely new piece is the combination itself–to my knowledge, the first to put a state-space audio encoder and a state-space LLM together in one LALM. The authors also do a clean ablation: DASS + LLaMA, then DASS + Mamba, so you can see which swap costs what. Training details, LoRA ranks, prompts, and generation settings are all specified. The models match LTU on most benchmarks and beat the weaker baselines (SALMONN, Pengi, AudioGPT) on average. The open-ended examples in Table II look reasonable, and the authors are candid that they do not know why training was faster and that multi-label performance is weak.\n\nWhere I would push back: the evaluation protocol for classification is the softest spot. They embed the LALM output and each class label with an unspecified text encoder and pick the label with highest cosine similarity. This is known to be sensitive to output phrasing and length. The ssLALM uses a non-instruction-tuned Mamba, which may produce shorter, more label-like answers than instruction-tuned Vicuna; if so, the comparison could favor the state-space model. That concern is not ruled out, and it is load-bearing because the central claim rests on Table I. The authors do not report which text encoder they used, no error bars, no significance tests, and no output-length or phrasing analysis. A few of the differences in the table are small enough that noise could flip them.\n\nThere are also two smaller gaps: the paper is motivated by efficiency but gives no wall-clock or memory measurements for inference, only training-time observations; and the \"first state-space LALM\" claim depends on a related-work section that does not cover recent SSM-audio LLM work beyond DASS and Audio Mamba. Both are fixable in revision.\n\nOverall: the architecture finding is probably real–state-space components can replace transformers in this pipeline without a major accuracy hit. But the evaluation needs tightening before the competitive claim is bulletproof. I would send it to peer review and ask for the text encoder choice, length controls, and error bars.\n\nFor a reading group, it is a useful example of how fast SSM components are infiltrating multimodal LLMs, with a good discussion hook about evaluation bias. I would not block engagement; I would send it out.","headline":"A clean state-space swap of LTU that performs competitively, but the under-specified cosine-similarity evaluation leaves the headline claim shaky.","tokens_in":9437,"tokens_out":3277,"would_cite":true,"duration_ms":30928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing both the audio encoder and the language model with state-space models yields an audio language model that matches transformer-based rivals while using about 40% of the parameters.","keywords":["State-space models","Large audio language models","Audio reasoning","DASS","Mamba","Low-rank adaptation","Audio classification","Audio captioning"],"falsifier":"Re-evaluate the same models on the same benchmarks using forced-choice label generation without a text-encoder similarity step, or with human raters judging the outputs; if transformer LALMs then clearly outperform ssLALMs on close-ended accuracy, the paper's competitive claim collapses.","tokens_in":8454,"feed_emoji":"🎧","tokens_out":5145,"duration_ms":40858,"temperature":0.7,"pith_summary":"This paper asks whether the two heavy transformer components of a large audio language model (LALM) — the audio perception encoder and the language model itself — can be swapped for state-space models without losing accuracy. It builds a hybrid model pairing the distilled audio state-space encoder DASS with the transformer LLaMA, and a full state-space LALM pairing DASS with a 2.8B state-space LLM. Across eight audio classification benchmarks and two captioning benchmarks, both models match or slightly exceed the transformer-based LTU baseline while using roughly 40% of the total parameters. The paper's claim is that state-space LALMs are a viable efficient alternative for close-ended audio reasoning tasks, retaining competitive performance with far lower memory and time overhead.","feed_headline":"State-space audio model matches transformer rivals at 40% of size","feed_subtitle":"A 2.8B state-space LALM scores on par with 7B transformer LALMs on audio classification and captioning.","key_machinery":"The load-bearing pieces are DASS, a distilled audio state-space encoder trained on AudioSet-2M whose teacher is the transformer AST; the state-space LLM-2.8B trained on the Pile; and low-rank LoRA adapters that keep trainable parameters small. The audio is passed as a 1024x128 spectrogram into DASS, producing a 32x4x768 feature map that is downsampled and projected to the LLM embedding size. Training uses next-token prediction on audio-question-answer triples from OpenAQA, and evaluation scores classification by cosine similarity between the text-encoder embedding of the model output and each class label, and captioning by SPICE.","core_discovery":"The central discovery is that state-space models can carry both ends of an audio-language system: a state-space audio encoder distilled from a transformer teacher, together with a state-space LLM, is enough to reach the accuracy of transformer-based LALMs on close-ended tasks. The paper reports that the full state-space ssLALM with 2.8B total parameters achieves a classification average of 51.4 (medium) versus LTU's 50.3, and a captioning average of 14.7–15.1 versus LTU's 14.5, with only 43–62M trainable parameters via LoRA. This is the first demonstration that a purely state-space LALM is competitive, suggesting quadratic attention is not necessary for audio-grounded question answering when the model is trained on the OpenAQA dataset.","pith_inferences":["If the cosine-similarity evaluation were replaced by direct label generation or human judgement, the ranking of ssLALM versus transformer LALMs could shift; the paper reports no such check.","The state-space LLM is 2.8B while LLaMA is 7B, so a matched-size comparison with a 7B state-space LLM would isolate the architecture effect from the scale effect — something the authors list as future work.","The competitive result may depend on label-phrase overlap between the OpenAQA training set and the cosine-similarity scoring protocol, which could inadvertently reward certain output phrasings."],"forward_implications":["State-space LALMs can be trained faster and with less memory: the paper reports batch size can rise from 4 to 16 on the same GPUs, cutting training time from roughly three days to under two.","Total parameter count falls from 6.8B (hybrid with LLaMA) to 2.8B (ssLALM) with no loss in average close-ended performance, implying a feasible path to on-device audio assistants.","Because the state-space LLM used here is not instruction-tuned, the competitive results suggest instruction tuning may be less critical when a strong audio encoder provides grounding.","Removing the quadratic attention bottleneck in both modules positions the same architecture for long-form audio inputs where transformer LALMs hit memory limits."],"supporting_citations":[{"why":"DASS: the distilled audio state-space encoder used as the perception module for both proposed LALMs.","marker":"[19]"},{"why":"LTU: the transformer-based LALM baseline, source of the training pipeline and evaluation protocol.","marker":"[9]"},{"why":"Mamba: the state-space LLM backbone for the fully state-space ssLALM.","marker":"[17]"},{"why":"AST: the transformer teacher used to distill DASS and the audio encoder in LTU.","marker":"[31]"},{"why":"AudioSet-2M: the pretraining dataset for the DASS audio encoder.","marker":"[35]"},{"why":"GAMA: source of the reported comparison numbers for SALMONN, Pengi, and AudioGPT.","marker":"[12]"},{"why":"LLaMA: the transformer LLM used in the hybrid LALM and in LTU.","marker":"[36]"},{"why":"Pengi: a transformer-based LALM baseline compared in Table I.","marker":"[7]"}],"fun_headline_variants":["State-space LALM rivals transformer accuracy at 40% size","First all-state-space audio LLM matches 7B transformer","2.8B state-space LALM holds its own against 7B","State-space audio LLM matches transformer at 40% size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison stands or falls on whether cosine similarity between the model's output text and the class-label text is a fair measure of audio reasoning, a protocol that could favor particular phrasings.","fun_headline_variants_meta":{"raw":{"variants":["State-space LALM rivals transformer accuracy at 40% size","First all-state-space audio LLM matches 7B transformer","2.8B state-space LALM holds its own against 7B","State-space audio LLM matches transformer at 40% size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3531,"prompt_tokens":878,"completion_tokens":2653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":2577}},"tokens_in":494,"tokens_out":2653,"duration_ms":19792,"temperature":1.0,"reasoning_tokens":2577,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:00:06.059147+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-evaluate the same models on the same benchmarks using forced-choice label generation without a text-encoder similarity step, or with human raters judging the outputs; if transformer LALMs then clearly outperform ssLALMs on close-ended accuracy, the paper's competitive claim collapses.","supporting_citations":[{"cited_title":"Pengi: An audio language model for audio tasks,","cited_arxiv_id":null,"evidence_quote":"Pengi: a transformer-based LALM baseline compared in Table I."}],"review_version":1}