{"id":"b4512db3-4845-4315-b440-ed8894f31d00","arxiv_id":"2509.15680","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 2.7B-parameter Mamba-2 based audio captioning model (MAC) matches or surpasses larger transformer-based audio-language models on several zero-shot classification and captioning benchmarks, with additional design-space analysis.","lead":"The full text of arXiv 2509.15680 is a Mamba-2 audio captioning paper (MAC), not the SAM system described in the attached abstract, and it reports that a 2.7B-parameter Mamba-2 model matches or beats larger transformer-based audio-language models on several benchmarks. A generalist reader might care because state-space backbones offer linear-time inference, so efficient audio-language models could reduce compute for long-audio understanding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Classification benchmarks rest on an unspecified CLAP-similarity protocol; until label-to-text mapping and aggregation are specified and validated, the headline comparisons in Table 1 are not established.","rationale":"The reader's weakest assumption is exactly the CLAP-based classification evaluation. I agree that this is the most load-bearing concern: the central comparative claim is quantified almost entirely through those columns. Without a specified, validated protocol, the numbers cannot be checked, and the comparison to LTU/GAMA/ssLALM may be unfair. I also flag the abstract/body number mismatch (SAM vs MAC; 21.1/17.6 vs 19.5/16.8) as an independent consistency problem, but the CLAP protocol is the deeper threat to the argument because it undermines the entire classification evidence base, not just one number. The proposed concrete test—release the script/checkpoint and reproduce a single classification column with an explicit label-template and aggregation rule—would settle whether the concern lands. I do not see a reason to move the reader's CONDITIONAL verdict; the concern supports keeping acceptance conditional on this clarification and, ideally, on code release. The paper contains useful internal ablations (LoRA rank, connector design, encoder fine-tuning) that would remain interesting even if the headline benchmark claim is weakened, so outright rejection is not warranted by this concern alone.","tokens_in":8432,"tokens_out":5113,"duration_ms":43056,"concrete_test":"Ask the authors to release the exact evaluation script and the MAC-2.7B checkpoint, then reproduce every classification column of Table 1 from raw generated captions. The script must specify: the prompt(s) used, the text form of each class label, the CLAP similarity pooling rule, and the mapping from similarity scores to accuracy/mAP for each dataset. Independently rerun the same script on the released LTU-7B and GAMA-7B checkpoints and compare with their published numbers. If the reproduced AudioSet mAP or ESC-50 accuracy differs materially (more than ~1 point) under a standard label-template and aggregation scheme, or if the baseline numbers cannot be reproduced with the same script, the headline comparison is not established. Also reconcile the abstract's 21.1/17.6 numbers with the table before relying on any flagship result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 3 — that MAC-2.7B (E6) achieves superior performance and surpasses larger transformer-based ALMs — is carried by Table 1. The classification columns (ESC-50, DCASE, VS, TUT, BJO, VGG, FSD, AudioSet) rest on the single sentence: 'we used the CLAP encoder to measure the similarity between the generated captions and the ground-truth labels.' This sentence does not specify how class labels are converted to text, whether templates are used, which generated caption per clip is scored, how per-clip similarities are reduced to accuracy or mAP, or whether the same protocol produced the baseline numbers. AudioSet mAP is a multi-label, 527-class metric; it requires per-class scores and a threshold or ranking rule, not one caption-to-label similarity. If the CLAP protocol is ad hoc or differs from the protocol used for LTU/GAMA/ssLALM, the claimed superiority is an artifact of the evaluation, not of the model. The concern is compounded by the abstract reporting SAM-2.7B numbers (21.1 mAP AudioSet, 17.6 SPICE AudioCaps) that match no row in Table 1 (E6: 19.5 and 16.8), and by the absence of released code or evaluation scripts. This is a missing-support problem in the evidence for the main claim, not a disagreement with the SSM-vs-transformer design choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MAC, a Mamba-2-based audio-language model (ALM) that combines an EAT audio encoder with Mamba-2 language models of sizes 130M, 780M, and 2.7B, using an MLP connector and LoRA finetuning. The authors report zero-shot classification results on eight audio benchmarks and captioning results on AudioCaps and Clotho, claiming that MAC-2.7B matches or surpasses larger transformer-based ALMs such as LTU-7B and GAMA-7B. They also present ablations on whether to finetune the audio encoder, how to initialize it, connector designs that vary token length and dimensionality, and analyses of effective rank and token similarity as functions of model size.","tokens_in":8810,"tokens_out":3809,"duration_ms":35075,"significance":"If the reported results are valid, this is a useful contribution: it is among the first studies of Mamba-2 as an ALM backbone, and the ablations on audio-encoder finetuning, initialization, and connector design provide practical guidance for building SSM-based audio-language models. The eRank and token-similarity analyses are a thoughtful attempt to understand why SSM backbones behave differently from transformers. The paper also reports consistent gains from increasing LoRA rank and model size, and it compares against strong baselines. However, the central empirical claim is currently supported only by an underspecified CLAP-similarity evaluation protocol and by comparisons to baselines whose training data and budgets are not matched; the abstract also reports numbers that do not appear in Table 1. These issues must be resolved before the headline conclusions can be accepted.","major_comments":[{"comment":"The classification evaluation is defined by a single sentence: 'we used the CLAP encoder to measure the similarity between the generated captions and the ground-truth labels.' This is insufficient to support the classification columns of Table 1. The paper must specify how class labels are converted to text (e.g., prompt templates, whether labels are verbatim), how per-clip similarities are aggregated into accuracy or mAP, and how multi-label datasets such as AudioSet (527 classes) are handled—mAP requires per-class scores and a ranking or threshold rule, not one caption-to-label similarity. It is also unclear whether the baseline numbers for LTU-7B, GAMA-7B, and ssLALM were computed with the same CLAP-based protocol or were taken from papers that used a different evaluation, which would make the comparisons inequitable.","section":"Section 3, Evaluation method and Table 1"},{"comment":"The claimed superiority over larger transformer ALMs is not yet established because training conditions are not matched. The text says the models are trained on OpenAQA with the LTU curriculum, batch size 256, and two learning rates, but the baseline models were trained on datasets that 'include the OpenAQA dataset'—different data mixtures, numbers of steps, and hardware budgets. Without a controlled comparison (same data, same number of updates, or at least a clear statement of what differs), the gains in Table 1 could be due to data or compute rather than to Mamba-2. The authors should either provide matched-training comparisons or soften the 'superior performance' claim accordingly.","section":"Section 3, Model training and Table 1"},{"comment":"There is a direct inconsistency between the abstract/title and the body. The abstract presents 'SAM-2.7B' as achieving 21.1 mAP on AudioSet and 17.6 SPICE on AudioCaps, and the title announces 'SAM: A Mamba-2 State-Space Audio-Language Model.' The body, however, names the model MAC and reports in Table 1 (E6) 19.5 mAP on AudioSet and 16.8 SPICE on AudioCaps. No row in Table 1 matches the abstract's numbers. The authors must reconcile the naming and verify that all reported numbers correspond to the same configuration, because as written the reader cannot tell which numbers are authoritative.","section":"Abstract and title versus Section 3/Table 1"},{"comment":"The representation-level analysis is presented as evidence for the claim that larger LLMs produce richer audio representations, but the link to the performance numbers is descriptive rather than quantitative. Table 3 reports eRank values for a few configurations and states that larger LLMs increase eRank, but no correlation or regression is given, and the eRank values for MAC-780M (19.45/19.83/18.80) and MAC-2.7B (19.78) are close, while their classification averages differ substantially (50.6 vs. 53.9). The authors should either qualify the claim as a hypothesis or provide a more direct analysis. Additionally, Equation (5), which defines the covariance matrix used for eRank, is garbled in the manuscript and should be rewritten.","section":"Section 4, eRank analysis"}],"minor_comments":[{"comment":"Equation (5) is typeset incorrectly: the covariance expression is missing parentheses and the outer-product structure is unclear. It should be written as a standard sample covariance matrix (1/N) Σ (h_i − h̄)(h_i − h̄)ᵀ.","section":"Equation (5)"},{"comment":"The table uses dagger symbols (†) for FSD and AudioSet, but the footnote text defining 'weak zero-shot' (ZS-) does not explain the daggers; the distinction between ZS- and ZS needs a clear definition and a list of which datasets fall into each category.","section":"Table 1, footnote symbols"},{"comment":"The 'Classif. Avg.' column in Table 1 is not defined; the paper should state whether it is a simple mean, a weighted mean, or another aggregation of the eight classification metrics.","section":"Section 3, Evaluation method"},{"comment":"In Table 4, the Audio-Flamingo 2 (AF-2) rows report model sizes 500M, 1.5B, and 3B, but the text does not specify how the AF-2 token-similarity values were obtained—whether from public checkpoints, from the authors' own runs, or from the original paper—nor how many audio clips were used for the average.","section":"Section 4, Token similarity"},{"comment":"The text references 'Figure 2' for the state update distance comparison, but the figure content is not included in the manuscript text supplied for review.","section":"Section 4, Figure 2"},{"comment":"The paper should cite a public source or give a fuller description of the OpenAQA dataset, since it is central to the training setup and the comparisons.","section":"Section 3, Model training"}],"recommendation":"major_revision","confidential_remarks":"The mismatch between the arXiv abstract (SAM, 21.1 mAP AudioSet, 17.6 SPICE) and the body (MAC, 19.5 mAP, 16.8 SPICE in Table 1) is concerning and should be checked carefully; it may indicate a last-minute rename or an unupdated table. Also, neither code nor evaluation scripts are released, which makes the underspecified CLAP protocol difficult to verify. The paper is otherwise within scope for a speech/audio conference or journal, and the ablated experiments are valuable, but the evaluation protocol needs to be nailed down before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before reading this one. The body is a decent design-space study of Mamba-2 for audio captioning, with some genuinely informative ablations. But the abstract on the arXiv page describes a 'SAM-2.7B' with numbers that match no row in Table 1, and the classification results that carry the main claim rest on a single sentence about a CLAP-similarity protocol that is never actually specified.\n\nWhat is new: this appears to be the first Mamba-2-based audio captioning model, and the systematic comparison of connectors that exploit linear-time sequence processing is a useful addition. The eRank and token-cosine diagnostics are new to audio-language modeling and give real insight into how SSMs handle audio tokens. The ablations are well structured: training the audio encoder helps, AudioSet-finetuned initialization beats masked-only pretraining, and larger LLMs produce richer audio representations. These are plausible empirical results, and the qualitative examples suggest the model actually generates reasonable captions.\n\nThe soft spots are concentrated in the evaluation. The CLAP protocol is underspecified to the point of unverifiable: no label-to-text mapping, no aggregation rule, no handling of multi-label AudioSet mAP. The baselines were likely evaluated with different protocols, so the claimed superiority of MAC-2.7B over LTU-7B and GAMA-7B is not established. The abstract mismatch is a second, independent problem: the headline numbers 21.1 mAP and 17.6 SPICE do not appear anywhere in the body; E6 reports 19.5 and 16.8. That inconsistency makes it hard to know which result the authors actually stand behind. Training data and budgets are asserted rather than matched, and no code is released. These issues do not sink the design-space analysis, but they do sink the flagship comparison as currently presented.\n\nThe audience is researchers working on efficient audio-language models or on SSM backbones for multimodal tasks. The design-space analysis and the representation diagnostics are worth reading even if the benchmark numbers are provisional. I would send this to referees, but with a direct request to specify and validate the CLAP protocol, reconcile the abstract with the tables, and release code. It is a major-revision paper, not a desk reject.","headline":"Worth a serious look, but the flagship claim is undercut by an underspecified CLAP evaluation and an abstract that does not match the paper's own tables.","tokens_in":9242,"tokens_out":2333,"would_cite":false,"duration_ms":20736,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 2.7B-parameter Mamba-2 audio-language model matches or beats larger 7B transformer models on audio understanding benchmarks.","keywords":["audio-language model","state-space model","Mamba-2","audio captioning","zero-shot audio classification","multimodal LLM","representation analysis","LoRA"],"falsifier":"Re-run the zero-shot classification benchmarks with a standard protocol—for example, a linear probe on EAT features, or top-1 matching of class-label CLAP embeddings to the audio—and check whether MAC-2.7B still ranks above LTU-7B, GAMA-7B, and ssLALM-2.8B. If the ranking changes, the central parity claim fails. Alternatively, evaluate the same model on a long-form audio QA benchmark with minutes-long clips to test the linear-time scaling benefit.","tokens_in":8211,"feed_emoji":"🎧","tokens_out":8458,"duration_ms":64664,"temperature":0.7,"pith_summary":"This paper tries to establish that a Mamba-2 state-space backbone can power an audio-language model that matches or beats much larger transformer-based models on zero-shot audio classification and audio captioning, at 2.7B parameters versus 7B baselines. Its evidence comes from a systematic design-space exploration: jointly finetuning the audio encoder, raising the LoRA rank, and using compact audio token representations. The paper also claims that despite SSMs' linear-time scaling, they profit more from short, information-dense audio token sequences than from long uncompressed ones, and that instruction-following supervision substantially improves reasoning, raising MMAU-Sound accuracy from 22.8 to 56.8. If right, this would make state-space models a practical, cheaper alternative for audio-language systems.","feed_headline":"2.7B state-space audio model matches 7B transformers","feed_subtitle":"Mamba-2 backbone with a tuned audio encoder tops AudioSet and AudioCaps at a fraction of the parameters.","key_machinery":"The load-bearing object is the Mamba-2 backbone, a state-space model that processes sequences in linear time via a selective scan implemented through structured state-space duality. The argument runs on three connected mechanisms: the EAT audio encoder and whether it is finetuned; the connector, which either compresses 512 tokens to 64 high-dimensional tokens by concatenation or keeps all tokens in time-major or frequency-major layouts; and two representation-level diagnostics—effective rank and pairwise cosine similarity of audio tokens—used to explain why larger models and shorter token sequences work better.","core_discovery":"The central claim is that a 2.7B-parameter Mamba-2 language model, paired with an EAT audio encoder and a two-layer MLP connector, performs on par with or better than 7B transformer-based audio-language models such as LTU-7B and GAMA-7B across zero-shot audio classification and captioning benchmarks. The supporting analysis contends that joint finetuning of the audio encoder is essential—freezing it costs several accuracy points—and that initializing the encoder from class-level supervised AudioSet finetuning transfers better than masked-modeling-only pretraining. The paper further claims that for SSMs, compressing 512 audio tokens into 64 concatenated tokens outperforms feeding all 520–576 tokens, because the recurrent state must carry information across each token update; this pressure shows up as higher token-level cosine similarity in smaller models. Finally, adding instruction-following supervision is reported to improve sound-reasoning accuracy from 22.8 to 56.8 on MMAU-Sound.","pith_inferences":["If the linear-time advantage carries over to longer audio, the same recipe should extend to minute- or hour-long recordings where transformer attention becomes impractical; this is the natural next stress test.","The evidence that full-length tokens hurt suggests that a learned compression connector (e.g., token merging or a query-based bottleneck) might push SSM audio-language models further by combining compression with semantic selection.","Because the classification evaluation relies on CLAP similarity between generated captions and class labels, the parity claim against 7B transformers should be re-checked with a standard classifier head; the authors' own protocol is the main unvalidated link.","The representation-level findings (effective rank growth with model size, cosine-similarity pressure with sequence length) likely transfer to other SSM-based multimodal models, predicting that smaller SSMs will compress visual tokens in the same way."],"forward_implications":["A 2.7B Mamba-2 audio-language model can match 7B transformer baselines on zero-shot classification and captioning, making state-space backbones a viable scaling path with fewer parameters.","Jointly finetuning the audio encoder is required; freezing it degrades performance across model sizes and connector types.","For SSM audio-language models, compact audio tokens (64) beat full-length 520–576 token sequences, so linear-time complexity does not make long token sequences free.","Raising the LoRA rank from 8 to 256 substantially improves accuracy, particularly for the smallest 130M model.","Adding instruction-following supervision can sharply improve sound-reasoning ability, as measured by MMAU-Sound."],"supporting_citations":[{"why":"Provides the curriculum-learning training strategy and the LTU evaluation protocol, and supplies the LTU-7B baseline the paper compares against.","marker":"[5]"},{"why":"Supplies the GAMA-7B baseline and its Q-Former connector design.","marker":"[6]"},{"why":"The prior state-space audio-language model baseline (ssLALM-2.8B) that this work extends and compares with.","marker":"[16]"},{"why":"The EAT audio encoder used in all experiments, pretrained on AudioSet-2M.","marker":"[18]"},{"why":"The Mamba-2 backbone definition, including structured state-space duality and its linear-time training.","marker":"[12]"},{"why":"The CLAP encoder used to compute similarity between generated captions and ground-truth labels in the classification evaluation.","marker":"[35]"},{"why":"Defines the SPICE metric used to score audio captioning.","marker":"[32]"},{"why":"Defines effective rank, used to analyze audio token diversity across model sizes.","marker":"[37]"},{"why":"Supplies the LoRA adapters whose rank is a key design variable.","marker":"[23]"}],"fun_headline_variants":["2.7B Mamba-2 audio model beats 7B transformers","Mamba-2 audio: compact tokens, tuned encoder, big wins","Instruction tuning lifts Mamba-2 audio reasoning to 56.8","State-space audio model: fewer params, more punch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation of zero-shot classification benchmarks relies on the CLAP encoder measuring similarity between generated captions and ground-truth labels, and if that protocol does not reproduce standard classifier accuracy, the headline comparison against the 7B transformer baselines is not established.","fun_headline_variants_meta":{"raw":{"variants":["2.7B Mamba-2 audio model beats 7B transformers","Mamba-2 audio: compact tokens, tuned encoder, big wins","Instruction tuning lifts Mamba-2 audio reasoning to 56.8","State-space audio model: fewer params, more punch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1353,"prompt_tokens":926,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":351}},"tokens_in":542,"tokens_out":427,"duration_ms":4025,"temperature":1.0,"reasoning_tokens":351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:49:34.541267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the zero-shot classification benchmarks with a standard protocol—for example, a linear probe on EAT features, or top-1 matching of class-label CLAP embeddings to the audio—and check whether MAC-2.7B still ranks above LTU-7B, GAMA-7B, and ssLALM-2.8B. If the ranking changes, the central parity claim fails. Alternatively, evaluate the same model on a long-form audio QA benchmark with minutes-long clips to test the linear-time scaling benefit.","supporting_citations":[{"cited_title":"Fsd50k: An open dataset of human-labeled sound events,","cited_arxiv_id":null,"evidence_quote":"Defines effective rank, used to analyze audio token diversity across model sizes."},{"cited_title":"On the parameterization and initialization of diagonal state space models,","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA adapters whose rank is a key design variable."},{"cited_title":"For future work, we plan to investigate the effects of advanced connector designs that enable token mixing and Mamba-Transformer hybrid architectures in audio language modeling","cited_arxiv_id":null,"evidence_quote":"Provides the curriculum-learning training strategy and the LTU evaluation protocol, and supplies the LTU-7B baseline the paper compares against."},{"cited_title":"GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities,","cited_arxiv_id":null,"evidence_quote":"Supplies the GAMA-7B baseline and its Q-Former connector design."},{"cited_title":"Mellow: a small audio language model for reasoning,","cited_arxiv_id":null,"evidence_quote":"The prior state-space audio-language model baseline (ssLALM-2.8B) that this work extends and compares with."},{"cited_title":"Eat: Self-supervised pre-training with efficient audio transformer,","cited_arxiv_id":null,"evidence_quote":"The EAT audio encoder used in all experiments, pretrained on AudioSet-2M."},{"cited_title":"Transformers are ssms: generalized models and efficient algorithms through structured state space duality,","cited_arxiv_id":null,"evidence_quote":"The Mamba-2 backbone definition, including structured state-space duality and its linear-time training."},{"cited_title":"V ocalsound: A dataset for improving human vocal sounds recognition,","cited_arxiv_id":null,"evidence_quote":"The CLAP encoder used to compute similarity between generated captions and ground-truth labels in the classification evaluation."},{"cited_title":"Flashattention-2: Faster attention with better par- allelism and work partitioning,","cited_arxiv_id":null,"evidence_quote":"Defines the SPICE metric used to score audio captioning."}],"review_version":2}