{"id":"cd53c94e-3fbf-4c12-845f-685445dc5654","arxiv_id":"2507.05729","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Replacing transformer temporal blocks with bidirectional Mamba gives competitive speech intelligibility prediction for hearing-impaired listeners with slightly fewer parameters.","lead":"The paper swaps the transformer blocks in a binaural speech intelligibility predictor for Mamba, a faster state-space model, and tests it on the Clarity Prediction Challenge 2 data. Accuracy stays about the same while parameter counts drop slightly, which could matter for hearing aid hardware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Efficiency motivation is unmeasured: the claimed low-latency/low-power benefit of Mamba applies only to the temporal block, not the frozen Whisper front-end that dominates inference; and the RMSE advantage over the baseline is not statistically significant.","rationale":"The central numerical claim—that Mamba-based temporal transform blocks give competitive RMSE with slightly fewer trainable parameters—is reported with sufficient detail to be credible: three or fifteen trials, public training code, and standard hyperparameters. My concern is not that the comparison is wrong but that the paper's framing overreaches. The efficiency motivation is never measured, and the spatial/contextual interpretation is built on a non-significant trend. These are addressable missing-evidence problems, not internal contradictions; they do not invalidate the competitive-accuracy result. I therefore keep the reader's conditional verdict: require runtime/memory benchmarks or remove the low-power motivation, and temper the significance and spatial-attribution language before acceptance.","tokens_in":8985,"tokens_out":8259,"duration_ms":95732,"concrete_test":"Fix an input duration and batch size (e.g., 5 seconds, batch 1) and measure end-to-end wall-clock latency and peak memory for the full pipeline—Whisper-Large-v2 feature extraction, temporal pooling, temporal/layer transform block, and SIP head—for the baseline transformer, unidirectional Mamba, and bidirectional Mamba on identical hardware. If the Mamba variants show no end-to-end latency or memory reduction (or are slower because Whisper dominates), the Section 1 motivation is unsupported; if in addition the paired 95% bootstrap CI for the RMSE difference (Bi-Mamba minus baseline) includes zero, the 'spatial information' interpretation should be removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 1 motivates the paper entirely by saying that self-attention is a computational bottleneck for low-latency, power-efficient devices and that Mamba's linear-time/constant-memory inference makes it suitable for low-computational and low-power devices. The paper contains no runtime, memory, or power measurement. The reported parameter reduction is limited to the trainable rear-end: Table 3 gives 5.01M parameters for Bi-Mamba versus 5.23M for the transformer baseline, a 0.22M difference, while the frozen Whisper-Large-v2 encoder (32 transformer layers, 1280-dim features) is executed at inference time and very likely dominates compute and memory. Additionally, Eq. (7) runs Mamba twice (forward and flipped), doubling temporal processing cost relative to unidirectional Mamba. The paper also claims in Section 5.2 that Bi-Mamba 'significantly improved' over the monaural model and 'effectively embeds contextual and spatial information,' but its own Wilcoxon signed-rank test (footnote 6) found no significant differences between models, so those interpretive claims exceed the evidence. What is supported is 'competitive performance at a slightly lower trainable parameter count'; what is not supported is the efficiency-based motivation for replacing the transformer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the transformer-based temporal transform blocks in the non-intrusive binaural speech intelligibility prediction (SIP) model E011 with Mamba blocks, and evaluates the resulting monaural and binaural models on the CPC2 dataset. The authors report that a binaural model using bidirectional Mamba achieves an average RMSE of 27.34, slightly lower than the transformer baseline's 27.49, while using fewer learnable parameters in the temporal block (5.01M vs. 5.23M). They also test LSTM baselines and vary the temporal pooling size of Whisper features. The central claim is that Mamba provides competitive prediction accuracy with a smaller temporal-block parameter count, and the authors further suggest that bidirectional Mamba captures contextual and spatial information from binaural signals.","tokens_in":9236,"tokens_out":3457,"duration_ms":40352,"significance":"If the reported accuracy holds, the paper demonstrates a credible drop-in replacement for transformer temporal blocks in a state-of-the-art SIP pipeline, using a public challenge dataset, a public baseline implementation, and official Mamba code. The work is reproducible in principle and addresses a practical question about efficient non-intrusive metrics for hearing-impaired listeners. However, the significance is limited in two ways. First, the efficiency motivation that frames the paper is never measured: no runtime, memory, or power figures are given, and the frozen Whisper feature extractor likely dominates inference cost. Second, the numerical advantage over the transformer baseline is small and, by the authors' own Wilcoxon signed-rank test, not statistically significant, so the interpretive claims about binaural information capture exceed the evidence.","major_comments":[{"comment":"The efficiency motivation is not supported by any measurement. Section 1 argues that self-attention is a computational bottleneck for low-latency, power-efficient hearing aids and that Mamba's linear-time, constant-memory inference makes it suitable for such devices, but the paper reports no runtime, memory, or power comparisons. Moreover, the frozen Whisper-Large-v2 feature extractor is executed for every input and likely dominates end-to-end inference cost, while the bidirectional Mamba in Eq. (7) requires two sequential passes over the sequence. The authors should either measure end-to-end inference latency and memory for the transformer and Mamba variants, or explicitly limit the efficiency claim to the temporal transform block rather than the full SIP pipeline.","section":"Section 1 and Eq. (7)"},{"comment":"The claim that the bidirectional Mamba model 'significantly improved' over the monaural model and 'effectively embeds contextual and spatial information' is inconsistent with the paper's own statistical test: footnote 6 states that the Wilcoxon signed-rank test found no significant differences between the models. The mean RMSE difference of 0.15 points in Table 3 is within the range of trial-to-trial variability. The authors should report confidence intervals or paired effect sizes, and should temper the interpretation to numerical trends unless a properly powered significance test supports the stronger claim.","section":"Section 5.2, Table 3, and footnote 6"},{"comment":"The tables report only mean RMSE and NCC values averaged over 3 and 15 trials, respectively, with no standard deviations or confidence intervals. Without dispersion measures, the reader cannot assess whether the reported differences, such as 28.25 vs. 29.45 in monaural Table 2 or 27.34 vs. 27.49 in binaural Table 3, are meaningful. At minimum, the authors should report standard deviations or per-trial intervals, especially since the paper already performs a Wilcoxon test that suggests high variability.","section":"Tables 2 and 3"},{"comment":"The conclusion that 'temporal fine structure component for speech perception... is widely distributed among Whisper features' is presented as a finding, but the experiment only shows that changing the pooling size has little effect on average RMSE in one dataset. This is a negative result without statistical testing or analysis of why the features behave this way. I recommend presenting Section 5.3 as an ablation observation and softening the mechanistic interpretation.","section":"Section 5.3, Table 4"}],"minor_comments":[{"comment":"The text contains a typo: 'Manba-based SIP model' should be 'Mamba-based SIP model'.","section":"Section 1"},{"comment":"Equation (3) is notationally imprecise: it writes '\\bar{A}, \\bar{B} = exp(\\Delta A), \\Delta B', but the discretization of a state-space model is not simply an exponential of the raw matrices. Clarify that this follows the zero-order hold discretization used in Mamba.","section":"Section 2"},{"comment":"The phrase 'applying an average pooling size p = 20' should read 'applying average pooling with a pooling size p = 20' to avoid ambiguity about what is pooled.","section":"Section 3.1.1"},{"comment":"The sentence 'All Whisper features were statistically normalized based on each train dataset' would benefit from specifying the normalization method (e.g., z-score with statistics computed on the training partition only).","section":"Section 4.2"},{"comment":"The text refers to 'CEC2.test.1' and 'CEC2.test.3' in the narrative; ensure the naming is consistent with the dataset partition labels used in Table 1 and elsewhere.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a compact challenge-oriented empirical comparison. Its main weakness is that the stated motivation and the strongest interpretive claims are not backed by measurements or statistical evidence, while the core competitive-accuracy result is plausible. If the authors add end-to-end efficiency experiments and either provide proper uncertainty quantification or soften the claims, the paper could become acceptable. I would not recommend rejection, because the central empirical comparison is clear and reproducible in principle."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper swaps transformer temporal blocks for Mamba in a non-intrusive binaural speech intelligibility prediction model for hearing-impaired listeners, evaluated on CPC2. It reports slightly better average RMSE at a slightly lower trainable parameter count (Bi-Mamba 5.01M vs transformer 5.23M; 27.34 vs 27.49 RMSE), plus a useful LSTM comparison and an honest temporal-pooling ablation. That is the real content, and it holds up.\n\nWhat it does well: public code and dataset, official Mamba implementation, 15 trials for binaural experiments, both RMSE and NCC reported, and the temporal-pooling result is reported even though it undercuts the expected 'higher resolution helps' story. The LSTM control is a good sanity check.\n\nThe soft spots are real but not fatal. The efficiency motivation — Mamba suits low-power devices — is never measured. There are no runtime, memory, or power numbers anywhere. The parameter saving is 0.22M in the trainable rear-end, while the frozen Whisper-Large-v2 front-end (32 transformer layers) dominates inference and the bidirectional Mamba runs two sequential passes. So the headline motivation is unsupported. Second, the paper's own Wilcoxon signed-rank test (footnote 6) found no significant differences between models, yet Section 5.2 says bidirectional Mamba 'significantly improved' over monaural and 'effectively embeds contextual and spatial information.' Those claims exceed the statistics. Third, there are no standard deviations or confidence intervals, so the 0.15 RMSE gap is within noise. These are addressable and do not break the central accuracy claim.\n\nWho it's for: people working on speech intelligibility metrics or efficient temporal modeling for hearing aids. It's a conference-level empirical contribution, not a breakthrough. It deserves a serious referee because the methodology is sound and the comparison is clean, but it needs revision: drop or soften the efficiency claims, add significance or variance reporting, and temper the interpretation.","headline":"A clean but small empirical swap of transformer for Mamba in a binaural SIP model; the accuracy claim holds, but the efficiency motivation and the spatial-information claims outrun the evidence.","tokens_in":9764,"tokens_out":2249,"would_cite":false,"duration_ms":22581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Mamba-based temporal transform block matches a transformer-based block in binaural speech intelligibility prediction while using fewer learnable parameters.","keywords":["speech intelligibility prediction","non-intrusive metric","Mamba","state-space model","binaural processing","hearing-impaired listeners","low-power devices","transformer"],"falsifier":"Measure end-to-end latency, peak memory, and energy for the full pipeline—frozen speech encoder plus temporal transform block—on the same low-power hearing-aid-class processor for both the transformer baseline and the bidirectional Mamba model; if the Mamba system is not faster or lighter, or if a paired significance test on a larger held-out set shows no RMSE difference, the paper's motivation and comparative claim would fail.","tokens_in":8775,"feed_emoji":"🦻","tokens_out":7850,"duration_ms":75394,"temperature":0.7,"pith_summary":"This paper tries to show that Mamba, a linear-time state-space sequence model, can replace the transformer self-attention blocks used for temporal processing in a non-intrusive binaural speech intelligibility prediction model. The motivation is that self-attention's quadratic cost is a bottleneck for low-latency, low-power hearing aids, and may hamper temporal processing. Replacing the transformer temporal block with a Mamba block yields prediction error that is competitive with, and on the binaural test set on average slightly better than, the transformer baseline while reducing learnable parameters. The authors also find that increasing temporal resolution of the input features does not improve Mamba's predictions, suggesting the relevant temporal information is already present in the frozen speech-encoder features.","feed_headline":"Mamba matches transformer on binaural speech intelligibility","feed_subtitle":"Bidirectional Mamba temporal blocks give equal accuracy at lower parameter count—a step toward low-power hearing aids.","key_machinery":"The central object is the Mamba block, a selective state-space model that updates a hidden state $\\mathbf{h}_t = \\bar{A}\\mathbf{h}_{t-1} + \\bar{B}\\mathbf{x}_t$ with input-dependent parameters $\\bar{A}$, $\\bar{B}$, $C$, and $\\Delta$, running the sequence scan in linear time with constant memory. The paper inserts this block as the temporal transform block in place of self-attention in the monaural model and in place of self-attention plus cross-attention in the binaural model. The binaural variant uses bidirectional Mamba, defined as the sum of a forward Mamba scan and a Mamba scan on the time-reversed sequence, followed by a skip connection and Gaussian error linear units, which is intended to capture interaural contextual and spatial information without an explicit cross-attention module. The rest of the pipeline—a frozen large speech-recognition encoder's layer features, temporal pooling, layer-wise transformer, and audiogram conditioning—is kept fixed, so the Mamba block is the only replaced component.","core_discovery":"On the binaural speech-intelligibility data used in this study, the bidirectional Mamba model achieves an average RMSE of 27.34 against the transformer baseline's 27.49, with 5.01 million versus 5.23 million learnable parameters, and gives the lowest average RMSE on two of the three test partitions and overall. The differences are not statistically significant by the paper's own Wilcoxon signed-rank test, so the paper's claim is that Mamba performs competitively, not that it outperforms. The binaural Mamba block replaces both self-attention and cross-attention with a forward and a time-reversed Mamba scan plus a skip connection and GELU nonlinearity, and the paper interprets the binaural model's larger improvement over its monaural version as evidence that bidirectional Mamba captures contextual and spatial binaural information. In monaural experiments, Mamba also lowers average RMSE relative to the transformer baseline (28.25 versus 29.45), while LSTM-based blocks do not.","pith_inferences":["The reported accuracy differences between Mamba and transformer are not statistically significant; a larger listener set or test set would be needed to establish that the two approaches are not interchangeable at the block level.","Because the frozen speech encoder likely dominates the full model's compute and memory, the end-to-end efficiency gain from swapping the temporal block could be small; measuring full-pipeline speed on hearing-aid-class hardware would settle whether the motivation holds.","The bidirectional Mamba uses two sequential passes over the sequence, so a unidirectional model with a larger state or a different binaural fusion rule might be a cheaper way to capture the same spatial context—a testable extension would be comparing Mamba fusion against simple inter-channel feature concatenation.","Since temporal pooling size had little effect, an adaptive or stride-based pooling schedule could reduce compute while retaining speech intelligibility accuracy, which the paper does not pursue."],"forward_implications":["Lower-parameter temporal processing is achievable: the binaural Mamba model's temporal transform block has about 5.0 million parameters versus the transformer's 5.2 million.","Binaural processing works without an explicit cross-attention module, with the bidirectional Mamba block plus skip connection and GELU appearing sufficient to leverage left-right spatial speech information.","Temporal resolution of the input features has little effect on Mamba-based prediction, so high-resolution temporal input is not needed to reach competitive accuracy.","Because Mamba scales linearly in sequence length, the temporal transform's cost should grow more slowly with longer inputs than self-attention's quadratic cost, which could benefit long-duration audio processing."],"supporting_citations":[{"why":"Defines the baseline transformer-based non-intrusive binaural speech intelligibility prediction architecture that the paper modifies.","marker":"[11]"},{"why":"Introduces the Mamba selective state-space model that replaces the transformer temporal blocks.","marker":"[21]"},{"why":"Provides the transformer and self-attention architecture whose quadratic cost motivates the replacement.","marker":"[12]"},{"why":"Supplies the frozen large-scale speech recognition encoder whose layer features are the model input.","marker":"[13]"},{"why":"Supports the use of Mamba for speech applications and the bidirectional formulation used here.","marker":"[22]"},{"why":"Supplies the temporal pooling setup and the Wilcoxon signed-rank test methodology used to compare models.","marker":"[18]"}],"fun_headline_variants":["Mamba ties transformer on binaural speech intelligibility","Binaural SIP: Mamba matches transformer at lower parameter count","Bidirectional Mamba matches transformer on binaural scores","Mamba rivals transformer on binaural intelligibility with fewer parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Mamba's theoretical linear-time, constant-memory inference advantage survives in the full model and that its prediction accuracy equals or improves on the transformer's, but the paper reports no runtime or memory measurements and its accuracy differences are not statistically significant.","fun_headline_variants_meta":{"raw":{"variants":["Mamba ties transformer on binaural speech intelligibility","Binaural SIP: Mamba matches transformer at lower parameter count","Bidirectional Mamba matches transformer on binaural scores","Mamba rivals transformer on binaural intelligibility with fewer parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3215,"prompt_tokens":904,"completion_tokens":2311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":2240}},"tokens_in":520,"tokens_out":2311,"duration_ms":19442,"temperature":1.0,"reasoning_tokens":2240,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:18:20.516635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure end-to-end latency, peak memory, and energy for the full pipeline—frozen speech encoder plus temporal transform block—on the same low-power hearing-aid-class processor for both the transformer baseline and the bidirectional Mamba model; if the Mamba system is not faster or lighter, or if a paired significance test on a larger held-out set shows no RMSE difference, the paper's motivation and comparative claim would fail.","supporting_citations":[{"cited_title":"Nonintrusive objective measurement of speech intelligibility: A review of methodology,","cited_arxiv_id":null,"evidence_quote":"Defines the baseline transformer-based non-intrusive binaural speech intelligibility prediction architecture that the paper modifies."},{"cited_title":"ASR-based speech intelligibility prediction: A review,","cited_arxiv_id":null,"evidence_quote":"Provides the transformer and self-attention architecture whose quadratic cost motivates the replacement."},{"cited_title":"Clarity-2021 Challenges: Machine Learning Challenges for Advancing Hearing Aid Pro- cessing,","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen large-scale speech recognition encoder whose layer features are the model input."},{"cited_title":"Unsupervised Uncertainty Mea- sures of Automatic Speech Recognition for Non-intrusive Speech Intelligibility Prediction,","cited_arxiv_id":null,"evidence_quote":"Supports the use of Mamba for speech applications and the bidirectional formulation used here."},{"cited_title":"Attention is All you Need,","cited_arxiv_id":null,"evidence_quote":"Supplies the temporal pooling setup and the Wilcoxon signed-rank test methodology used to compare models."}],"review_version":1}