{"id":"32a82b47-a5e2-4627-a9f4-f917810f6adc","arxiv_id":"2506.02181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A modern Conformer ASR model's predictions are tied to vowel formants (F1 and F2), sibilant fricative spectral peaks, and plosive release bursts, per SPES feature attributions on TIMIT.","lead":"This paper uses a saliency attribution method to identify which audio spectrogram regions a modern Conformer-based speech recognizer relies on when recognizing English vowels, fricatives, and plosives. It finds the model leans on vowel formants F1 and F2, the noise spectra of sibilant fricatives, and the release bursts of plosives, echoing human phonetic cues.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Top-3% saliency threshold chosen by manual inspection is not validated; all time-coverage and spectral-match conclusions may shift or vanish under alternate thresholds or a chance baseline.","rationale":"The study is a descriptive interpretability analysis whose conclusions are inherently comparative: vowels are attended more than consonants, sibilants more than non-sibilants, release more than closure, F1/F2 more than F3/F4. All comparisons are made on binary maps obtained by a single ad-hoc cutoff. If the cutoff is moved, the number of salient bins per frame changes linearly, directly altering TC's chance level and SM's hit rate. The absence of a null model is equally important: with 80 mel channels, top 3% means ~2.4 salient bins per frame, so a random map would show high TC for any long segment; the vowel 'full time span' result could partly reflect duration. The authors aggregate across many phoneme instances, which mitigates noise, but aggregation does not remove threshold bias. The most direct check is a threshold sweep plus a permutation null. If patterns are stable, the conclusions are robust; if not, the headline claims would need to be qualified or dropped. The reader's CONDITIONAL verdict already requires such robustness checks, so our stress test does not move the verdict.","tokens_in":8936,"tokens_out":3542,"duration_ms":34421,"concrete_test":"Recompute TC (Fig. 2) and SM (Tables 1-2) using the released code at thresholds 0.5%, 1%, 2%, 5%, and 10%, and also with saliency maps randomly shuffled across time-frequency bins (preserving the number of salient elements) as a null model. The central claims survive only if the rank order of vowel vs. consonant coverage, sibilant vs. non-sibilant, release vs. closure, and F1/F2 vs. F3/F4 is unchanged across thresholds and exceeds the random baseline by a margin comparable to the reported effect sizes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 states that S' retains 'the top 3% of elements' based on 'manual inspection of the saliency distribution,' with no sensitivity analysis. Every headline result—vowels' near-100% time coverage, plosive release>closure, sibilant>non-sibilant spectral match, and F1/F2>F3/F4—is computed on this single binarization. The threshold is especially consequential because TC is defined as the fraction of frames with at least one salient bin: with F=80 mel channels, top 3% yields ~2.4 salient bins per frame on average, so even a random or diffuse map gives high TC for long phonemes. Similarly, SM is the percentage of formant/peak values that fall in a salient bin; changing the cutoff changes the base rate of a 'hit' for all phoneme classes. Without sweeping the threshold and comparing against a random-attribution null, the reported class differences cannot be distinguished from threshold-dependent artifacts. The paper's own limitation section does not mention this dependency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper applies the SPES feature-attribution method to a 113M-parameter Conformer-based ASR model trained on CommonVoice, LibriSpeech, TED-LIUM, and VoxPopuli, and analyzes saliency maps on error-free TIMIT SX sentences. It defines time coverage (TC) and spectral match (SM) to test whether the model's salient input regions align with phonetic cues: full vowel duration and F1/F2 formants, sibilant frication spectra, and plosive release bursts. The main claims are that vowels receive near-complete time coverage, sibilants are better matched spectrally than non-sibilants, plosive release is more salient than closure, and alignment is stronger for male than female vowels.","tokens_in":9270,"tokens_out":5844,"duration_ms":53861,"significance":"The study is a useful step beyond earlier analyses of outdated architectures, since it evaluates a modern Conformer/Transformer model on a wider phoneme inventory and provides code. The TC/SM metrics are simple and interpretable, and several qualitative patterns (e.g., low SM for /f/ and /v/, high SM for /S/) are visible in the tables. However, the headline time-coverage result is confounded by the arbitrary top-3% threshold and the lack of a chance baseline, and the gender and class differences are not supported by significance testing. If the threshold sensitivity and statistical issues are addressed, the conclusions about phonetic alignment would be genuinely informative for the interpretability community.","major_comments":[{"comment":"The binary map S' retains the top 3% of elements based on 'manual inspection', and this single threshold drives the time-coverage results in Figure 2. Because F=80 (Section 3), a null map with the same number of salient bins placed uniformly at random would already produce an expected TC of 1-(0.97)^80 ≈ 91% per frame. The near-100% values for vowels are therefore close to chance, and the word-level reference line in Figure 2 is not a chance baseline. Please report TC and SM across a range of thresholds (e.g., 0.5%, 1%, 2%, 5%, 10%) and against a permutation null that preserves the total number of salient bins, and state whether the vowel/sibilant/release patterns are robust.","section":"Section 2, TC definition and Figure 2"},{"comment":"The claims that the model captures sibilant spectra better than non-sibilants, that release bursts differ by place of articulation, and that vowel saliency is stronger for male speech rest on point estimates without confidence intervals or significance tests. For example, average F2 spectral match is 73.7 for men and 72.4 for women, a difference unlikely to be reliable without error bars; the gender claim in the abstract is directly based on such small differences. Add per-instance bootstrap or permutation confidence intervals (or at least effect sizes and counts) for the main TC and SM comparisons.","section":"Section 4.2, Tables 1 and 2"},{"comment":"The analysis is restricted to the 2,191 error-free TIMIT SX predictions out of 3,150, while the abstract and conclusions are phrased about the model in general ('the ASR model relies...'). Please either limit the claims to correctly predicted utterances or analyze whether the pattern changes on the full set, including errors, which would also speak to the robustness gaps the introduction motivates.","section":"Section 3, Data paragraph"}],"minor_comments":[{"comment":"The introduction contains typos: 'ona wider range' and 'bya modern' should be 'on a wider range' and 'by a modern'.","section":"Section 1"},{"comment":"The phrase 'mean-standard normalization' is ambiguous; it should be described as z-score normalization, or the specific normalization formula should be given.","section":"Section 2"},{"comment":"The tables do not report the number of phoneme instances per cell; uneven counts across phonemes and genders could affect the averages, so counts (or a statement that counts are balanced) should be provided.","section":"Tables 1 and 2"},{"comment":"The 'average coverage for a single word' reference line is not defined precisely; state how word-level TC was computed and whether it uses the same binary threshold.","section":"Figure 2"},{"comment":"The Limitations section does not mention the sensitivity of all quantitative results to the top-3% threshold or the restriction to error-free predictions; these are more consequential for the paper's claims than the omitted spectral moments.","section":"Section 5, Limitations"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a solid descriptive interpretability study, and it deserves a serious referee. The new part is real: the authors apply SPES to a modern 113M Conformer encoder-decoder trained on multiple corpora, and they map saliency over a much wider phoneme inventory than previous work—11 vowels, 8 fricatives, 6 plosives—with clean TIMIT alignments and gender-stratified spectral matches. Code is released. The qualitative findings align with phonetic knowledge: near-full time coverage for vowels with emphasis on F1/F2, better spectral match for sibilant than non-sibilant fricatives, and release over closure for plosives with burst characteristics. Those patterns are visible in the distributions and are probably right.\n\nThe soft spots are real but manageable. The 3% saliency threshold is the load-bearing joint. It was chosen by manual inspection, with no sensitivity analysis or random baseline. With 80 mel bins, a random 3% map puts at least one salient bin in roughly 91% of frames, so the time-coverage metric has a high base rate. If the threshold moves, the class differences could compress or shift. The paper's own limitation section doesn't mention this dependency. Second, all results come from the 2191 error-free TIMIT SX sentences; the choice is reasonable, but the excluded 30% could behave differently and the paper doesn't discuss it. Third, the gender and phoneme-class claims have no confidence intervals or significance tests; some of the male/female gaps in Table 1 look small or reversed (e.g., F2 for /0/ and /u/), so the \"greater saliency in male speech\" abstract line is a trend that needs support, not a firm finding.\n\nNone of this undermines the core picture. SPES is a published attribution method, independent of these phonetic results, so nothing circular is going on. The paper is a useful map, not a paradigm shift. I'd send it to peer review with a request for threshold sensitivity analysis, a chance baseline, and at least simple bootstraps or sign tests on the key comparisons. That's a standard revision, not a rejection.\n\nWho it's for: ASR interpretability researchers and anyone interested in whether modern attention-based models use human-perceptually relevant cues. I'd bring it to a reading group, though I wouldn't necessarily cite it in my own work unless I needed these specific phoneme-level numbers.\n\nRecommendation: engage with it. Send it out.","headline":"A solid descriptive interpretability study of a Conformer ASR model, but the manually chosen 3% saliency threshold and missing statistical tests keep it at 'plausible' rather than 'established'.","tokens_in":9632,"tokens_out":3076,"would_cite":false,"duration_ms":28585,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A modern Conformer-based ASR model relies on the same spectro-temporal cues that matter to human listeners: vowels' first two formants, sibilant spectra, and plosive bursts.","keywords":["automatic speech recognition","feature attribution","explainability","saliency maps","acoustic phonetics","Conformer","spectral match","TIMIT"],"falsifier":"On the same TIMIT SX subset, recompute time coverage and spectral match at saliency cutoffs of 1% and 10%; if the vowel-over-consonant and sibilant-over-non-sibilant gaps invert or vanish, the conclusions are an artifact of the 3% threshold rather than of model behavior.","tokens_in":8781,"feed_emoji":"🎙️","tokens_out":5725,"duration_ms":48210,"temperature":0.7,"pith_summary":"This paper tries to establish what a state-of-the-art, Conformer-based English ASR model actually listens to, by attributing each predicted word's importance to specific spectrogram elements and checking those saliency maps against phonetic knowledge. The authors find that the model relies on vowels over their full duration, weighting the first two formants (F1 and F2) more than higher formants; that it captures the spectral peaks of sibilant fricatives far better than those of non-sibilant fricatives; and that in plosives it prioritizes the release phase, especially burst characteristics. These alignments are strongest when the acoustic cue is spectrally well defined and weaker for diffuse cues, suggesting the model's listening strategy is closer to human phonetics than earlier architectures were thought to be.","feed_headline":"Speech AI listens to vowels' formants and plosive bursts","feed_subtitle":"Saliency maps show a modern ASR model tracks the same acoustic cues human listeners use.","key_machinery":"The load-bearing object is SPES (Spectrogram Perturbation for Explainable Speech-to-Text), a feature attribution method that clusters spectrogram elements by energy, perturbs those clusters to measure their impact on each predicted subword, and outputs a saliency map per token. The paper binarizes each map to the top 3% of salient elements and aligns them with TIMIT phonetic segmentation, then computes two metrics: time coverage (TC), the share of a phoneme's frames that contain at least one salient element, and spectral match (SM), the share of measured formants or spectral peaks that fall on salient elements. These two metrics carry the entire argument: all differences between vowels and consonants, sibilants and non-sibilants, and closure versus release are read off them.","core_discovery":"On the paper's own terms, the central discovery is that a modern 113M-parameter Conformer encoder-decoder ASR model, trained on four large English corpora and tested on TIMIT, assigns prediction-relevant saliency in a way that tracks established acoustic-phonetic cues. For vowels, time coverage approaches 100% and spectral match is highest for F1 and F2 (averaging 71.4/64.8 and 73.7/72.4 for male/female speakers) and lower for F3 and F4, with systematically higher scores for male vowels. For fricatives, sibilants such as /s, z, S, Z/ show strong spectral-match scores and clear high-frequency saliency peaks, while non-sibilants such as /f, v, T, D/ show flat, low-density saliency; for plosives, saliency concentrates on the release phase rather than the closure, and burst spectral peaks are matched for velar and alveolar stops but poorly for labials. The authors conclude that the model's reliance on acoustic cues aligns with human speech perception where cues are well defined and degrades where they are not.","pith_inferences":["The top-3% threshold is the main knob: re-running the same pipeline at other cutoffs would reveal whether the reported phoneme-class contrasts are a property of the model or an artifact of binarization.","If the alignment-with-human-phonetics interpretation is right, then models trained with phonetic inductive biases (e.g., formant-based augmentation) should show even sharper saliency, while adversarially masking F1/F2 should expose the model's dependence.","The same attribution pipeline applied to a self-supervised speech encoder or a different decoder family would separate architectural effects from training-data effects: whether the Conformer, the CTC/decoder objective, or the English corpora produce the pattern.","The gender asymmetry could be probed by resynthesizing identical vowels with male vs female F0/formant values to see if the saliency shift is physiological rather than sociolinguistic."],"forward_implications":["If the central claim holds, ASR interpretability tools can move from neuron-level probing to phonetically meaningful statements: explaining a prediction means pointing to a formant, a frication noise band, or a burst.","The model's strong reliance on F1 and F2 predicts that perturbing or masking those formant regions will degrade recognition more than perturbing F3/F4 or consonant closures, which can be tested directly.","Because saliency for male vowels systematically exceeds that for female vowels, the paper predicts a link between feature-attribution patterns and known gender performance gaps in ASR.","Weak spectral match for non-sibilant fricatives and labial plosives implies those phoneme classes are the likeliest failure points under noise or channel conditions that blur their diffuse cues."],"supporting_citations":[{"why":"Defines SPES, the feature attribution method that produces the saliency maps on which the entire analysis relies.","marker":"[15]"},{"why":"Introduces the Conformer architecture that constitutes the ASR model under analysis.","marker":"[21]"},{"why":"Provides the TIMIT dataset with time-aligned phoneme transcriptions used to segment phoneme spans.","marker":"[36]"},{"why":"Supplies the acoustic-phonetic account of vowels in which F1 and F2 determine vowel quality.","marker":"[16]"},{"why":"Characterizes English fricative spectral properties used as the reference for spectral-match scoring.","marker":"[19]"},{"why":"Supports the distinction between sibilant and non-sibilant fricative spectral clarity used to interpret the saliency results.","marker":"[20]"},{"why":"Provides the phonetic account of plosive release and burst cues as place-of-articulation information.","marker":"[18]"},{"why":"Supplies burst-spectrum evidence for stop contrasts, grounding the burst-peak analysis for plosives.","marker":"[38]"},{"why":"Documents gender performance gaps in ASR, motivating the paper's gender-split analysis of saliency.","marker":"[37]"}],"fun_headline_variants":["ASR's inner ear: vowel formants, sibilant spectra, plosive bursts","AI speech model hears like humans, saliency maps show","What ASR really listens to: formants, sibilants, burst releases","Feature attribution unpicks ASR's acoustic cues","Vowel formants and burst releases dominate ASR saliency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis keeps only the 3 percent of spectrogram elements deemed most important, and that cutoff was chosen by eye; the paper's comparisons all depend on that choice.","fun_headline_variants_meta":{"raw":{"variants":["ASR's inner ear: vowel formants, sibilant spectra, plosive bursts","AI speech model hears like humans, saliency maps show","What ASR really listens to: formants, sibilants, burst releases","Feature attribution unpicks ASR's acoustic cues","Vowel formants and burst releases dominate ASR saliency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000833,"raw_usage":{"total_tokens":3636,"prompt_tokens":943,"completion_tokens":2693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2600}},"tokens_in":559,"tokens_out":2693,"duration_ms":18957,"temperature":1.0,"reasoning_tokens":2600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:28:06.475889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same TIMIT SX subset, recompute time coverage and spectral match at saliency cutoffs of 1% and 10%; if the vowel-over-consonant and sibilant-over-non-sibilant gaps invert or vanish, the conclusions are an artifact of the 3% threshold rather than of model behavior.","supporting_citations":[{"cited_title":"Learn- ing weakly supervised multimodal phoneme embeddings,","cited_arxiv_id":null,"evidence_quote":"Defines SPES, the feature attribution method that produces the saliency maps on which the entire analysis relies."},{"cited_title":"Gradient-Adjusted Neuron Activation Profiles for Comprehensive Introspection of Convolutional Speech Recognition Models","cited_arxiv_id":"2002.08125","evidence_quote":"Introduces the Conformer architecture that constitutes the ASR model under analysis."},{"cited_title":"Com- mon V oice: A Massively-Multilingual Speech Corpus,","cited_arxiv_id":null,"evidence_quote":"Provides the TIMIT dataset with time-aligned phoneme transcriptions used to segment phoneme spans."},{"cited_title":"Acous- tic characteristics of American English vowels,","cited_arxiv_id":null,"evidence_quote":"Supplies the acoustic-phonetic account of vowels in which F1 and F2 determine vowel quality."},{"cited_title":"Introspection for convolutional automatic speech recognition,","cited_arxiv_id":null,"evidence_quote":"Characterizes English fricative spectral properties used as the reference for spectral-match scoring."},{"cited_title":"End-to-end acoustic modeling using convolutional neural networks for HMM- based automatic speech recognition,","cited_arxiv_id":null,"evidence_quote":"Supports the distinction between sibilant and non-sibilant fricative spectral clarity used to interpret the saliency results."},{"cited_title":"Interpretable Convolutional Filters with SincNet,","cited_arxiv_id":null,"evidence_quote":"Provides the phonetic account of plosive release and burst cues as place-of-articulation information."},{"cited_title":"TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation,","cited_arxiv_id":null,"evidence_quote":"Supplies burst-spectrum evidence for stop contrasts, grounding the burst-peak analysis for plosives."}],"review_version":1}