Pith. sign in

REVIEW 4 major objections 6 minor 48 references

Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Phoneme similarity soft labels cut phonetic error rates by a third

desk verdict The PER gain from soft-training is credible and independent, but the headline WPER/AER gains in Table 3 are partly by construction because both metrics share the same similarity/articulatory model used to build the training targets. read the letter →

arxiv 2507.14346 v1 pith:4L7BDUVE submitted 2025-07-18 eess.AS cs.SD

Xuanru Zhou , Jiachen Lian , Cheol Jun Cho , Tejas Prabhune , Shuhe Li , William Li , Rodrigo Ortiz , Zoe Ezzes
show 7 more authors
Jet Vonk Brittany Morin Rian Bogley Lisa Wauters Zachary Miller Maria Gorno-Tempini Gopala Anumanchipalli
This is my paper
classification eess.AScs.SD
keywords phoneticerrordetectionverbatimphonemerecognitionsimilaritysoftlabelsmulti-tasklearningpronunciationassessmentVCTK-accentarticulatoryrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a speech recognizer trained to transcribe phonemes verbatim—what a speaker actually says, not what a script says they should say—gets markedly better when the training loss itself knows which phonemes are confusable. The authors build a phoneme similarity matrix by three routes (heuristic features, articulatory kinematics, and syllabic embeddings), then feed those similarity scores into both a CTC loss and an autoregressive mapping loss as soft labels. On their simulated VCTK-accent corpus, adding the soft labels lowers phoneme error rate from 18.31% to 12.37% and pushes weighted and articulatory error rates down by roughly eight percentage points. They also release the simulated dataset and two new metrics (WPER and AER) as a benchmark for phonetic error detection. If the effect transfers to real speech, pronunciation-assessment systems and clinical dysfluency tools would finally be able to hear the error a learner actually made.

What carries the argument

The load-bearing object is the phoneme similarity matrix $S \in \mathbb{R}^{N\times N}$, whose entries in $(0,1)$ score how confusable any two phonemes are. It enters the loss in two places: the soft-CTC loss weights emission probabilities by the target phoneme's similarity column, so predicting a similar phoneme is penalized less than a distant one, and the soft-mapping loss replaces the one-hot target with the same soft label in an autoregressive frame-level objective. Three constructions of $S$ are compared: a heuristic matrix from eight articulatory-linguistic features, a data-driven matrix from acoustic-to-articulatory inversion on VCTK, and a matrix from Sylber syllabic embeddings fine-tuned for phoneme classification. The matrix also defines the proposed WPER metric, where a substitution contributes $1-S(p_r,p_s)$ instead of a full error.

What would settle it

A reader could take the released VCTK-accent test set, re-label a random sample of the real-speech datasets with independent transcribers, and check whether the model's reported PER, WPER, and AER on those datasets stay close to the stated values with human-verified labels; if independent labels disagree substantially or a plain CTC baseline with one-hot targets matches the soft-trained model on real speech, the central claim would be weakened.

Watch

Extended reading notes

Core claim

The central claim is that phonetic error detection should be framed as verbatim phoneme recognition—transcribing what was pronounced—and that this task is materially improved by modeling phoneme similarity inside the training objective rather than treating phonemes as independent classes. The paper shows that replacing one-hot targets with a similarity-weighted soft label, in both the CTC alignment loss and the mapping loss, reduces PER from 18.31% to 12.37% on VCTK-accent, with weighted phoneme error rate falling from 16.98% to 7.41% and articulatory error rate from 17.42% to 10.53%. The articulatory-based similarity matrix performs best among the three proposed constructions. The authors state this establishes a new benchmark for phonetic error detection, and they position the work against earlier approaches that transcribed what speakers should have said rather than what they actually said.

Load-bearing premise

The whole evaluation depends on the idea that speech synthesized by VITS with injected phoneme substitutions behaves enough like real mispronunciation, and on the authors' own hand-made phoneme labels for the real datasets, so if simulated errors are unrealistic or the labels are wrong, the reported gains may not hold in practice.

Editorial extensions

If this is right

  • Verbatim phoneme recognition becomes the explicit target of pronunciation assessment, so errors are scored against what was uttered, not the canonical prompt.
  • Adding soft similarity labels is a drop-in change to CTC and mapping losses, so other ASR architectures could adopt it without new data.
  • The WPER and AER metrics give graded, articulator-aware error scores instead of binary substitution counts.
  • The open-source VCTK-accent corpus (323.9 hours of synthesized speech with injected substitutions) gives the field a shared training and evaluation set.
  • Articulatory-based similarity outperforms heuristic and syllabic similarity on the simulated tests, pointing to articulatory kinematics as the most informative similarity source.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the evaluation sets on real speech were segmented and labeled by the authors themselves with no reported inter-annotator agreement, the real-speech error rates should be read as indicative until independent labels exist.
  • The biggest gains come on synthetic TTS speech; if the loss is doing its job by exploiting confusability structure, the same soft-label trick may help other low-resource tasks where phoneme-level labels are noisy, such as child speech or disordered speech.
  • One testable extension: replace the fixed similarity matrix with a per-speaker or per-dialect matrix, since confusability is likely speaker- and accent-dependent.
  • The WPER metric could be inverted into a pronunciation score for CAPT apps, since a substitution between similar phonemes would lower the score far less than a substitution between distant ones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a verbatim phoneme recognition framework for phonetic error detection. The model uses a conformer encoder on WavLM features with multi-task training combining a soft-CTC loss and a soft-mapping loss, both of which incorporate phoneme similarity matrices. Three similarity modeling methods are introduced: heuristic-based, articulatory-based, and Sylber-based. A simulated dataset, VCTK-accent, is generated by injecting phoneme substitutions into VCTK text and synthesizing speech with VITS. Two new evaluation metrics are proposed: Weighted Phoneme Error Rate (WPER) and Articulatory Error Rate (AER). Experiments on VCTK-accent and four real speech datasets report that multi-task soft-training reduces PER, WPER, and AER, and the authors claim this establishes a new benchmark for phonetic error detection.

Significance. If the central claim holds, incorporating phoneme similarity as soft labels in CTC and mapping losses is a plausible and potentially useful technique for phoneme-level error detection and mispronunciation diagnosis, and the open-sourced VCTK-accent dataset could serve as a useful resource. The paper's clean ablation on the synthetic test set provides some evidence that soft-training helps verbatim phoneme recognition, independent of the proposed metrics. However, the significance is currently limited by two factors: the proposed WPER and AER metrics share the same similarity/articulatory models used in training, making part of the reported improvement circular, and the evaluation contains no comparisons to existing phoneme recognizers or mispronunciation detectors, so the claimed benchmark status is not established.

major comments (4)
  1. [2.3.2 and 3.3.2] The WPER metric (Eq. 2) weights substitution errors using the same similarity matrix S that provides the soft labels in the training loss (Eq. 1). Since soft-CTC and soft-mapping explicitly encourage the model to emit probabilities concentrated on phonemes similar to the target, the model is rewarded for reproducing the training objective. Therefore, the WPER reduction from 16.98% to 7.41% in Table 3 is not an independent measure of transcription quality. The authors should either evaluate WPER with a similarity matrix that was not used in training (e.g., an external IPA-based distance or human perceptual ratings) or restrict the central ablation claim to PER.
  2. [3.3.3 and 2.2.2] The AER metric uses the same acoustics-to-articulatory inversion (AAI) model and the same articulatory reference construction that generates the articulatory similarity matrix, which is also the variant reported as best in Table 3. The 17.42% to 10.53% AER improvement in Table 3 thus partly reflects how closely the model matches its own training targets, not necessarily an improvement in phonetic error detection. An external articulatory or perceptual evaluation is needed to support this metric.
  3. [3.2 and Table 4] The evaluation on L2-ARCTIC, Speechocean762, MultiPA, and PPA Speech relies on the authors' own segmentation and phoneme labels, with no inter-annotator agreement reported. For L2-ARCTIC and Speechocean762, existing forced-aligned transcriptions and mispronunciation annotations are available; using those instead of self-made labels would provide a more objective and reproducible evaluation. Without such validation, the real-world error rates in Table 4 and the generalization conclusions are not adequately supported.
  4. [3.5] The claim that there are no established benchmarks suitable for direct comparison is too strong. Existing mispronunciation detection systems, standard phoneme recognizers, and prior work on phoneme similarity in CTC training (e.g., [23]) provide reasonable points of comparison. The absence of any baseline comparison means that the central claim of significantly reduced error rates and a new benchmark is not yet demonstrated. At minimum, the authors should compare their model against a standard WavLM+conformer CTC baseline and a public mispronunciation detection model on the same test sets.
minor comments (6)
  1. [2.2.1] The heuristic similarity weights (0.2, 0.1, 0.15, 0.15, 0.1, 0.2, 0.2, 0.1) are stated without justification or sensitivity analysis; since these weights directly affect training and evaluation, a brief discussion of their robustness would be helpful.
  2. [2.3.1] There is a duplicated phrase: For the auto-regressive mechanism, For the autoregressive mechanism, at each timestep... The sentence should be cleaned up.
  3. [Equation (1)] The text after the equation reads and and N is the phoneme dictionary size. The duplicate and should be removed.
  4. [2.3] The section title V erbatim Phoneme Recognition contains an extra space; please fix the typo.
  5. [3.1] For VCTK-accent, please describe the train/test split explicitly: the paper says 90/10 train/test split in Section 3.2 but does not state whether the split is by utterance, speaker, or word, nor whether speakers are held out.
  6. [4] The limitation statement that CMU phonemes may not align well with actual speech is important and could be substantiated with examples, since the entire method is built on the CMU dictionary.

Circularity Check

2 steps flagged · score 6.0 of 10

The WPER and AER improvements reported in Table 3 are partly by construction: both metrics reuse the same phoneme-similarity matrix or articulatory mapping that supplies the soft training labels, so only the PER gain is fully independent evidence.

  1. self definitional [Section 2.3.2, Eq. (1); Section 3.3.2, Eq. (2); Table 3]
    "We can treat this vector as a soft label of this phoneme. For soft-CTC loss, we utilize the target phoneme’s soft label to weight the emission probability, thereby reducing the penalty for prediction errors between similar phonemes. ... In the case of substitutions, we replace the count of substitutions with the sum of the phoneme similarities between the substituted pairs. WPER = (D + Σ_{(pr,ps)}(1 − S(pr, ps)) + I)/L (2)"

    Equation (1) trains the model to put probability mass on phonemes similar to the target by using the identical matrix S as rows of soft labels in soft-CTC and soft-mapping losses. Equation (2) then defines WPER by penalizing a substitution (pr, ps) with weight 1 − S(pr, ps), which is exactly the inverse of the soft-label similarity that the training objective rewards. A model that follows its soft-training objective and confuses similar phonemes is therefore rewarded in WPER by the same quantity it was trained to maximize. The Table 3 WPER drop (16.98% to 7.41%) thus partially measures how closely the model reproduced the S-induced confusion pattern rather than how accurately it transcribed the acoustic signal.

  2. self definitional [Section 2.2.2; Section 3.3.3; Table 3]
    "We first construct a reference articulatory position for each phoneme using the VCTK corpus and an acoustic-to-articulatory inversion (AAI) model [25]. ... We then calculate the L2 distance between the articulatory features of the current frame and the target phoneme, using the mapping we constructed in Sec. 2.2.2. If the distance exceeds a threshold, denoted as τ, the frame is classified as negative."

    The articulatory-based similarity matrix, which is the variant used for the best '+smap + sCTC' row in Table 3, is built from reference articulatory positions per phoneme obtained with the AAI model in Section 2.2.2. AER is then computed with the same AAI model and the same reference mapping: a frame is counted as an articulatory error when its distance to the target phoneme's reference position exceeds τ. Consequently, the AER metric and the best-performing soft-training similarity matrix share the same definition of phoneme closeness. The reported AER improvement (17.42% to 10.53%) is therefore at least partly a restatement of the articulatory prior already encoded in the training soft labels, not an independent verification of phonetic error detection.

full rationale

The paper's core claim that multi-task soft-training improves verbatim phoneme recognition has genuine independent support in the PER metric, which does not use the phoneme similarity matrix: PER drops from 18.31% to 12.37% in Table 3. However, the two newly proposed metrics used to support the phonetic-error-detection claim, WPER and AER, are partly circular as evidence. WPER (Eq. 2) is defined using the same matrix S whose rows are used as soft labels in the training loss (Eq. 1), so soft-training is rewarded for doing exactly what the metric measures. AER uses the same AAI-based articulatory reference mapping that defines the articulatory similarity matrix selected as the best method and used in the main comparison. Thus the large WPER and AER reductions in Table 3 are not independent of the training intervention; they partly encode the training target into the evaluation. The paper's own limitation section mentions TTS quality and CMU phoneme alignment but does not acknowledge this metric-training coupling. No load-bearing circularity was found in the simulation pipeline or in the self-citations to prior dysfluency work; those are methodological borrowings, not definitional loops. Overall score 6 reflects that the central effectiveness claim rests on a mix of one independent metric (PER) and two metrics whose improvement is partially forced by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several design choices that are either hand-set or assumed: the heuristic similarity weights, the AER threshold, the loss weighting, and the validity of TTS simulation and self-annotation. These are not derived from first principles or validated against external benchmarks.

free parameters (3)
  • Heuristic feature weights = 0.2, 0.1, 0.15, 0.15, 0.1, 0.2, 0.2, 0.1
    Set by hand for eight phonetic features in Section 2.2.1; no optimization procedure or sensitivity analysis is given. The similarity matrix S and hence all soft-training and WPER results depend on these values.
  • AER threshold tau = 0.5 * distance between two most distant phonemes
    Arbitrary threshold in Section 3.3.3 that determines the negative frame ratio for AER; no tuning or robustness check is reported.
  • Loss weights lambda_ctc, lambda_map = 0.8, 0.2
    Chosen for the multi-task objective in Section 2.3.2; no search is described.
assumptions (4)
  • domain assumption VITS text-to-speech with injected CMU phoneme substitutions produces speech that contains realistic phonetic errors, and the modified phoneme sequence is a valid ground truth.
    Used to build VCTK-accent (Section 2.1) and to train/evaluate the model; if TTS errors are not perceptually equivalent to human mispronunciations, the benchmark is unrepresentative.
  • domain assumption The AAI model [25] accurately reconstructs articulatory positions from audio for constructing the articulatory similarity matrix and computing AER.
    Section 2.2.2 and 3.3.3 rely on the AAI model's outputs; no validation against measured articulation is provided.
  • domain assumption Fine-tuned Sylber embeddings provide a phoneme representation aligned with human syllabic perception.
    Section 2.2.3 assumes Sylber features are suitable for phoneme similarity; no direct comparison to human perception is given.
  • domain assumption WavLM features are a sufficient acoustic representation for verbatim phoneme recognition.
    Section 2.3 uses WavLM as the input feature extractor without comparison to other features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling." pith.science (2026). https://pith.science/paper/4L7BDUVE

@misc{pith2026250714346,
  author       = {Pith},
  title        = {Pith review of: Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4L7BDUVE}},
  note         = {Machine review of arXiv:2507.14346}
}
read the original abstract

Phonetic error detection, a core subtask of automatic pronunciation assessment, identifies pronunciation deviations at the phoneme level. Speech variability from accents and dysfluencies challenges accurate phoneme recognition, with current models failing to capture these discrepancies effectively. We propose a verbatim phoneme recognition framework using multi-task training with novel phoneme similarity modeling that transcribes what speakers actually say rather than what they're supposed to say. We develop and open-source \textit{VCTK-accent}, a simulated dataset containing phonetic errors, and propose two novel metrics for assessing pronunciation differences. Our work establishes a new benchmark for phonetic error detection.

Figures

Figures reproduced from arXiv: 2507.14346 by the authors.

Figure 1
Figure 1. Demo of our method. Model transcribes user speech into phoneme sequences, detects errors, scores using metrics, and generates articulator visualizations and feedback [25]: re￾lax the tongue and place it against the roof of the mouth, with the tip lightly touching the teeth. dysarthric speech, their approach focuses on transcribing what speakers should have said rather than what they actually said, which is our resea… view at source ↗
Figure 2
Figure 2. Heuristic-based phoneme similarity matrix each pair of phonemes and apply min-max normalization to ob￾tain the similarity scores. 2.2.3. Sylber-based Human speech segmentation is natually syllabic [27]. We then utilize the Sylber [27] feature, as it offers a clean and robust syllabic structure. First, we fine-tune Sylber using a phoneme classification task with a single linear classifier layer, employing the VCTK co… view at source ↗
Figure 3
Figure 3. Pipeline of phoneme recognition and error detection: phonetic error of "IY" -> "EY" with a similarity score 79% in the word "Please", and the articulatory feedback is moving the tongue towards the front of the mouth. represents the similarity score of this phoneme and all other phonemes. We can treat this vector as a soft label of this phoneme. For soft-CTC loss, we utilize the target phoneme’s soft label to weight … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 43 canonical work pages

  1. [23]

    Stutter-solver: End-to-end multi-lingual dysfluency detection,

    X. Zhou, C. J. Cho, A. Sharma, B. Morin, D. Baquirin, J. V onk, Z. Ezzes, Z. Miller, B. L. Tee, M. L. Gorno-Tempini et al. , “Stutter-solver: End-to-end multi-lingual dysfluency detection,” in 2024 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 2024, pp. 1039–1046

    work page 2024

  2. [1]

    Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling

    Introduction Speech pronunciation assessment plays a crucial role in lan- guage learning [1, 2] and diagnosis of speech disorders [3]. As traditional human assessment is time-consuming and lacks uni- fied standards, recent advancement has shifted to Computer- Aided Pronunciation Training (CAPT) [4]. Parallel to the bloom of textual large language models [...

    work page Pith review arXiv

  3. [2]

    IY" -> EY

    Method 2.1. Data Simulation To train the phoneme recognition model, accurate labels are es- sential, mapping directly to the word being pronounced. Thus, we follow the TTS-based dysfluency simulation pipeline de- scribed in [16]. First, we inject phonetic substitutions into each word from the text of the VCTK corpus [28] based on a prede- fined set of com...

    work page

  4. [3]

    Experiment 3.1. Datasets • VCTK-Accent is a TTS-based [16] simulated datasets, ex- tended from VCTK corpus [28], which contains vowel and consonant phonetic errors, with simulation details provided in Sec. 2.1. The total duration of the dataset is 323.9 hours. • L2-ARCTIC [34] includes recordings from 24 non-native English speakers, each recording about o...

    work page

  5. [4]

    The model is trained for 30 epochs with total of 75 hours on an RTX A6000. 3.3. Evaluation Metrics 3.3.1. Phoneme Error Rate (PER) PER measure of how many errors (inserted, deleted, and sub- stitute phonemes) are predicting phoneme sequences compared to the actual phoneme sequence. It calculated by dividing the number of phoneme errors by the total number...

    work page

  6. [5]

    Results show that modeling similarity significantly improves transcrip- tion accuracy and sets a new benchmark for phonetic error de- tection

    Limitation and Conclusion This paper presents a framework for verbatim phoneme recog- nition and error detection using multi-task learning and soft training, incorporating phoneme similarity modeling. Results show that modeling similarity significantly improves transcrip- tion accuracy and sets a new benchmark for phonetic error de- tection. Nonetheless, ...

    work page

  7. [6]

    Acknowledgements Thanks for support from UC Noyce Initiative, Society of Hell- man Fellows, NIH/NIDCD, and the Schwab Innovation fund

    work page

  8. [7]

    Automatic pronun- ciation assessment - a review,

    Y . El Kheir, A. Ali, and S. A. Chowdhury, “Automatic pronun- ciation assessment - a review,” in Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8304–8324

    work page 2023

Show all 48 references
  1. [8]

    Automatic pronunciation error detection: an acoustic-phonetic approach,

    K. Truong, A. Neri, C. Cucchiarini, and H. Strik, “Automatic pronunciation error detection: an acoustic-phonetic approach,” in InSTIL/ICALL 2004 Symposium on Computer Assisted Learning , 2004, p. paper 032

    2004

  2. [9]

    Ssdm: Scalable speech dysfluency modeling,

    J. Lian, X. Zhou, Z. Ezzes, J. V onk, B. Morin, D. P. Baquirin, Z. Miller, M. L. Gorno Tempini, and G. Anumanchipalli, “Ssdm: Scalable speech dysfluency modeling,” in Advances in Neural In- formation Processing Systems, vol. 37, 2024

    2024

  3. [10]

    Language-independent methods for computer- assisted pronunciation training,

    A. Lee et al. , “Language-independent methods for computer- assisted pronunciation training,” Ph.D. dissertation, Mas- sachusetts Institute of Technology, 2016

    2016

  4. [11]

    Chatgpt,

    OpenAI, “Chatgpt,” 2022. [Online]. Available: https://openai. com/chatgpt/

    2022

  5. [12]

    Wavchat: A survey of spoken dialogue models,

    S. Ji, Y . Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng et al., “Wavchat: A survey of spoken dialogue models,” arXiv preprint arXiv:2411.13577, 2024

    2024 arXiv

  6. [13]

    Dynamic- superb phase-2: A collaboratively expanding benchmark for mea- suring the capabilities of spoken language models with 180 tasks,

    C.-y. Huang, W.-C. Chen, S.-w. Yang, A. T. Liu, C.-A. Li, Y .-X. Lin, W.-C. Tseng, A. Diwan, Y .-J. Shih, J. Shi et al., “Dynamic- superb phase-2: A collaboratively expanding benchmark for mea- suring the capabilities of spoken language models with 180 tasks,” ICLR, 2025

    2025

  7. [14]

    The im- pact of allophony versus contrast on speech perception,

    A. Boomershine, K. C. Hall, E. Hume, and K. Johnson, “The im- pact of allophony versus contrast on speech perception,”Contrast in phonology, pp. 143–172, 2008

    2008

  8. [15]

    Duolingo: Language learning app,

    Duolingo, “Duolingo: Language learning app,” 2025. [Online]. Available: https://www.duolingo.com/

    2025

  9. [16]

    Speak: Ai-powered language learning,

    Speak, “Speak: Ai-powered language learning,” 2025. [Online]. Available: https://www.speak.com/

    2025

  10. [17]

    Gpt-4o real time,

    OpenAI, “Gpt-4o real time,” 2024. [Online]. Available: https: //openai.com/index/hello-gpt-4o/

    2024

  11. [18]

    Improving mispronunciation detection with wav2vec2- based momentum pseudo-labeling for accentedness and intelligi- bility assessment,

    M. Yang, K. Hirschi, S. D. Looney, O. Kang, and J. H. Hansen, “Improving mispronunciation detection with wav2vec2- based momentum pseudo-labeling for accentedness and intelligi- bility assessment,” in Interspeech 2022, 2022, pp. 4481–4485

    2022

  12. [19]

    Unconstrained dysfluency modeling for dysfluent speech transcription and de- tection,

    J. Lian, C. Feng, N. Farooqi, S. Li, A. Kashyap, C. J. Cho, P. Wu, R. Netzorg, T. Li, and G. K. Anumanchipalli, “Unconstrained dysfluency modeling for dysfluent speech transcription and de- tection,” in 2023 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU)...

    2023

  13. [20]

    Towards hierarchical spoken lan- guage disfluency modeling,

    J. Lian and G. Anumanchipalli, “Towards hierarchical spoken lan- guage disfluency modeling,” in Proceedings of the 18th Confer- ence of the European Chapter of the Association for Computa- tional Linguistics, Mar. 2024, pp. 539–551

    2024

  14. [21]

    Ssdm 2.0: Time-accurate speech rich transcription with non-fluencies,

    J. Lian, X. Zhou, Z. Ezzes, J. V onk, B. Morin, D. Baquirin, Z. Mille, M. L. G. Tempini, and G. K. Anumanchipalli, “Ssdm 2.0: Time-accurate speech rich transcription with non-fluencies,” arXiv preprint arXiv:2412.00265, 2024

    2024 arXiv

  15. [22]

    Yolo-stutter: End-to-end region-wise speech dys- fluency detection,

    X. Zhou, A. Kashyap, S. Li, A. Sharma, B. Morin, D. Baquirin, J. V onk, Z. Ezzes, Z. Miller, M. Tempini, J. Lian, and G. Anu- manchipalli, “Yolo-stutter: End-to-end region-wise speech dys- fluency detection,” in Interspeech 2024, 2024, pp. 937–941

    2024

  16. [24]

    Time and tokens: Benchmarking end- to-end speech dysfluency detection,

    X. Zhou, J. Lian, C. J. Cho, J. Liu, Z. Ye, J. Zhang, B. Morin, D. Baquirin, J. V onk, Z. Ezzes, Z. Miller, M. L. G. Tempini, and G. Anumanchipalli, “Time and tokens: Benchmarking end- to-end speech dysfluency detection,” 2024. [Online]. Available: https://arxiv.org/abs/2409.13582

    2024 arXiv

  17. [25]

    Analysis and evaluation of synthetic data generation in speech dysfluency detection,

    J. Zhang, X. Zhou, J. Lian, S. Li, W. Li, Z. Ezzes, R. Bogley, L. Wauters, Z. Miller, J. V onk, B. Morin, M. Gorno-Tempini, and G. Anumanchipalli, “Analysis and evaluation of synthetic data generation in speech dysfluency detection,” Interspeech, 2025

    2025

  18. [26]

    Dysfluent wfst: A framework for zero-shot speech dysfluency transcription and de- tection,

    C. Guo, J. Lian, X. Zhou, J. Zhang, S. Li, Z. Ye, H. J. Park, A. Das, Z. Ezzes, J. V onk, B. Morin, R. Bogley, L. Wauters, Z. Miller, M. Gorno-Tempini, and G. Anumanchipalli, “Dysfluent wfst: A framework for zero-shot speech dysfluency transcription and de- tection,” Interspeech, 2025

    2025

  19. [27]

    Seamless dysfluent speech text alignment for disordered speech analysis,

    Z. Ye, J. Lian, X. Zhou, J. Zhang, H. Li, S. Li, C. Guo, A. Das, P. Park, Z. Ezzes, J. V onk, B. Morin, R. Bogley, L. Wauters, Z. Miller, M. Gorno-Tempini, and G. Anumanchipalli, “Seamless dysfluent speech text alignment for disordered speech analysis,” Interspeech, 2025

    2025

  20. [28]

    Automatic detection of articulatory-based disfluen- cies in primary progressive aphasia,

    J. Lian, X. Zhou, C. Guo, Z. Ye, Z. Ezzes, J. M. V onk, B. Morin, D. Baquirin, Z. Miller, M. L. Gorno-Tempini, and G. K. Anu- manchipalli, “Automatic detection of articulatory-based disfluen- cies in primary progressive aphasia,” IEEE Journal of Selected Topics in Signal Proce...

    2025

  21. [29]

    Dypcl: Dy- namic phoneme-level contrastive learning for dysarthric speech recognition,

    W. Lee, S. Im, H. Do, Y . Kim, J. Ok, and G. G. Lee, “Dypcl: Dy- namic phoneme-level contrastive learning for dysarthric speech recognition,” NAACL, 2025

    2025

  22. [30]

    Leveraging allophony in self-supervised speech models for atyp- ical pronunciation assessment,

    K. Choi, E. Yeo, K. Chang, S. Watanabe, and D. Mortensen, “Leveraging allophony in self-supervised speech models for atyp- ical pronunciation assessment,” in NAACL, 2025

    2025

  23. [31]

    Coding speech through vocal tract kinematics,

    C. J. Cho, P. Wu, T. S. Prabhune, D. Agarwal, and G. K. Anu- manchipalli, “Coding speech through vocal tract kinematics,” in IEEE JSTSP, 2025

    2025

  24. [32]

    Multi-task learning with deep neural networks: A survey,

    M. Crawshaw, “Multi-task learning with deep neural networks: A survey,” 2020. [Online]. Available: https://arxiv.org/abs/2009. 09796

    2020

  25. [33]

    Sylber: Syllabic embedding repre- sentation of speech from raw audio,

    C. J. Cho, N. Lee, A. Gupta, D. Agarwal, E. Chen, A. W. Black, and G. K. Anumanchipalli, “Sylber: Syllabic embedding repre- sentation of speech from raw audio,” in ICLR, 2025

    2025

  26. [34]

    Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (ver- sion 0.92),

    J. Yamagishi, C. Veaux, K. MacDonald et al., “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (ver- sion 0.92),” University of Edinburgh. The Centre for Speech Tech- nology Research (CSTR), 2019

    2019

  27. [35]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,

    J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in Inter- national Conference on Machine Learning, 2021

    2021

  28. [36]

    PanPhon: A resource for mapping IPA segments to articulatory feature vectors,

    D. R. Mortensen, P. Littell, A. Bharadwaj, K. Goyal, C. Dyer, and L. Levin, “PanPhon: A resource for mapping IPA segments to articulatory feature vectors,” in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. Osaka, J...

    2016

  29. [37]

    Anderson, Essentials of linguistics

    C. Anderson, Essentials of linguistics . McMaster University, 2018

    2018

  30. [38]

    Wavlm: Large- scale self-supervised pre-training for full stack speech process- ing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “Wavlm: Large- scale self-supervised pre-training for full stack speech process- ing,” IEEE JSTSP, 2022

    2022

  31. [39]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in Interspeech 2020, 2020, pp. 5036–5040

    2020

  32. [40]

    L2-arctic: A non- native english speech corpus,

    G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev- Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-arctic: A non- native english speech corpus,” in Interspeech, 2018

    2018

  33. [41]

    speechocean762: An open-source non- native english speech corpus for pronunciation assessment,

    J. Zhang, Z. Zhang, Y . Wang, Z. Yan, Q. Song, Y . Huang, K. Li, D. Povey, and Y . Wang, “speechocean762: An open-source non- native english speech corpus for pronunciation assessment,” inIn- terspeech 2021, 2021, pp. 3710–3714

    2021

  34. [42]

    Multipa: A multi-task speech pronunciation assessment model for open response sce- narios,

    Y .-W. Chen, Z. Yu, and J. Hirschberg, “Multipa: A multi-task speech pronunciation assessment model for open response sce- narios,” in Interspeech 2024, 2024, pp. 297–301

    2024

  35. [43]

    Classification of primary progressive aphasia and its variants,

    M. L. Gorno-Tempini, A. E. Hillis, S. Weintraub, A. Kertesz, M. Mendez, S. F. Cappa, J. M. Ogar, J. D. Rohrer, S. Black, B. F. Boeve et al., “Classification of primary progressive aphasia and its variants,” Neurology, vol. 76, no. 11, pp. 1006–1014, 2011

    2011

  36. [44]

    International phonetic alphabet

    Wikipedia contributors, “International phonetic alphabet.” [On- line]. Available: https://en.wikipedia.org/wiki/International_ Phonetic_Alphabet

  37. [45]

    Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition,

    J. Lian, A. W. Black, L. Goldstein, and G. K. Anumanchipalli, “Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition,” inProc. Interspeech 2022, 2022, pp. 4686–4690

    2022

  38. [46]

    Articulatory representation learning via joint factor analysis and neural matrix factorization,

    J. Lian, A. W. Black, Y . Lu, L. Goldstein, S. Watanabe, and G. K. Anumanchipalli, “Articulatory representation learning via joint factor analysis and neural matrix factorization,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (IC...

    2023

  39. [47]

    Deep Speech Synthesis from MRI-Based Articulatory Representations,

    P. Wu, T. Li, Y . Lu, Y . Zhang, J. Lian, A. W. Black, L. Goldstein, S. Watanabe, and G. K. Anumanchipalli, “Deep Speech Synthesis from MRI-Based Articulatory Representations,” in Proc. INTER- SPEECH 2023, 2023, pp. 5132–5136

    2023

  40. [48]

    Av-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations,

    J. Lian, A. Baevski, W.-N. Hsu, and M. Auli, “Av-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations,” in 2023 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.