Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss

Berlin Chen; Fu-An Chao

REVIEW 3 major objections 4 minor 1 cited by

Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HMamba, a hierarchical bidirectional Mamba model with a decoupled cross-entropy loss, jointly performs pronunciation assessment and mispronunciation diagnosis, reporting an MDD F1 of 63.85% on speechocean762 and broad APA gains over prior…

desk verdict Solid APA results and a new loss, but the headline MDD F1 is tuned on the test set — treat the 63.85% as an upper bound, not a robust estimate. read the letter →

arxiv 2502.07575 v2 pith:NVPXVMBB submitted 2025-02-11 eess.AS cs.CL

Fu-An Chao , Berlin Chen This is my paper · ORCID

classification eess.AScs.CL

keywords Mambaselectivestatespacemodelautomaticpronunciationassessmentmispronunciationdetectionanddiagnosiscomputer-assistedtrainingdecoupledcross-entropylosshierarchicalmodelingspeechocean762

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a single hierarchical model built on a bidirectional selective state space model (Mamba) can do both jobs a pronunciation tutor needs—scoring how well a learner speaks and pinpointing exactly which phones are mispronounced—at the same time, without the computational cost of a Transformer. On the speechocean762 corpus, the proposed HMamba reports the best or tied-best Pearson correlations on almost every assessment aspect across phone, word, and utterance levels, and lifts mispronunciation detection F1 to 63.85%, a 22.35-point gain over the only prior joint model. The paper attributes much of that gain to a new loss, decoupled cross-entropy (deXent), which reweights the loss on mispronounced phones separately from correct ones so the model does not simply echo the text prompt. If the results hold, a full CAPT system need not choose between assessment and diagnosis.

What carries the argument

The load-bearing object is the bidirectional selective state space (Mamba) block used hierarchically, together with the decoupled cross-entropy loss. The Mamba block replaces multi-head attention with an input-dependent state space model run forward and backward over the flipped sequence; stacking these blocks at phone, word, and utterance levels lets representations move upward so coarse scores sit on top of fine ones. deXent splits the MDD cross-entropy into a term for correct phones and a term for mispronounced phones, reweighting the latter by $(\mu_h/\mu_m)^\alpha$ to counter the class imbalance and give the user a precision-recall knob.

What would settle it

Rerun the MDD evaluation with $\alpha$ fixed at a value chosen only on training or validation data, or averaged over several values, and report F1 on held-out speechocean762 speakers; if the fixed-$\alpha$ F1 no longer beats Joint-CAPT-L1's 41.50% by the claimed margin, the test-set-swept result was optimistic.

Watch

Extended reading notes

Core claim

HMamba is a hierarchical, bidirectional Mamba encoder: phone-level Mamba blocks feed a phone classifier and a phone accuracy regressor, then word-level blocks plus a word-level convolution feed three word regressors, then utterance-level blocks and attention pooling feed five utterance regressors. The same phone classifier performs free phone recognition, so error states are obtained by comparing recognized phones with the canonical prompt phones. For MDD, the paper replaces plain cross-entropy with deXent, which computes a cross-entropy term over correctly pronounced positions and a reweighted term $(\mu_h/\mu_m)^\alpha$ over mispronounced positions, letting $\alpha$ trade precision against recall. With $\alpha=0.7$ the model reaches 63.85% F1 and 2.72% phone error rate on the speechocean762 test set, compared with 41.50% F1 and 9.93% for the Joint-CAPT-L1 baseline, while APA improves on most reported aspects over single-task and multi-task baselines.

Load-bearing premise

The headline MDD improvement comes from picking $\alpha$ to maximize F1 on the test set, with no separate validation set, so the gain over the baseline may shrink when $\alpha$ is fixed in advance on unseen speakers.

Editorial extensions

If this is right

Joint APA and MDD is feasible in one efficient model: the paper's comparison shows Mamba blocks use fewer parameters and MACs than Transformer blocks while converging faster.
deXent provides a single scalar $\alpha$ that calibrates how aggressively the MDD head hunts for errors, which is directly useful for tuning CAPT feedback to be more or less conservative.
The hierarchical design matters: predicting phone- and word-level scores in lower layers beats both a flat model and a parallel CLS-token model on the reported metrics.
The ablations show canonical phone embeddings are the single most important feature for MDD; without them F1 collapses from 63.85% to 28.06%, so text-prompt information is doing essential work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported 63.85% F1 is selected by sweeping $\alpha$ on the test set (Table 4) with no separated validation set, so the headline number is likely optimistic; a fixed $\alpha$ chosen on held-out data would give a fairer estimate.
deXent is a general imbalance-aware loss for sequence labeling and could transfer to other tasks with rare positive classes, not just pronunciation error detection.
Because the evaluation only covers Mandarin-accented read speech, the joint model's advantage over single-task systems is unproven for other accents or spontaneous speech; testing on a multi-accent corpus would be a natural next step.
The reported completeness score trails two baselines, which suggests the model's phone-focused training may make it less sensitive to whole words being skipped; a word-level deletion-aware signal could close that gap.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a genuine empirical contribution on the APA side, and the deXent loss is a sensible idea worth testing. But the headline MDD number is softer than the abstract suggests. The stress-test note is right: Table 4 shows alpha swept over {0.3, 0.5, 0.7, 0.9} on the test set, and the Limitations section explicitly says the reported MDD F1 is the maximizer over that sweep. No validation split is described. That makes 63.85% an optimistically biased upper bound, probably by 1–3 F1 points given the spread in Table 4. The comparison to Joint-CAPT-L1 (F1=41.50%) is also the only MDD baseline, and it is a multi-task model from 2023, not a dedicated MDD system. Those two issues together mean the 'considerable improvement' claim is not yet established.

What is genuinely new: HMamba is, as far as I know, the first Mamba-based CAPT model, and the hierarchical bidirectional Mamba structure with parallel APA and MDD is a reasonable combination. The APA results are strong and consistent across phone, word, and utterance levels, with meaningful gains over 3MH. The ablations are thorough: hierarchical structure, feature ablation (canonical phoneme embeddings really matter, as shown in Appendix B), and Mamba-versus-Transformer efficiency (fewer parameters, fewer MACs, better performance). The authors also released code. That is real evidence and should be credited.

Soft spots beyond the MDD evaluation: no error bars or significance tests despite five seeds, and the completeness score is actually worse than 3M/3MH, which they note but attribute to task focus. That is a minor defensive point, not a fatal flaw.

Who benefits: researchers working on CAPT or pronunciation assessment, especially those interested in Mamba-based speech models or multi-task losses for imbalanced phone classification. The paper deserves a serious referee, but the MDD section needs major revision before the central claim can be accepted. I would send it to review with a clear request: add a validation split for alpha selection, compare against single-task MDD baselines (e.g., a dedicated MDD classifier), report variance across seeds, and temper the abstract.

My own verdict: conditional. The APA part stands, but the MDD headline should not be cited as-is.

Referee Report

3 major / 4 minor

Summary. The paper proposes HMamba, a hierarchical bidirectional selective state space model that jointly performs automatic pronunciation assessment (APA) across phone, word, and utterance levels and mispronunciation detection and diagnosis (MDD) at the phone level, evaluated on the speechocean762 corpus. A second contribution is a decoupled cross-entropy loss (deXent) that reweights the mispronunciation term by the ratio of training-set class frequencies, controlled by a hyperparameter alpha. The authors report consistent APA improvements over prior systems and a headline MDD F1 of 63.85%, a large gain over the Joint-CAPT-L1 baseline (F1=41.50%). The paper includes ablations of hierarchical structure, acoustic and phonological features, and a Mamba-versus-Transformer comparison, and it releases code.

Significance. If the MDD result is unbiased, the paper makes a useful twofold contribution: it is, to my knowledge, the first application of Mamba/SSM to CAPT, and it shows that a simple class-frequency-rebalanced cross-entropy loss can substantially improve recall in end-to-end MDD without a separate detection module. The APA results are broadly convincing and supported by multiple ablations, including a feature ablation and a structural comparison. The Mamba-versus-Transformer comparison also shows both quality and efficiency advantages, with code released. However, the central MDD claim currently rests on a test-set-selected hyperparameter and a single weak baseline, so the significance of that particular claim is conditional on additional validation.

major comments (3)

[Section 4.2, Table 4, and Limitations] The headline MDD F1 of 63.85% in Table 3 is obtained by sweeping alpha over {0.3, 0.5, 0.7, 0.9} and selecting the value that maximizes F1 on the official speechocean762 test set. The Limitations section explicitly states that 'the MDD performance of HMamba is reported based on maximizing the F1 score using the deXent,' and Appendix A reports alpha as tuned to 0.7. No validation split is described anywhere in the paper. Because the reported number is the maximum of a small hyperparameter grid evaluated on the test set, it is an optimistically biased estimate; the same table shows F1 values between 61.04% and 63.85% across alpha, suggesting optimism on the order of 1-3 F1 points. The central claim of 'considerable improvement' over the baseline should be re-evaluated with alpha selected on a held-out validation set (or via cross-validation), and the final test-set result should be reported with that fixed alpha.
[Table 3 and Section 3.3] The MDD comparison uses only Joint-CAPT-L1, a 2023 multi-task model, as the baseline. The authors acknowledge this is the only prior joint APA+MDD model on speechocean762, but a single baseline from a different modeling family is insufficient to support the claim that HMamba achieves strong MDD performance in absolute terms. I recommend adding dedicated MDD baselines from the literature (e.g., methods reported on speechocean762 for phone-level error detection) or reimplementing one or two strong MDD systems under the same acoustic features and alignment, so that the F1 improvement is contextualized rather than relative to a single system.
[Section 3.2 and Tables 3-4] The paper reports averages over 5 independent trials but gives no standard deviations or significance tests for the MDD F1, precision, recall, or PER. Given that the headline gain over Joint-CAPT-L1 is central to the paper and that the alpha sweep in Table 4 produces nearby F1 values, reporting variance or a confidence interval would help establish that the improvement is not within run-to-run noise. This is particularly important for the MDD metrics, where class imbalance makes small F1 differences meaningful.

minor comments (4)

[Abstract and Section 2.2] The abstract says HMamba 'seamlessly integrates APA and MDD tasks in parallel,' but in the architecture MDD is performed only at the phone level while APA spans three granularities. The word 'parallel' is misleading; the tasks share the phone-level backbone and are trained jointly, not processed in parallel at all granularities. Please clarify this wording.
[Table 1] Some baseline rows contain dashes for missing metrics, which is acceptable, but the table would be easier to read if missing entries were marked with a footnote explaining that the original papers did not report those metrics on speechocean762.
[Section 2.3, Eq. (20)] The notation (mu_h / mu_m)^alpha is defined as a frequency ratio, but the text does not state whether mu_h and mu_m are raw counts or normalized frequencies; since the ratio is invariant to a common scale, this is not a technical error, but it should be stated explicitly.
[Appendix A] The hyperparameter search for alpha and beta is described only as 'tuned'; please specify the search ranges and, for beta, whether it was also selected on the test set or on a validation split.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: HMamba's APA and MDD results are empirical evaluations against the external speechocean762 benchmark, and deXent is a class-frequency reweighting of cross-entropy, not an input-output identity.

full rationale

The paper's claimed results are benchmark evaluations, not first-principles derivations, so the circularity test is whether any reported prediction is entailed by its own inputs. The model is trained with supervised labels from speechocean762, and deXent (Eqs. 18-20) reweights the mispronunciation term by (mu_h/mu_m)^alpha, where mu_h and mu_m are training-set class frequencies; these are data statistics, not predicted labels. No equation defines a predicted score or error state in terms of the same score or error state. The hierarchical Mamba architecture uses lower-level predicted scores as attention-pooling queries (Eq. 14), but the utterance-level outputs are a learned function of pooled hidden states, not a restatement of the lower-level scores. The paper cites the authors' prior 3M/3MH work for hierarchical modeling, but it independently verifies the structure with LMamba/PMamba ablations (Table 2) on the same external benchmark, so the self-citation is not load-bearing. The Limitations section explicitly discloses that MDD performance is reported by maximizing F1 over alpha on the test set (Table 4); this is a test-set hyperparameter selection that can inflate the headline F1 relative to a fixed configuration, but it is a statistical validity concern rather than a circularity, because the F1 values are empirical measurements and are not forced by an equation. No fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work. The central claims therefore have independent content, and the circularity score is low.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the speechocean762 benchmark, a Kaldi aligner, three frozen SSL encoders, and several hand-tuned hyperparameters (notably alpha and beta, tuned without a validation split). No new physical or conceptual entities are introduced; deXent is a loss reweighting scheme, not an invented entity.

free parameters (5)

alpha (deXent reweighting exponent) = 0.7
Swept over {0.3, 0.5, 0.7, 0.9} in Table 4; the value maximizing test-set F1 is reported. No validation split is described, so this is a test-set-tuned hyperparameter.
beta (MDD loss weight) = 0.003
Tuned without a validation set to balance APA and MDD losses (Eq. 21).
tau (attention pooling temperature) = 1.0
Set in Appendix A with no sensitivity analysis.
Layer counts Lp, Lw, Lu = 3, 1, 1
Depth of Mamba blocks per granularity chosen in Appendix A; no ablation for these counts.
Word-level conv kernels and kernel size = 256, 3
Chosen in Appendix A without ablations; affects local dependency modeling.

assumptions (6)

domain assumption speechocean762 annotations (APA score rubrics and MDD phone labels) are accurate ground truth
All training and evaluation use these labels; annotation noise is inherited without analysis.
domain assumption The Kaldi acoustic model's phone alignment and boundaries are correct
Feature extraction (GOP, duration, energy, SSL averaging) depends on these alignments per Appendix A.
domain assumption SSL features (wav2vec2, HuBERT, WavLM) provide complementary phonetic information for L2 speech
Features are concatenated in Eq. (1); ablation shows all contribute, but the mechanism is not analyzed.
domain assumption Free phone recognition with argmax comparison to canonical phones is a valid MDD formulation
Section 2.2 defines error states by comparing recognized phones to the prompt; this assumes the comparison label is a reliable diagnosis.
domain assumption Mamba is a drop-in alternative to Transformer for this task
The choice is validated empirically in Appendix C rather than derived; results depend on that empirical regularity.
standard math Standard Mamba selective SSM mathematics
Adopted from Gu and Dao (2023) without modification; accepted as prior work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss." pith.science (2026). https://pith.science/paper/NVPXVMBB

@misc{pith2026250207575,
  author       = {Pith},
  title        = {Pith review of: Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVPXVMBB}},
  note         = {Machine review of arXiv:2502.07575}
}

read the original abstract

Prior efforts in building computer-assisted pronunciation training (CAPT) systems often treat automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD) as separate fronts: the former aims to provide multiple pronunciation aspect scores across diverse linguistic levels, while the latter focuses instead on pinpointing the precise phonetic pronunciation errors made by non-native language learners. However, it is generally expected that a full-fledged CAPT system should perform both functionalities simultaneously and efficiently. In response to this surging demand, we in this work first propose HMamba, a novel CAPT approach that seamlessly integrates APA and MDD tasks in parallel. In addition, we introduce a novel loss function, decoupled cross-entropy loss (deXent), specifically tailored for MDD to facilitate better-supervised learning for detecting mispronounced phones, thereby enhancing overall performance. A comprehensive set of empirical results on the speechocean762 benchmark dataset demonstrates the effectiveness of our approach on APA. Notably, our proposed approach also yields a considerable improvement in MDD performance over a strong baseline, achieving an F1-score of 63.85%. Our codes are made available at https://github.com/Fuann/hmamba

Figures

Figures reproduced from arXiv: 2502.07575 by the authors.

**Figure 3.** Difference between (a) the original crossentropy loss and (b) the decoupled cross-entropy loss, given the text prompt “crime.” [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

JCAPT: A Joint Modeling Approach for CAPT
cs.CL 2025-06 conditional novelty 4.0 of 10

JCAPT, a Mamba-based joint APA and MDD model with phonological features and think tokens, improves mispronunciation detection and several scoring aspects on speechocean762 over JAM.

Reference graph

Works this paper leans on

18 extracted references · 12 canonical work pages · cited by 1 Pith paper

[1]

However, it is generally expected that a full-fledged CAPT system should perform both functionalities simultaneously and efficiently

Abstract Prior efforts in building computer-assisted pronunciation training (CAPT) systems often treat automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD) as separate fronts: the former aims to provide multiple pronunciation aspect scores across diverse linguistic levels, while the latter focuses instead on pinpoint...

work page 2009
[2]

These modules collectively generate the corresponding aspect score sequence 𝐬𝑔 for each linguistic granularity 𝑔, as well as the phonetic error states 𝐞 and diagnosis 𝐲

Specifically, HMamba synthesizes the APA and MDD modules, each of which contains multiple regressors and a classifier, respectively. These modules collectively generate the corresponding aspect score sequence 𝐬𝑔 for each linguistic granularity 𝑔, as well as the phonetic error states 𝐞 and diagnosis 𝐲. Furthermore, each classifier and regressor is implemen...

work page 2000
[3]

These errors usually have clear-cut distinctions between correct and incorrect ones, and can be easily quantified through deletions, substitutions, and insertions

In contrast to APA, MDD focuses more on non-native speakers’ phonetic pronunciation errors (Chen and Li, 2016). These errors usually have clear-cut distinctions between correct and incorrect ones, and can be easily quantified through deletions, substitutions, and insertions. For instance, a number of MDD models are designed to capitalize on classifier-bas...

work page 2021
[5]

Notably, there are several studies investigating the bidirectional processing of Mamba (Liang et al., 2024; Zhang et al., 2024; Jiang et al., 2024)

Specifically, for input 𝐇𝑔𝑖 to the Mamba block at granularity level 𝑔, the output 𝐇𝑔𝑖+1 of the block is: 𝐇′𝑔𝑖=BiMamba(LayerNorm(𝐇𝑔𝑖))+𝐇𝑔𝑖 (4) 𝐇𝑔𝑖+1=FFN(LayerNorm(𝐇′𝑔𝑖))+𝐇′𝑔𝑖 (5) 3https://huggingface.co/microsoft/wavlm-large where BiMamba denotes the bidirectional Mamba layer and FFN refers to the feed-forward module, respectively. Notably, there are sever...

work page 2024
[7]

The APA module contains one regressor that aims to predict the phone-level aspect score 𝑠0𝑔𝑝ℎ𝑛(accuracy)

In phone-level modeling, we first use 𝐇𝑔0𝑝ℎ𝑛 as the input into 𝐿𝑝-layer Mamba blocks to obtain the phone-level contextualized representations 𝐇𝑔𝐿𝑝𝑝ℎ𝑛: 𝐇𝑔𝐿𝑝𝑝ℎ𝑛=MambaBlock𝑝ℎ𝑛(𝐇𝑔0𝑝ℎ𝑛) (11) Subsequently, 𝐇𝑔𝐿𝑝𝑝ℎ𝑛 are then propagated forward into the APA module and the MDD module for solving a regression and a sequence classification problem, respectively. The ...

work page 2022
[12]

arXiv preprint arXiv:2312.00752

Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed

arXiv
[14]

arXiv preprint arXiv:2404.15772

Bi-Mamba4TS: Bidirectional mamba for time series forecasting. arXiv preprint arXiv:2404.15772. Binghuai Lin, Liyuan Wang, Hongwei Ding, Xiaoli Feng. 2021a. Improving L2 English rhythm evaluation with automatic sentence stress detection. In Proceedings of IEEE Spoken Language Technology Workshop (SLT), pages 713-719. Binghuai Lin and Liyuan Wang. 2021b. De...

arXiv
[15]

arXiv preprint arXiv:2401.09417

Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417. Xiangyu Zhang, Qiquan Zhang, Hexin Liu, Tianyi Xiao, Xinyuan Qian, Beena Ahmed, Eliathamby Ambikairajah, Haizhou Li, and Julien Epps

arXiv

Show all 18 references

[16]

arXiv preprint arXiv:2405.12609

Mamba in Speech: Towards an alternative to self-attention. arXiv preprint arXiv:2405.12609. Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey, and Yujun Wang

arXiv
[17]

The combining weights 𝜔𝑔 for APA loss are uniformly set to 1.0 for each granularity level 𝑔

Hyperparameters setting: 𝜏 in attention pooling layer is set to 1.0. The combining weights 𝜔𝑔 for APA loss are uniformly set to 1.0 for each granularity level 𝑔. Parameters 𝛼 and 𝛽 are tuned to be 0.7 and 0.003, respectively. 5 https://kaldi-asr.org/models/m13 B Feature Ablati...

2022
[18]

and speech enhancement (Zhang et al., 2024). Computational efficiency: We further investigate the computational efficiency of two variants of architectures with the number of their parameters and multiply-accumulate operations (MACs). In Table 7, we observe that the model equi...

2024
[2009]

reading-aloud

developed by Educational Testing Service (ETS). Typically, a de-facto archetype system for CAPT encompasses a “reading-aloud” scenario, where a non-native speaker is given a text prompt and instructed to pronounce it correctly. In this context, previous literature broadly divi...

2023
[2015]

ETS Research Report Series 2015(1):1–11

Automated scoring for the TOEFL Junior® comprehensive writing and speaking test. ETS Research Report Series 2015(1):1–11. Kaiqi Fu, Jones Lin, Dengfeng Ke, Yanlu Xie, Jinsong Zhang, and Binghuai Lin

2015
[2017]

and class-balanced loss (Cui et al., 2019), to tackle the imbalance issue in MDD. However, in most end-to-end MDD methods, where the labels are phones instead of mispronunciations (0 or 1s), directly applying the existing loss-balancing methods on phones is implicit and can be...

2021
[2021]

arXiv preprint arXiv:2104.08428

A full text-dependent end to end mispronunciation detection and diagnosis with easy data augmentation techniques. arXiv preprint arXiv:2104.08428. Yuan Gong, Ziyi Chen, Iek-Heng Chu, Peng Chang, and James Glass

arXiv
[2022]

that use prepended [CLS] tokens to predict utterance-level scores. According to the results shown in Table 2, HMamba outperforms PMamba and LMamba across all assessment aspects, highlighting the advantages of its hierarchical structure for the APA task. This finding aligns wit...

2023
[2023]

have suggested that hierarchical modeling structures is more amenable than parallel modeling structures (Gong et al., 2022). To capture the linguistic hierarchy while retaining the cross-aspect relations within the same linguistic unit, we design and instantiate our model with...

2022
[2024]

Yassine Kheir, Ahmed Ali, and Shammur Chowdhury

arXiv preprint arXiv:2403.18257. Yassine Kheir, Ahmed Ali, and Shammur Chowdhury

arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

However, it is generally expected that a full-fledged CAPT system should perform both functionalities simultaneously and efficiently

Abstract Prior efforts in building computer-assisted pronunciation training (CAPT) systems often treat automatic pronunciation assessment (APA) and mispronunciation detection and diagnosis (MDD) as separate fronts: the former aims to provide multiple pronunciation aspect scores across diverse linguistic levels, while the latter focuses instead on pinpoint...

work page 2009

[2] [2]

These modules collectively generate the corresponding aspect score sequence 𝐬𝑔 for each linguistic granularity 𝑔, as well as the phonetic error states 𝐞 and diagnosis 𝐲

Specifically, HMamba synthesizes the APA and MDD modules, each of which contains multiple regressors and a classifier, respectively. These modules collectively generate the corresponding aspect score sequence 𝐬𝑔 for each linguistic granularity 𝑔, as well as the phonetic error states 𝐞 and diagnosis 𝐲. Furthermore, each classifier and regressor is implemen...

work page 2000

[3] [3]

These errors usually have clear-cut distinctions between correct and incorrect ones, and can be easily quantified through deletions, substitutions, and insertions

In contrast to APA, MDD focuses more on non-native speakers’ phonetic pronunciation errors (Chen and Li, 2016). These errors usually have clear-cut distinctions between correct and incorrect ones, and can be easily quantified through deletions, substitutions, and insertions. For instance, a number of MDD models are designed to capitalize on classifier-bas...

work page 2021

[4] [5]

Notably, there are several studies investigating the bidirectional processing of Mamba (Liang et al., 2024; Zhang et al., 2024; Jiang et al., 2024)

Specifically, for input 𝐇𝑔𝑖 to the Mamba block at granularity level 𝑔, the output 𝐇𝑔𝑖+1 of the block is: 𝐇′𝑔𝑖=BiMamba(LayerNorm(𝐇𝑔𝑖))+𝐇𝑔𝑖 (4) 𝐇𝑔𝑖+1=FFN(LayerNorm(𝐇′𝑔𝑖))+𝐇′𝑔𝑖 (5) 3https://huggingface.co/microsoft/wavlm-large where BiMamba denotes the bidirectional Mamba layer and FFN refers to the feed-forward module, respectively. Notably, there are sever...

work page 2024

[5] [7]

The APA module contains one regressor that aims to predict the phone-level aspect score 𝑠0𝑔𝑝ℎ𝑛(accuracy)

In phone-level modeling, we first use 𝐇𝑔0𝑝ℎ𝑛 as the input into 𝐿𝑝-layer Mamba blocks to obtain the phone-level contextualized representations 𝐇𝑔𝐿𝑝𝑝ℎ𝑛: 𝐇𝑔𝐿𝑝𝑝ℎ𝑛=MambaBlock𝑝ℎ𝑛(𝐇𝑔0𝑝ℎ𝑛) (11) Subsequently, 𝐇𝑔𝐿𝑝𝑝ℎ𝑛 are then propagated forward into the APA module and the MDD module for solving a regression and a sequence classification problem, respectively. The ...

work page 2022

[6] [12]

arXiv preprint arXiv:2312.00752

Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed

arXiv

[7] [14]

arXiv preprint arXiv:2404.15772

Bi-Mamba4TS: Bidirectional mamba for time series forecasting. arXiv preprint arXiv:2404.15772. Binghuai Lin, Liyuan Wang, Hongwei Ding, Xiaoli Feng. 2021a. Improving L2 English rhythm evaluation with automatic sentence stress detection. In Proceedings of IEEE Spoken Language Technology Workshop (SLT), pages 713-719. Binghuai Lin and Liyuan Wang. 2021b. De...

arXiv

[8] [15]

arXiv preprint arXiv:2401.09417

Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417. Xiangyu Zhang, Qiquan Zhang, Hexin Liu, Tianyi Xiao, Xinyuan Qian, Beena Ahmed, Eliathamby Ambikairajah, Haizhou Li, and Julien Epps

arXiv