Syllabic-Structure Decoder for Automatic Speech Recognition in Vietnamese

Kiet Van Nguyen; Long Hoang Huu Nguyen; Ngan Luu-Thuy Nguyen; Nghia Hieu Nguyen; Quan Ngoc Hoang

arxiv: 2605.27874 · v1 · pith:ML3DFPIMnew · submitted 2026-05-27 · 💻 cs.CL

Syllabic-Structure Decoder for Automatic Speech Recognition in Vietnamese

Nghia Hieu Nguyen , Quan Ngoc Hoang , Long Hoang Huu Nguyen , Kiet Van Nguyen , Ngan Luu-Thuy Nguyen This is my paper

Pith reviewed 2026-06-29 13:22 UTC · model grok-4.3

classification 💻 cs.CL

keywords automatic speech recognitionVietnamesephoneme modelingsyllabic structuredecodermulti-dialectvocabulary size

0 comments

The pith

A syllabic-structure decoder models Vietnamese ASR at the phoneme level and outperforms orthographic baselines with a smaller vocabulary.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a decoder for Vietnamese automatic speech recognition that works at the phoneme level rather than with characters or words. It explicitly builds in the phonological rules that form valid syllables, so the model produces only legal syllable structures from a compact set of phonemes. This replaces orthographic units that ignore phonetic structure and demand large vocabularies. On the LSVSC benchmark for standard speech and the UIT-ViMD multi-dialect set, the approach beats strong pretrained systems such as PhoWhisper and Wav2Vec2 while using no extra training data. The design reduces vocabulary size yet improves transcription accuracy by staying closer to how speech is actually produced.

Core claim

The syllabic-structure decoder models speech at the phoneme level instead of the orthographic level by explicitly capturing the phonological composition of syllables, enabling the decoder to generate valid syllabic structures from a compact phonemic inventory. This design more closely aligns with the phonetic realization of speech while significantly reducing vocabulary size. On the LSVSC and UIT-ViMD benchmarks the method consistently outperforms baselines including PhoWhisper and Wav2Vec2 despite the smaller vocabulary and absence of additional training resources.

What carries the argument

The Syllabic-Structure Decoder, which captures phonological composition of syllables to produce only valid structures from a compact phonemic inventory.

If this is right

Transcription accuracy rises on both standard and multi-dialect Vietnamese speech without added training data or larger vocabularies.
Vocabulary size drops substantially while coverage of valid syllables remains complete.
Phoneme-level modeling with syllable constraints replaces the need for large orthographic inventories in this language.
The decoder aligns output generation more directly with phonetic production rules than character or word prediction does.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same explicit syllable modeling may help other tonal languages where syllable boundaries carry phonetic weight.
Smaller vocabularies could reduce memory and compute needs for on-device Vietnamese ASR.
Language-specific phonetic rules might substitute for some of the scale currently obtained through large pretraining.

Load-bearing premise

Explicitly encoding syllabic phonological rules in the decoder produces structures that match phonetic speech realization better than orthographic units do.

What would settle it

An experiment in which the syllabic decoder is given the same vocabulary size as an orthographic baseline and still shows no accuracy gain on either LSVSC or UIT-ViMD.

Figures

Figures reproduced from arXiv: 2605.27874 by Kiet Van Nguyen, Long Hoang Huu Nguyen, Ngan Luu-Thuy Nguyen, Nghia Hieu Nguyen, Quan Ngoc Hoang.

**Figure 1.** Figure 1: Syllabic Structure in Vietnamese languages. Notable examples include Whisper (Le et al., 2024) and Wav2Vec (Baevski et al., 2020). These systems typically predict orthographic tokens, such as characters or subwords. Phoneme-based representations have long been used in traditional ASR systems (Rabiner, 1989; Jelinek, 1997; Hinton et al., 2012) because they better reflect the acoustic realization of speech … view at source ↗

**Figure 2.** Figure 2: The Syllabic-Structure Decoder. vector is then projected back to the latent space of the model by a linear function: embk = Weembk + b ∈ R dim (2) where We ∈ R dim×3·dim represents the weights of the linear layer and b ∈ R dim is the bias vector. 3.3.2 Attention over Acoustic Context This step is similar to the standard decoding mechanism (Vaswani et al., 2017a), where the unified representation attends t… view at source ↗

**Figure 3.** Figure 3: Phone error rate (PER, %) by province for Transformer and Conformer models using the proposed [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Example for the consistency between graphemes and phonemes in Vietnamese. – o /O/. Eg: chăm sóc, mong ngóng, trong veo. – oo /O:/. Eg: xoong chảo. – ô /o/. Eg: trống đồng, ống hút, cố nhân. – ơ /@/. Eg: mơ mộng, cơ nhỡ, chơi bời. • 10 final consonants have 12 writing forms: – i or y /i “ /. Eg: làng chài, mỏi mệt, chạy đua, bay nhảy. – m /m/. Eg: êm ấm, nhiệm màu, mâm cỗ. – n /n/. Eg: nan giải, non nớt, tả… view at source ↗

read the original abstract

Most Automatic Speech Recognition (ASR) systems formulate transcription as a prediction problem over orthographic units such as characters, subwords, or words. Although effective, such representations do not explicitly reflect the phonetic structure of speech and often require large vocabularies to maintain adequate coverage. In this work, we are motivated from the phonemic features of Vietnamese to propose a Syllabic-Structure Decoder for ASR, which models speech at the phoneme level instead of the orthographic level. Our approach explicitly captures the phonological composition of syllables, enabling the decoder to generate valid syllabic structures from a compact phonemic inventory. This design more closely aligns with the phonetic realization of speech while significantly reducing vocabulary size. Experimental results on two benchmarks: LSVSC, representing standard speech, and UIT-ViMD, a multi-dialect corpus containing diverse regional pronunciations, show that our method consistently outperforms strong previous baselines, especially pretrained baselines such as PhoWhisper and Wav2Vec2, despite using a substantially smaller vocabulary and no additional training resources. These results highlight the effectiveness of phoneme-based syllabic modeling for ASR in this language. Code for experimental reproducibility will be publicly available upon the acceptance of this paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The syllabic decoder is a straightforward language-specific tweak that trades orthographic units for phoneme-level syllable rules and claims smaller vocab plus better results on two Vietnamese benchmarks.

read the letter

The main takeaway is a decoder that models at the phoneme level while enforcing valid syllable structures for Vietnamese, which lets them drop vocabulary size sharply compared with character or subword baselines and still report lower error rates than PhoWhisper and Wav2Vec2 on LSVSC and UIT-ViMD.

What the work does is take the phonological fact that Vietnamese syllables have a clear internal composition and bake that constraint directly into the decoder instead of hoping the model learns it from data. That is the concrete difference from the cited orthographic approaches, and the abstract presents it as the reason for both the vocab reduction and the performance edge without needing extra training resources.

The soft spots are the usual ones for an abstract-only view: no error rates, no error bars, no ablation on how the syllable rules are actually enforced in the architecture, and no detail on whether the pretrained baselines received the same fine-tuning protocol. Those gaps make it impossible to judge how much of the gain comes from the syllabic modeling versus other implementation choices. The promise of public code is noted but does not substitute for the missing numbers and controls in the current text.

This is for people working on ASR for syllable-timed or tonal languages who want decoder designs that respect phonology rather than generic subword tokenization. A reader looking for incremental but practical improvements in low-resource settings would find the idea worth examining once the full methods and tables appear.

The paper deserves a serious referee because the claim is falsifiable, the benchmarks are named, and the motivation is tied to measurable language properties rather than vague intuition.

Referee Report

1 major / 1 minor

Summary. The manuscript proposes a Syllabic-Structure Decoder for Vietnamese ASR that models transcription at the phoneme level rather than orthographic units (characters, subwords, or words). By explicitly capturing the phonological composition of syllables, the decoder generates valid syllabic structures from a compact phonemic inventory. This design is claimed to align more closely with phonetic realization while reducing vocabulary size. Experiments on the LSVSC (standard speech) and UIT-ViMD (multi-dialect) benchmarks are reported to show consistent outperformance over baselines including PhoWhisper and Wav2Vec2, with no additional training resources used. Code release is promised upon acceptance.

Significance. If the empirical claims hold with proper controls and ablations, the work could demonstrate a linguistically grounded efficiency gain for ASR in tonal languages with clear syllabic structure, by leveraging phonemic modeling over larger orthographic vocabularies. The explicit commitment to public code supports reproducibility.

major comments (1)

[Abstract] Abstract: The central claim of consistent outperformance on LSVSC and UIT-ViMD (including over pretrained models PhoWhisper and Wav2Vec2) is presented without any quantitative results, error rates (e.g., WER or CER), statistical tests, error bars, ablation studies, or implementation details. This absence prevents verification that the data support the claim and makes the empirical contribution impossible to assess.

minor comments (1)

[Abstract] Abstract: The phrase 'especially pretrained baselines' is imprecise; clarify whether the gains hold against all listed baselines or only the pretrained subset.

Simulated Author's Rebuttal

1 responses · 0 unresolved

Thank you for the constructive feedback on our manuscript. We appreciate the referee's focus on ensuring the empirical claims are clearly supported. We address the single major comment below and will revise the abstract accordingly.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim of consistent outperformance on LSVSC and UIT-ViMD (including over pretrained models PhoWhisper and Wav2Vec2) is presented without any quantitative results, error rates (e.g., WER or CER), statistical tests, error bars, ablation studies, or implementation details. This absence prevents verification that the data support the claim and makes the empirical contribution impossible to assess.

Authors: We agree that the abstract would be strengthened by including key quantitative results to support the claims. The body of the manuscript already reports detailed WER/CER metrics, comparisons against PhoWhisper and Wav2Vec2, ablations, and implementation details on the two benchmarks. In the revised version, we will update the abstract to highlight specific performance numbers (e.g., WER reductions) and standard evaluation metrics. We will also note the compact vocabulary size and lack of additional training resources. Code release is already promised upon acceptance. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper presents an empirical ASR method using a phoneme-level syllabic decoder and reports direct performance gains over external baselines (PhoWhisper, Wav2Vec2) on LSVSC and UIT-ViMD. No equations, fitted parameters relabeled as predictions, self-citations, or uniqueness theorems appear in the supplied text. The central claim rests on standard benchmark comparisons with a smaller vocabulary and no extra resources; the derivation chain contains no self-referential reductions.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no information on free parameters, axioms, or invented entities; the contribution is described as an empirical modeling change.

pith-pipeline@v0.9.1-grok · 5759 in / 1206 out tokens · 46383 ms · 2026-06-29T13:22:02.538551+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages

[1]

SpeechBrain: A general-purpose speech toolkit,

Conformer: Convolution-augmented trans- former for speech recognition. In21st Annual Con- ference of the International Speech Communication Association, Interspeech 2020, Virtual Event, Shang- hai, China, October 25-29, 2020, pages 5036–5040. ISCA. Nghia Hieu Nguyen, Dat Tien Nguyen, and Ngan Luu- Thuy Nguyen. 2025. Vietnamese words are not con- structed ...

work page arXiv 2020
[2]

Vietnamese is a monosyllabic language
[3]

That is, in this language, we do not have the linking pronunciation as in English, and every phoneme has persistent writing forms

The correspondence between graphemes and phonemes in Vietnamese is consistent. That is, in this language, we do not have the linking pronunciation as in English, and every phoneme has persistent writing forms. In Vietnamese, each syllable has three components: initials, rhymes, and tones. Rhyme has smaller components, which are glide, vowel, and final. We...

[1] [1]

SpeechBrain: A general-purpose speech toolkit,

Conformer: Convolution-augmented trans- former for speech recognition. In21st Annual Con- ference of the International Speech Communication Association, Interspeech 2020, Virtual Event, Shang- hai, China, October 25-29, 2020, pages 5036–5040. ISCA. Nghia Hieu Nguyen, Dat Tien Nguyen, and Ngan Luu- Thuy Nguyen. 2025. Vietnamese words are not con- structed ...

work page arXiv 2020

[2] [2]

Vietnamese is a monosyllabic language

[3] [3]

That is, in this language, we do not have the linking pronunciation as in English, and every phoneme has persistent writing forms

The correspondence between graphemes and phonemes in Vietnamese is consistent. That is, in this language, we do not have the linking pronunciation as in English, and every phoneme has persistent writing forms. In Vietnamese, each syllable has three components: initials, rhymes, and tones. Rhyme has smaller components, which are glide, vowel, and final. We...