Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR

Eng Siong Chng; Hexin Liu; Yizhou Peng

REVIEW 2 major objections 5 minor 26 references

Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A speech LLM that conditions on language-specific prompts and bi-directional transcript context reduces multilingual conversational ASR error by 18 percent relative, beating a model trained on four times more data.

desk verdict Solid engineering paper on context-enhanced multilingual conversational ASR; the big context-beats-data claim rests on a 0.07 MER gap on the validation set, but the core method is plausible and the paper deserves refereeing. read the letter →

arxiv 2506.13396 v2 pith:5HJF6FA5 submitted 2025-06-16 cs.CL eess.AS

Yizhou Peng , Hexin Liu , Eng Siong Chng This is my paper · ORCID

classification cs.CLeess.AS

keywords conversationalASRspeechlargelanguagemodelbi-directionalcontextlanguage-specificpromptscontextualmaskingtwo-stagedecodingmultilingualrecognitionMLC-SLM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the missing ingredient in multilingual conversational ASR is context, not just data. It proposes a speech large language model that receives language-specific prompts and both previous and future transcript context, and that is trained with a character-level masking strategy so it does not become dependent on perfect context. At inference, a two-stage pipeline first decodes each segment alone, then re-decodes it with neighboring hypotheses as context. On the 1500-hour MLC-SLM validation set this reaches 13.56% average mix error rate, an 18% relative improvement over the 16.60% baseline, and it edges out a model trained on 6000 hours of data. A sympathetic reader would take this as evidence that context-aware modeling beats brute-force data scaling for conversational ASR.

What carries the argument

The mechanism that carries the argument is a combination of context-conditioned training and context-conditioned decoding. Training samples pair each speech segment with a language-specific prompt and, when available, the preceding and following transcript as text context; a character-level masking algorithm independently drops 0–25% of each context side in one to three random spans with 50% probability, simulating the imperfect hypotheses the model will see at inference. Decoding runs in two stages: first each segment is transcribed without context, then those hypotheses are fed back as history and future context for a second, context-aware decode. The gap between the second-stage result (13.56%) and a ground-truth-context upper bound (13.16%) measures how much imperfect first-pass hypotheses still cost.

What would settle it

Score both S3-Stage2 and S4 on the held-out evaluation set once reference transcriptions are released, and compute a bootstrap confidence interval or paired significance test on the 32-hour validation set; if the 13.56%-to-13.63% difference is not statistically significant, the claim that context beats four times the training data loses its support.

Watch

Extended reading notes

Core claim

The paper's central claim is that conditioning a speech LLM on language-specific prompts and bi-directional transcript context—both during training and through a two-stage decoding pipeline—substantially improves multilingual conversational ASR. With the 1500-hour MLC-SLM training set, the best configuration (S3-Stage2) reaches 13.56% average mix error rate on the validation set, down from 16.60% for the baseline, an 18% relative gain. This configuration also beats S4 (13.63%), a model trained with 6000 hours of data that includes 4500 hours of external single-segment speech, despite using one quarter of the training data. The paper interprets the result as evidence that contextual modeling yields larger gains than data scale-up for continuous conversational ASR.

Load-bearing premise

The load-bearing premise is that the 0.07-point gap between S3-Stage2 (13.56%) and S4 (13.63%) is real signal, not noise; the paper measures it on one 32-hour validation set with no confidence interval, ignores the held-out evaluation set because its transcriptions are unavailable, and also uses the same validation set for early stopping.

Editorial extensions

If this is right

Language-specific prompts alone cut average validation error from 16.60% to 14.87%, with the largest single-language drop in Japanese CER (24.07% to 17.98%).
Training with context improves even context-agnostic decoding: S2-Stage1 reaches 14.30% and S3-Stage1 reaches 13.84% without using context at inference.
Adding the second decoding stage yields further gains, bringing S3 to 13.56% from 13.84%.
Using ground-truth context in Stage 2 gives 13.16%, so roughly 0.40 MER of headroom remains if first-pass hypotheses were perfect.
A 1500-hour context-aware model outperforms a 6000-hour context-agnostic model, implying diminishing returns from data scaling relative to context modeling on this benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural testable extension is to replace random character masking with masks derived from actual ASR errors, which should show whether the robustness gain transfers to realistic corruption patterns.
The same two-stage context-then-redecode recipe could transfer to other conditional generation tasks with ordered context, such as document-level machine translation or dialogue summarization.
If the validation-set advantage holds on held-out evaluation data, the result suggests that for low-resource multilingual conversational ASR, context engineering may be a more cost-effective investment than collecting another 4500 hours of read speech.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: the paper has a genuinely useful method for feeding context into a speech LLM for conversational ASR, and the big headline claim—that context beats data scaling—is not supported by the reported numbers. The method itself is worth taking seriously.

What's new: the combination of language-specific prompts, bi-directional history/future context, character-level contextual masking during training, and a two-stage decode (context-agnostic first, then context-conditioned re-decode). The masking strategy is a thoughtful way to handle the train-test mismatch from feeding the model its own noisy hypotheses. The ablations are properly structured: S1 isolates prompts, S2 adds history, S3 adds future, S4 is the data-scale control. The early results are large and consistent: baseline 16.60 to S1 14.87 to S3-Stage1 13.84, with gains in nearly every language. Those I trust. The oracle upper bound (S3-Stage2-G at 13.16) is a sensible sanity check.

Soft spots: the abstract and conclusion emphasize that the 1500-hour S3-Stage2 (13.56) outperforms the 6000-hour S4 (13.63). That 0.07 MER gap is the load-bearing claim and it is not statistically supported. It is computed on a single 32-hour validation set used for early stopping, there are no confidence intervals or significance tests, and per-language results split almost evenly (8 vs 7). A paired test across language-level error rates would likely not reject the null. They also ignore the evaluation set for lack of transcriptions—honest, but it leaves no held-out check. The fix is straightforward: either run significance testing on a proper held-out test set, or tone the claim down to 'comparable to S4 with a quarter of the data.' As written, the abstract overstates the evidence. Minor: no code or data release, and the masking hyperparameters (p, alpha, k) are not swept, though the chosen values seem reasonable.

Bottom line: solid engineering contribution, plausible mechanism, overstated headline. I'd send it to peer review and ask for the claim to be rebalanced and significance testing added. It's a good fit for a speech conference once that's done.

Referee Report

2 major / 5 minor

Summary. The paper proposes a recipe for multilingual conversational ASR with a speech LLM (Whisper-large-v3 encoder, linear adapter, Gemma-2-2B backbone): language-specific prompts, bi-directional (history and future) context as text prefixes, character-level random masking of those contexts during training, and a two-stage decoding pipeline that first decodes segments independently and then re-decodes with neighboring hypotheses as context. Experiments on the 1500-hour MLC-SLM corpus with a 32-hour validation set compare a baseline (English prompt, 1500 h) to S1 (language-specific prompts), S2 (history context), S3 (bi-directional context), and S4 (language-specific prompts plus 4467 h of CommonVoice). The main empirical claim is that S3 with Stage-2 decoding reaches 13.56% MER, an 18% relative improvement over the 16.60% baseline, and outperforms the 6000-hour S4 system (13.63% MER). An oracle variant using ground-truth context (13.16% MER) is presented as an upper bound.

Significance. If substantiated, the main contribution is a simple and potentially practical demonstration that context conditioning, rather than brute-force data scaling, is the dominant factor for this conversational ASR setting: the S3-Stage2 result (13.56% MER) is close to the ground-truth-context upper bound (13.16%) and matches or beats the much larger-data model. The paper's strengths are the clean ablation ladder (Baseline to S1 to S2 to S3), the oracle upper-bound sanity check, and the publication of per-language and per-accent numbers. However, the paper is purely empirical and does not provide code, confidence intervals, or significance tests; the most striking claim (beating the 6000-hour model) therefore rests on a 0.07-point gap on a single validation set. With corrections to the data-quantity accounting and added uncertainty quantification, the paper would be a useful contribution.

major comments (2)

[Abstract; Section 3.2/Table 2; Section 4] The data-comparison claim is overstated. S2 and S3 are trained on the original 1500-hour MLC-SLM training set combined with another 1500 hours of context-augmented training samples, giving 3000 hours of training data (Table 2: 1500x2). Therefore the abstract's 'only 1500 hours of training data' and Section 4's 'only one quarter of the data' are incorrect; S3 uses half as much training data as S4, not a quarter. This should be corrected in the abstract, introduction, and Section 4, and the interpretation of diminishing returns from data scaling should be revised accordingly.
[Section 4/Table 3] The headline claim that S3-Stage2 (13.56% MER) outperforms S4 (13.63%) is not statistically supported. The 0.07-point gap is measured on a single 32-hour validation set that is also used for early stopping (Section 3.2), and no confidence intervals or significance tests are reported. Per-language results are split nearly evenly: S3 is better on 8 of the 15 language/accent groups and S4 is better on 7, with several per-language differences in the opposite direction exceeding the average gap (e.g., Thai 10.90 vs 9.92; Spanish 10.60 vs 10.00). The authors should add uncertainty quantification, such as segment-level bootstrap confidence intervals or a paired test across the 15 groups, and verify on the evaluation set when transcriptions become available. Without this, the abstract and conclusion's claim of outperforming the 6000-hour model is not established.

minor comments (5)

[Section 4, first paragraph] The sentence 'our strong Baseline system shows 5% absolute MER degradation compared against the official Whisper-Qwen baseline and Whisper-Llama baseline' has the sign backwards: the baseline is 16.60% versus the official 21.49%/21.56%, i.e., an improvement of about 4.9 absolute points, not a degradation.
[Table 2] The column header 'Stratagy' should be 'Strategy', and the entry '1500x2' would be clearer if the text explicitly stated that S2 and S3 use 3000 hours of training samples formed by adding 1500 hours of context-augmented samples to the original 1500-hour set.
[Section 2.1 and Figure 1] The sentence 'When only history or future context exists, we set half of the context and discard the remaining prompts' is ambiguous; specify whether half of the context tokens or half of the prompt template is meant, and state the exact template used when only one side is available.
[Algorithm 1] The notation for the randomly selected span (shown as 'r i') is broken by typesetting, and the edge case where alpha=0 yields k=1 and s=0 should be clarified so the removal is a no-op rather than an empty-span removal.
[General] The paper would benefit from a statement on code and data availability, including the scripts that generate the context-augmented training samples, since the masking strategy is central to reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports measured ASR results from fixed training and decoding pipelines, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper makes an empirical performance claim rather than a derived prediction. The central results (Baseline 16.60% MER, S1 14.87%, S3-Stage1 13.84%, S3-Stage2 13.56%, and S4 13.63% on the MLC-SLM validation set) are obtained by executing standard training and decoding procedures: language-specific prompts, a randomized context-masking augmentation (Algorithm 1), and a two-stage decoding pipeline (Section 2.3). No parameter is fitted to the reported outcome and then renamed as a prediction; the groundtruth-context result S3-Stage2-G is explicitly presented as an oracle upper bound, not as a claimed method. The model configurations in Table 2 differ by training data and prompt design, so each row is an independent measurement rather than a construction that forces its own output. The paper does not invoke a uniqueness theorem, and no load-bearing claim reduces to a self-citation; citations to SLAM-LLM, Whisper, Gemma, CommonVoice, and baselines are external resources or standard toolkits. The fact that the small S3-vs-S4 gap is measured on a single 32-hour validation set without confidence intervals, and that early stopping used the same set, is a legitimate statistical-robustness concern, but it is not circularity: the reported numbers remain honest measurements of the systems as trained. Therefore the derivation chain is self-contained, and the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or formal entities; it configures existing components (Whisper, Gemma, linear projector) with hand-chosen hyperparameters. The main implicit assumptions are about transferability of the pretrained encoders and the adequacy of the 32-hour validation set for drawing strong conclusions.

free parameters (6)

Context masking probability p = 0.5
Each context side is kept intact with probability 0.5 in Algorithm 1; chosen by hand and affects how often the model sees corrupted context.
Context removal ratio alpha = Uniform(0, 0.25)
When masking is applied, 0 to 25% of characters are removed in Algorithm 1; this range was chosen by hand.
Number of masked spans k = 1 to 3
The total removal budget is split into one to three contiguous spans in Algorithm 1.
Learning rate = 5e-5
Used for all models as stated in Section 3.2; not justified by a search.
Early stopping tolerance = 2000 steps
Stops training if validation accuracy does not improve for 2000 steps (Section 3.2); this makes the validation set part of the model selection process.
Beam size and max repeated n-grams = Beam 4; max 5-grams
Decoding hyperparameters chosen to prevent hallucination (Section 3.2).

assumptions (5)

domain assumption Whisper-large-v3 encoder features are informative enough for all eleven MLC-SLM languages to support fine-tuned decoding.
The model freezes the Whisper encoder and relies on its audio features across the eleven languages (Section 2.1).
domain assumption Gemma-2-2B is a suitable LLM backbone and can learn acoustic-to-text mapping when fully fine-tuned.
No comparison with alternative LLM backbones is provided; the SLLM architecture assumes this capacity (Section 2.1).
domain assumption The MLC-SLM validation set with oracle segmentation and speaker labels is a fair and sufficient measure of conversational ASR quality.
All conclusions use the 32-hour validation set; no test set is used (Sections 3.1 and 4).
ad hoc to paper Character-level masking with 0 to 25% removal mimics realistic hypothesis errors during two-stage inference.
This modeling assumption motivates Algorithm 1 but is not validated against actual Stage 1 error patterns.
domain assumption WER for alphabetic languages and CER for Japanese, Korean, and Thai can be averaged together as a meaningful Mix Error Rate.
The paper reports average MER across languages using different error metrics (Section 4), which assumes comparability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR." pith.science (2026). https://pith.science/paper/5HJF6FA5

@misc{pith2026250613396,
  author       = {Pith},
  title        = {Pith review of: Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5HJF6FA5}},
  note         = {Machine review of arXiv:2506.13396}
}

read the original abstract

This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR). We propose a character-level contextual masking strategy during training, which randomly removes portions of the context to enhance robustness and better emulate the flawed transcriptions that may occur during inference. For decoding, a two-stage pipeline is utilized: initial isolated segment decoding followed by context-aware re-decoding using neighboring hypotheses. Evaluated on the 1500-hour Multilingual Conversational Speech and Language Model (MLC-SLM) corpus covering eleven languages, our method achieves an 18% relative improvement compared to a strong baseline, outperforming even the model trained on 6000 hours of data for the MLC-SLM competition. These results underscore the significant benefit of incorporating contextual information in multilingual continuous conversational ASR.

Figures

Figures reproduced from arXiv: 2506.13396 by the authors.

**Figure 1.** Proposed Model Architecture. In our model, we utilize the Whisper-large-v3 encoder as an audio encoder, and the Gemma2-2B as the backbone LLM. During training, the audio encoder is frozen, and both the linear projector and the LLM are fully finetuned. Some examples of language-specific and contextual templates, which serve as the text prompt for the SLLM, are shown on the righthand side of the figure. All prompts … view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 12 canonical work pages

[1]

Introduction Conversational speech recognition (Conv-ASR), which aims to transcribe natural spoken language accurately, remains a signif- icant challenge in the speech processing area [1, 2]. Unlike iso- lated speech segments, conversational speech typically involves spontaneous, unstructured language, occasional speaker inter- ruptions, overlapping, and ...

work page
[2]

Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR

Proposed Methods In this section, we present the framework of the SLLM-based multilingual ASR system, along with our proposed methods. 2.1. Model Architectures The model employs a post-alignment design, projecting speech features into the same semantic embedding space as the pre- trained LLM. Its overall architecture is shown in Figure 1, con- sisting of ...

work page Pith review arXiv 2025
[3]

Experiments In this section, we detail the dataset we utilize and the technical specifications for both the training and inference phases. 3.1. Dataset Our training set comprises approximately 1500 hours of two- speaker conversational speech in eleven languages provided by NexData 1, namely MLC-SLM competition dataset, including English (American, British...

work page 2000
[4]

Experimental Results Table 3 summarizes the Word Error Rate (WER) and Character Error Rate (CER) achieved by our models across eleven lan- guages and five accents on the validation set. In detail, we cal- culate CER for Japanese, Korean, and Thai, while WER is used for the rest of the languages based on the characteristics of each Table 3:Word Error Rate ...

work page arXiv
[5]

Conclusion and Future Work In this work, we introduce a context-enhanced SLLM that combines language-specific prompts and bi-directional context, along with a two-stage decoding pipeline, achieving 13.56% MER on the validation set of the MLC-SLM conversational cor- pus. This outperforms the system trained with a larger-scale dataset, up to 6000 hours, dem...

work page
[6]

Conversational speech recognition by learning conversation-level characteristics,

K. Wei, Y . Zhang, S. Sun, L. Xie, and L. Ma, “Conversational speech recognition by learning conversation-level characteristics,” inICASSP 2022 - 2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), 2022, pp. 6752– 6756

work page 2022
[7]

Conversational speech recognition by learning audio-textual cross-modal contex- tual representation,

K. Wei, B. Li, H. Lv, Q. Lu, N. Jiang, and L. Xie, “Conversational speech recognition by learning audio-textual cross-modal contex- tual representation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2432–2444, 2024

work page 2024
[8]

The fisher corpus: a resource for the next generations of speech-to-text,

C. Cieri, D. Miller, and K. Walker, “The fisher corpus: a resource for the next generations of speech-to-text,” inProceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), M. T. Lino, M. F. Xavier, F. Ferreira, R. Costa, and R. Silva, Eds. Lisbon, Portugal: European Language Resources Association (ELRA), May 2004. ...

work page 2004

Show all 26 references

[9]

Switchboard: tele- phone speech corpus for research and development,

J. Godfrey, E. Holliman, and J. McDaniel, “Switchboard: tele- phone speech corpus for research and development,” in[Proceed- ings] ICASSP-92: 1992 IEEE International Conference on Acous- tics, Speech, and Signal Processing, vol. 1, 1992, pp. 517–520 vol.1

1992
[10]

Bert attends the conversation: Improving low-resource conversational asr,

P. Ortiz and S. Burud, “Bert attends the conversation: Improving low-resource conversational asr,”arXiv preprint arXiv:2110.02267, 2021

2021 arXiv
[11]

Robust speech recognition via large-scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

2023
[12]

Language models are unsupervised multitask learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

2019
[13]

Llama: Open and efficient foundation language models,

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[14]

Qwen technical report,

J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

2023 arXiv
[15]

WavLLM: Towards robust and adaptive speech large language model,

S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liu, and F. Wei, “WavLLM: Towards robust and adaptive speech large language model,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Nov. 2024, pp. 4552–4572

2024
[16]

Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,

Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,” 2023. [Online]. Available: https://arxiv.org/abs/2311.07919

2023 arXiv
[17]

Meralion-audiollm: Bridging audio and language with large language models,

MERaLiON Team, “Meralion-audiollm: Bridging audio and language with large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2412.09818

2024 arXiv
[18]

Qwen2-audio technical report,

Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-audio technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2407.10759

2024 arXiv
[19]

Step-audio: Unified understanding and generation in intelligent speech interaction,

A. Huang, B. Wu, B. Wang, C. Yan, C. Hu, C. Feng, F. Tian, F. Shen, J. Li, M. Chenet al., “Step-audio: Unified understanding and generation in intelligent speech interaction,”arXiv preprint arXiv:2502.11946, 2025

2025 arXiv
[20]

Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

2022
[21]

Promptasr for contextualized asr with controllable style,

X. Yang, W. Kang, Z. Yao, Y . Yang, L. Guo, F. Kuang, L. Lin, and D. Povey, “Promptasr for contextualized asr with controllable style,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 536–10 540

2024
[22]

A survey on in-context learning,

Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liuet al., “A survey on in-context learning,”arXiv preprint arXiv:2301.00234, 2022

2022 arXiv
[23]

Mozilla Common V oice 21.0,

Mozilla Foundation, “Mozilla Common V oice 21.0,” https://commonvoice.mozilla.org/en/datasets, 2025

2025
[24]

Gemma 2: Improving open language models at a practical size,

G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhu- patiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram ´eet al., “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024

2024 arXiv
[25]

Lora: Low-rank adaptation of large language models

E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.”ICLR, vol. 1, no. 2, p. 3, 2022

2022
[26]

An embarrassingly simple approach for llm with strong asr capacity,

Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhanget al., “An embarrassingly simple approach for llm with strong asr capacity,”arXiv preprint arXiv:2402.08846, 2024

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Introduction Conversational speech recognition (Conv-ASR), which aims to transcribe natural spoken language accurately, remains a signif- icant challenge in the speech processing area [1, 2]. Unlike iso- lated speech segments, conversational speech typically involves spontaneous, unstructured language, occasional speaker inter- ruptions, overlapping, and ...

work page

[2] [2]

Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR

Proposed Methods In this section, we present the framework of the SLLM-based multilingual ASR system, along with our proposed methods. 2.1. Model Architectures The model employs a post-alignment design, projecting speech features into the same semantic embedding space as the pre- trained LLM. Its overall architecture is shown in Figure 1, con- sisting of ...

work page Pith review arXiv 2025

[3] [3]

Experiments In this section, we detail the dataset we utilize and the technical specifications for both the training and inference phases. 3.1. Dataset Our training set comprises approximately 1500 hours of two- speaker conversational speech in eleven languages provided by NexData 1, namely MLC-SLM competition dataset, including English (American, British...

work page 2000

[4] [4]

Experimental Results Table 3 summarizes the Word Error Rate (WER) and Character Error Rate (CER) achieved by our models across eleven lan- guages and five accents on the validation set. In detail, we cal- culate CER for Japanese, Korean, and Thai, while WER is used for the rest of the languages based on the characteristics of each Table 3:Word Error Rate ...

work page arXiv

[5] [5]

Conclusion and Future Work In this work, we introduce a context-enhanced SLLM that combines language-specific prompts and bi-directional context, along with a two-stage decoding pipeline, achieving 13.56% MER on the validation set of the MLC-SLM conversational cor- pus. This outperforms the system trained with a larger-scale dataset, up to 6000 hours, dem...

work page

[6] [6]

Conversational speech recognition by learning conversation-level characteristics,

K. Wei, Y . Zhang, S. Sun, L. Xie, and L. Ma, “Conversational speech recognition by learning conversation-level characteristics,” inICASSP 2022 - 2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), 2022, pp. 6752– 6756

work page 2022

[7] [7]

Conversational speech recognition by learning audio-textual cross-modal contex- tual representation,

K. Wei, B. Li, H. Lv, Q. Lu, N. Jiang, and L. Xie, “Conversational speech recognition by learning audio-textual cross-modal contex- tual representation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2432–2444, 2024

work page 2024

[8] [8]

The fisher corpus: a resource for the next generations of speech-to-text,

C. Cieri, D. Miller, and K. Walker, “The fisher corpus: a resource for the next generations of speech-to-text,” inProceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), M. T. Lino, M. F. Xavier, F. Ferreira, R. Costa, and R. Silva, Eds. Lisbon, Portugal: European Language Resources Association (ELRA), May 2004. ...

work page 2004