REVIEW 9 cited by
Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Audio language models process audio inputs using textual prompts for tasks like speech recognition and audio captioning. Although built on multilingual pre-trained components, most are trained primarily on English, limiting their usability for other languages. This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations. To address this, we explore data mixtures that optimize audio language models for both a target language and English while integrating audio comprehension and speech instruction-following into a unified model. Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data. The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai.
Forward citations
Cited by 9 Pith papers
-
AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation
With prompt engineering (audio concatenation plus in-context examples), large audio models rank speech synthesis systems in line with human preferences, reaching up to 0.91 Spearman correlation.
-
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?
A new audio-visual benchmark shows current multimodal LLMs perform barely above random guessing, with audio perception errors as the dominant failure mode.
-
LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
LLaSO releases a 3.8B speech-language model, 25.5M training instances, and an evaluation benchmark, claiming a normalized score of 0.72.
-
Weakly Supervised Data Refinement and Flexible Sequence Compression for Efficient Thai LLM-based ASR
EThai-ASR combines a self-refined Zipformer encoder with a Thai LLM and reports SOTA CER on Thai test sets plus a cosine-similarity frame pruning that gives 1.5-2.1x speedups in some modes.
-
Enhancing Non-Core Language Instruction-Following in Speech LLMs via Semi-Implicit Cross-Lingual CoT Reasoning
XS-CoT trains speech LLMs to answer non-core language questions by generating an English reasoning chain before the final target-language answer, and a semi-implicit variant compresses that chain to cut latency.
-
Contrastive Learning for Task-Independent SpeechLLM-Pretraining
Contrastive pre-training that aligns speech and text across all model layers beats ASR-based pre-training and, with 10% of task data, matches or exceeds specialized models on translation and question answering.
-
Speechless: Speech Instruction Training Without Speech for Low Resource Languages
Fine-tuning an LLM on text instructions converted to Whisper semantic tokens enables it to understand spoken instructions at inference, bypassing TTS and speech instruction data.
-
Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models
Typhoon 2 improves Thai LLM performance through continual pre-training on curated Thai data and post-training, releasing text, vision, audio, and safety models.
-
Breaking the Barriers of Text-Hungry and Audio-Deficient AI
A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.
Discussion (0). Continue with ORCID to comment.