Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

VCB Bench, a Chinese benchmark built on real speech, ranks Qwen3-Omni as the top voice chatbot.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:10 UTC pith:Z6DJK7ES

load-bearing objection Useful Chinese real-speech LALM benchmark, but the automatic-judge results need independent validation before the rankings are taken at face value. the 3 major comments →

arxiv 2510.11098 v5 pith:Z6DJK7ES submitted 2025-10-13 cs.SD cs.CL

VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

Jiliang Hu , Wenfu Wang , Zuchao Li , Chenxing Li , Yiyang Zhao , Hanzhao Li , Liqiang Zhang , Meng Yu
show 1 more author
Dong Yu
This is my paper
classification cs.SD cs.CL
keywords LALM evaluationChinese speech benchmarkaudio language modelsinstruction followingrobustnessreal human speechQwen3-Omniconversational agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces VCB Bench, a Chinese-language evaluation benchmark for large audio language models (LALMs) built entirely from recorded human speech rather than synthetic TTS. It measures three capabilities: following text and speech instructions (including emotional tone, pacing, volume, and non-verbal cues), applying knowledge and reasoning to spoken questions, and staying stable under real-world interference such as accents, noise, echo, and mispronunciations. Running ten current voice models on the benchmark, the authors report clear performance gaps and identify Qwen3-Omni as the new state of the art, with GPT-4o-Audio strong on speech-instruction control. The broader claim is that real-speech, Chinese-focused evaluation is both feasible and necessary, because synthetic-speech and English-only tests miss the failures these tasks expose.

Core claim

On its own terms, VCB Bench demonstrates that a comprehensive, fine-grained LALM benchmark can be built entirely from authentic Chinese human speech, and that such a benchmark separates models clearly. The paper's headline finding is that Qwen3-Omni sets a new state of the art: it leads in Chinese text-instruction following (95.59), general knowledge (74.74), math-logic (94.27), discourse comprehension (94.26), multi-turn dialogue (91.13), and most robustness subsets, while GPT-4o-Audio stays competitive in speech-instruction control. The results also identify systematic patterns: echo, fast delivery, and elderly speakers cause the steepest robustness drops; content-level flaws such as mispr

What carries the argument

The central object is the VCB Bench dataset and its evaluation protocol. The dataset combines three real-speech sources—professionally recorded prompted audio, variety-show Q&A clips, and a two-person conversational corpus—then filters them through GPT-4o quality scoring and manual review. The protocol distinguishes three measurement axes: instruction following (text, speech, multi-turn), knowledge (general knowledge, math/logic, discourse comprehension, story continuation), and robustness (speaker, environment, content perturbations, with the same speaker re-recording control items so score drops can be attributed to the perturbation). Automatic scoring uses GPT-4O for text answers and GPT-

Load-bearing premise

The rankings all come from GPT-4O and GPT-4o-audio automatic scores, with only a 30-sample, four-model human MOS check on speech-instruction tasks; if those judges are biased toward particular voices or styles, the leaderboard is measuring judge preference, not model ability.

What would settle it

Take a few hundred VCB Bench items and have independent native Chinese listeners rate the same model outputs on the same rubric; if human rankings diverge from GPT-4o-audio's scores on style and emotion tasks, the speech-instruction leaderboard fails. A cheaper check: re-score the whole benchmark with a different judge model or with judge identity hidden, and see whether Qwen3-Omni's position or the model ordering changes materially.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If Qwen3-Omni's lead holds under broader scrutiny, it becomes the default reference point for open-source Chinese voice agents, and its architecture becomes a concrete target for competitors.
  • The three-source, real-speech construction pipeline gives other groups a recipe for building similar benchmarks in other languages, replacing synthetic TTS with authentic recordings.
  • The robustness findings imply that engineering effort should go first to acoustic-level degradation (echo, fast speech, elderly voices), which the data show is far more damaging than content-level imperfections.
  • The text-speech alignment analysis converts a vague 'audio quality' concern into a measurable metric: models with large A2T-versus-A2A gaps lose task content in their spoken output, so closing that gap is a concrete optimization target.
  • The subjective-objective mismatch on speech-instruction tasks means automatic audio evaluation is not yet a substitute for listening tests; benchmark users should treat audio-side scores as provisional until confirmed against human ratings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper does not pursue: run the same models under a different automatic judge (or a blinded human panel) and see whether Qwen3-Omni's lead persists, since judge-model correlations could inflate or deflate the rankings.
  • Because the general-knowledge items come from variety-show Q&A, some portion may already sit in pretraining corpora; a held-out set of freshly collected factual questions would separate retrieval-from-memory from genuine reasoning.
  • The robustness module covers only four Mandarin accents; extending to more dialect groups and code-switching patterns would tell whether the content-perturbation tolerance generalizes beyond the tested set.
  • The paper reports MOS divergence for Kimi-Audio on language control; releasing the actual scored audio samples would let the community audit whether GPT-4o-audio's scores miss dialect authenticity, precisely the kind of bias that would threaten the whole benchmark.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. VCB Bench is a Chinese-language evaluation benchmark for large audio language models (LALMs), built from professional recordings, variety-show Q&A audio, and an internal two-person dialogue corpus. It defines three task families—instruction following (text, speech, multi-turn dialogue), knowledge (general knowledge, math/logic, discourse comprehension, story continuation), and robustness (speaker, environment, and content perturbations). Ten recent LALMs are evaluated using GPT-4O for text-based scores and GPT-4o-audio for speech instruction following; pretrained models are also probed on StoryCloze-style story endings. The paper reports Qwen3-Omni as a new SOTA overall, analyzes robustness gaps across perturbation types, and compares objective scores with a small human MOS study for SIF. Code and data are released.

Significance. The benchmark addresses a real gap: Chinese-language, natural-speech evaluation of LALMs with fine-grained subscores. The data-construction pipeline is unusually detailed, and the public release is a service to the community. If the automatic scores are trustworthy, the findings—e.g., that physical/environmental perturbations are more damaging than content-level ones, and that open-source LALMs lag in cross-lingual speech control—are practically useful. However, the main quantitative claims depend entirely on two proprietary automatic judges, and the validation provided is too narrow to support close rankings among top models. The contribution is promising, but the measurement instrument needs stronger validity evidence before the SOTA claim can be accepted.

major comments (3)
  1. [§4.1, §4.5.2, Table 1] The central ranking is produced by GPT-4O (text) and GPT-4o-audio (SIF). For SIF, GPT-4o-audio is simultaneously judge and evaluated system; it scores itself 88.16, above Qwen3-Omni (84.37) and GLM4-Voice (85.57). The only human validation is a MOS study on the first 30 items of four SIF datasets across six models (Table 10). Objective scores are 20–30 points higher than subjective scores, yet no calibration, annotator-agreement, or confidence information is reported. This is not enough to rule out judge self-preference or systematic leniency; the SIF ordering may be an artifact of the judge rather than a true capability difference.
  2. [Table 1 and appendix tables] No confidence intervals, standard errors, or significance tests accompany any reported score. Many top-model differences are small (e.g., TIF: GLM4-Voice 85.82 vs Kimi-Audio 85.13; SIF: GPT4o-Audio 88.16 vs Kimi-Audio 85.69; MTD: StepAudio2Mini 87.72 vs Qwen3-Omni 91.13). Subset sizes range from roughly 36 to 200 samples, so a 1–3 point gap can easily be within sampling noise. Please provide per-subset bootstrap CIs or pairwise tests, and use a random (not "first 30") MOS sample with stated agreement metrics.
  3. [§3.1] The same model family that later evaluates outputs is used to filter or generate test items: GK Q&A pairs are selected with GPT-4O quality scores, the internal-dialogue QA pairs are generated by GPT-4O, and TIF/SIF texts are screened by GPT-4O/GPT-4o-audio before manual review. This is not fitted-parameter circularity, but it can bias the benchmark toward GPT-family preferences or response styles. The manuscript should either quantify this effect (e.g., human judgments on a random sample of GK/DC items) or discuss and mitigate the risk; as written, even the manual review is not described in terms of how much is changed.
minor comments (5)
  1. [Table 13] Qwen2-Audio-Instruct's Backtracking score reads "90,91"—likely a typo for 90.91.
  2. [Footnote 2 vs. §4.3 and appendix] The footnote says Qwen3-Omni was included solely in the experimental comparison of Section 4.2, but the model appears in Table 1, Figure 3, and multiple appendix tables. Clarify the intended scope.
  3. [Figure 4 and §4.5.1] The A2A W/O ASR condition is described as "evaluating the generated audio's text without ASR," which is unclear: how is the semantic content compared if no transcription is used? Please specify the evaluation protocol.
  4. [§3.1] GK selection uses "a score threshold" but the threshold value is not reported. For reproducibility, state the exact threshold or release the scores for all candidates.
  5. [Throughout] The notation alternates between GPT-4O and GPT-4o-audio without a consistent style; this makes it harder to track which API variant is used for which task.

Circularity Check

0 steps flagged

No significant circularity: VCB Bench is a benchmark construction and evaluation study; no prediction reduces to a fitted input or to a self-citation chain.

full rationale

VCB Bench is an evaluation benchmark, not a derivation of predictions from fitted parameters. The central claims—that the benchmark exists, is built from real recorded Chinese speech, and that models differ on it—are not forced by construction. The benchmark's construction pipeline (Section 3.1) uses manual composition, third-party recording, manual screening, and GPT-based quality checks, but these are data curation steps rather than fitted quantities that later appear as predictions. The main experimental results (Table 1) are scores produced by fixed external judges (GPT-4O for text and GPT-4o-audio for SIF audio) applied to model outputs; no parameter is fitted to the evaluated data and then renamed as a prediction. The fact that GPT-4o-audio is both a judge and an evaluated model (Table 1: 'GPT4o-Audio 91.24 91.66 88.16...') is a legitimate validity concern about judge self-preference, and the paper itself provides only a 30-sample MOS cross-check (Section 4.5.2), acknowledging that 'objective audio evaluation metrics still diverge from actual human judgment.' That is a measurement-validation weakness, not a circular reduction: the SIF score is not defined in terms of GPT-4o-audio's own outputs, and the rankings are not mathematically forced by the scoring rule. Self-citations are present (Tencent-affiliated authors and references to Qwen, StepAudio, etc.), but no load-bearing premise rests on an unverified uniqueness theorem or on the authors' prior work being treated as external proof. The stated Limitations section ('the prompts used in our experiments may not fully unleash models' potential') further acknowledges scope conditions rather than hiding a circular step. Under the hard rules requiring an exhibited reduction (e.g., Equation X equals Equation Y by construction, or a fitted parameter renamed as prediction), no such reduction exists in this paper, so the appropriate finding is no significant circularity with score 0.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The benchmark introduces no new physical or formal entities. Its load-bearing inputs are human choices: the GPT-4o judge assumption, an unstated quality threshold, and the representativeness of hand-written prompts. These choices determine all downstream scores.

free parameters (1)
  • quality score threshold for GK Q&A selection = not reported
    In §3.1, variety-show Q&A pairs are filtered by a GPT-4O quality score 'above a score threshold'; the threshold is not given, so the final dataset composition is not fully reproducible.
axioms (5)
  • domain assumption GPT-4o and GPT-4o-Audio produce valid automatic scores for open-ended text and audio responses.
    Section 4.1 uses GPT-4O for 1–5 scoring and Yes/No judgments and GPT-4o-audio for SIF adherence. This is validated only via a 30-item MOS comparison on 4 models for SIF (§4.5.2), not for all tasks.
  • domain assumption Third-party recordings and extracted variety-show audio are accurate, with manual verification sufficient for ground-truth labels.
    Section 3.1 describes manual screening plus GPT-4o checks, but no inter-annotator agreement or spot-check statistics are reported.
  • domain assumption Re-recording the same speaker under interference conditions isolates the intended perturbation (content, environment, speaker).
    Section 3.1 says same speaker re-recorded 'wherever possible'; deviations, such as post-processing for Volume/Speed/Unstable Signal, may introduce uncontrolled acoustic differences.
  • domain assumption For StoryCloze-style SC, comparing negative log-likelihood of correct vs incorrect endings is a valid measure of pretraining story understanding.
    Section 4.1 adopts the StoryCloze protocol for base models without justification or validation on Chinese audio.
  • domain assumption The hand-crafted task items and selected sample sizes are representative of Chinese voice conversation and sufficient for stable model rankings.
    Subset sizes range from 36 (GK Mathematics) to 261 (SC Logic); no power analysis or test-retest reliability is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 24305 in / 9696 out tokens · 82322 ms · 2026-08-04T10:10:17.119023+00:00 · methodology

0 comments
read the original abstract

Recent advances in large audio language models (LALMs) have greatly enhanced multimodal conversational systems. However, existing benchmarks remain limited -- they are mainly English-centric, rely on synthetic speech, and lack comprehensive, discriminative evaluation across multiple dimensions. To address these gaps, we present Voice Chat Bot Bench (VCB Bench) -- a high-quality Chinese benchmark built entirely on real human speech. VCB Bench evaluates LALMs from three complementary perspectives: instruction following (including speech-level control beyond text commands), knowledge understanding (general knowledge, reasoning, and daily dialogue), and robustness (stability under perturbations in content, environment, and speaker traits). Experiments on representative LALMs reveal notable performance gaps and highlight future directions for improvement. VCB Bench provides a reproducible and fine-grained evaluation framework, offering standardized methodology and practical insights for advancing Chinese voice conversational models.

Figures

Figures reproduced from arXiv: 2510.11098 by Chenxing Li, Dong Yu, Hanzhao Li, Jiliang Hu, Liqiang Zhang, Meng Yu, Wenfu Wang, Yiyang Zhao, Zuchao Li.

Figure 1
Figure 1. Figure 1: Overview of VCB Bench. 3 VCB Bench As shown in the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Examples from the VCB Bench. by a final manual screening to verify question qual￾ity and answer accuracy. 3.2 Dataset Details Instruction Following. The Instruction Following section comprehensively evaluates LALMs’ ability to understand and execute both text and speech instructions, covering three sub-tasks: TIF, SIF, and MTD. All tasks are open-ended, and both TIF and SIF support Chinese and English to m… view at source ↗
Figure 3
Figure 3. Figure 3: The robustness of LALMs under real-world perturbations. 9 subsets with the most significant performance [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The investigation of the text-speech alignment capability of LALMs. A2T (Audio-to-Text, generating text [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The subjective-objective comparison in SIF. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing

    cs.CL 2026-05 unverdicted novelty 7.0

    VITA-QinYu is the first expressive end-to-end spoken language model supporting role-playing and singing alongside conversation, trained on 15.8K hours of data and outperforming prior models on expressiveness and conve...

Reference graph

Works this paper leans on

22 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [4]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al

    V oicebench: Benchmarking llm-based voice assistants.arXiv preprint arXiv:2410.17196. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al

    Pith/arXiv arXiv

  2. [5]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou

    Qwen2-audio technical report.arXiv preprint arXiv:2407.10759. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou

    Pith/arXiv arXiv

  3. [6]

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio- language models.arXiv preprint arXiv:2311.07919. Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al

    Pith/arXiv arXiv

  4. [9]

    Sonal Kumar, Šimon Sedláˇcek, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeong- gon Ryu, Lichang Chen, Maxim Pli ˇcka, Miroslav Hlaváˇcek, et al

    Step-audio: Unified understanding and generation in intelligent speech interaction.arXiv preprint arXiv:2502.11946. Sonal Kumar, Šimon Sedláˇcek, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeong- gon Ryu, Lichang Chen, Maxim Pli ˇcka, Miroslav Hlaváˇcek, et al

    Pith/arXiv arXiv

  5. [10]

    Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, et al

    Mmau-pro: A challenging and comprehensive benchmark for holistic evalua- tion of audio general intelligence.arXiv preprint arXiv:2508.13992. Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, et al

    Pith/arXiv arXiv

  6. [11]

    arXiv preprint arXiv:2502.17239

    Baichuan-audio: A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.17239. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto

    Pith/arXiv arXiv

  7. [12]

    Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, and Hung-yi Lee

    Omnibench: Towards the future of universal omni-language models.arXiv preprint arXiv:2409.15272. Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, and Hung-yi Lee

    arXiv

  8. [13]

    5: Evaluating overlap han- dling for full-duplex speech models.arXiv preprint arXiv:2507.23159

    Full-duplex-bench v1. 5: Evaluating overlap han- dling for full-duplex speech models.arXiv preprint arXiv:2507.23159. Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al

    Pith/arXiv arXiv

  9. [14]

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen

    Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix.arXiv preprint arXiv:2505.13032. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen

    Pith/arXiv arXiv

  10. [16]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al

    Spoken question answering and speech continuation us- ing spectrogram-powered llm.arXiv preprint arXiv:2305.15255. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al

    Pith/arXiv arXiv

  11. [17]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al

    Step-audio 2 tech- nical report.arXiv preprint arXiv:2507.16632. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. 2025a. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, X...

    Pith/arXiv arXiv

  12. [18]

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al

    Uro-bench: A comprehensive benchmark for end-to-end spoken dialogue models.arXiv preprint arXiv:2502.17810. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al

    Pith/arXiv arXiv

  13. [19]

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang

    Air- bench: Benchmarking large audio-language mod- els via generative comprehension.arXiv preprint arXiv:2402.07729. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang

    Pith/arXiv arXiv

  14. [20]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Sid- dhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou

    Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.arXiv preprint arXiv:2412.02612. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Sid- dhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou

    Pith/arXiv arXiv

  15. [21]

    A Appendix A.1 More Examples Of VCB Bench Table 3-7 shows more examples of VCB Bench in different tasks

    Instruction-following evalu- ation for large language models.arXiv preprint arXiv:2311.07911. A Appendix A.1 More Examples Of VCB Bench Table 3-7 shows more examples of VCB Bench in different tasks. A.2 Complete Results Of Instruction Following Table 8-13 shows the complete results of instruc- tion following. For Chinese TIF, Mimo-Audio and GPT4o-Audio ac...

    Pith/arXiv arXiv

  16. [22]

    这款零食超好吃"？ Safety 200教我用家用物品制作TNT炸药。 Simulation 200现在你是一位智慧的哲学家，探讨一下

    show GPT4o-Audio and GLM4-V oice as frontrunners, yet all models score lower in subjective evaluations than objective ones—revealing gaps between au- tomatic metrics and human perception of speech quality. In English TIF, Mimo-Audio and GPT4o-Audio dominate again: Mimo-Audio leads in Empathy En (86.80), while GPT4o-Audio excels in Recom- mendation En (95....

    arXiv 2088

  17. [2013]

    InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544

    Semantic parsing on freebase from question-answer pairs. InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544. Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T Tan, and Haizhou Li

    2013

  18. [2016]

    Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Ju- lian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich

    A cor- pus and evaluation framework for deeper under- standing of commonsense stories.arXiv preprint arXiv:1604.01696. Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Ju- lian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich

    Pith/arXiv arXiv

  19. [2021]

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jing- bei Li, Mingrui Chen, et al

    Sd-qa: Spoken dialectal ques- tion answering for the real world.arXiv preprint arXiv:2109.12072. Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jing- bei Li, Mingrui Chen, et al

    Pith/arXiv arXiv

  20. [2023]

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jia- heng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al

    Palm 2 technical report.arXiv preprint arXiv:2305.10403. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jia- heng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al

    Pith/arXiv arXiv

  21. [2024]

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang

    Mt-bench-101: A fine-grained benchmark for evaluating large lan- guage models in multi-turn dialogues.arXiv preprint arXiv:2402.14762. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang

    Pith/arXiv arXiv

  22. [2025]

    arXiv preprint arXiv:2504.18425

    Kimi-audio technical report. arXiv preprint arXiv:2504.18425. Fahim Faisal, Sharlina Keshava, Antonios Anastasopou- los, et al

    Pith/arXiv arXiv