Pith. sign in

REVIEW 4 major objections 4 minor 36 references

SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SpokenNativQA benchmarks LLMs on 33,000 everyday spoken questions and shows that cascaded speech-to-text pipelines lose accuracy, while an audio-native model matches or beats gold-text answers.

desk verdict A real, released Arabic/English spoken QA dataset with solid baselines, but the 'naturally spoken conversational' framing overreaches; the recordings are read speech. read the letter →

arxiv 2505.19163 v1 pith:GCGPSGLY submitted 2025-05-25 cs.CL cs.AI

Firoj Alam , Md Arid Hasan , Shammur Absar Chowdhury This is my paper
classification cs.CLcs.AI
keywords spokenquestionansweringmultilingualbenchmarkASRerrorpropagationeverydayqueriesArabicEnglishLLMevaluationBERTScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpokenNativQA is a new benchmark of roughly 33,000 spoken question-answer pairs in Arabic and English, built so that large language models (LLMs) can be tested on everyday, culture-specific queries as they would be asked aloud. The paper's central claim is that the usual cascade — automatic speech recognition (ASR) producing a transcript, then a text-only LLM answering — loses accuracy on such queries: giving the LLM the gold text question always scores higher than giving it any ASR transcript. It also reports that the audio-native GPT-4o Audio model scores 0.55 F1 on Arabic and 0.62 on English, surpassing the gold-text setup for Arabic and matching it for English. If correct, the paper contributes a reusable multilingual benchmark and evidence that cascade-less, audio-native systems deserve the development effort.

What carries the argument

The central object is the dataset itself: human-recorded audio of real-user questions paired with human-curated answers, spanning 18 everyday and culturally aligned topics in Arabic and English. The experimental machinery is the controlled comparison of three input conditions — gold text questions, ASR transcriptions from Google, Azure, Whisper, and Fanar, and direct audio input to GPT-4o Audio — all scored with BERTScore F1. That three-way comparison isolates the effect of speech-to-text error propagation on downstream LLM answer quality.

What would settle it

Record a new set of unscripted, spontaneous conversational queries with disfluencies, hesitations, and background noise from the same speakers, run the same four ASR systems and the same text LLMs, and compare BERTScore F1 across setups; if cascaded ASR pipelines match or beat the gold-text and audio-native setups, or if GPT-4o Audio no longer exceeds the No-ASR setup, the paper's cascade-less conclusion would not hold for natural speech.

Watch

Extended reading notes

Core claim

The paper introduces SpokenNativQA and uses it to benchmark spoken question answering in Arabic and English. The questions come from the MultiNativQA test sets, in which real users asked culture- and region-specific everyday questions and answers were manually curated; the paper recorded those questions read aloud by 12 Arabic L1 speakers and 11 English L2 speakers, yielding about 30 hours of audio across 33,081 samples. In the benchmark, the No-ASR setup (gold text questions) consistently achieves the highest BERTScore F1 across GPT-4o, Fanar, and ALLaM in both languages, averaging 0.536 for Arabic and 0.619 for English, while every ASR transcription lowers the score. Among the four ASR systems, Whisper gives the best overall SQA performance, and the audio-native GPT-4o Audio reaches 0.55 for Arabic and 0.62 for English, beating the No-ASR setup for Arabic and matching it for English. The authors conclude that ASR errors propagate through cascaded systems and argue for cascade-less models, especially for everyday queries with real-world stakes such as health.

Load-bearing premise

The load-bearing premise is that audio of speakers reading displayed questions is a faithful proxy for naturally spoken, real-world conversational queries, even though it is read speech rather than spontaneous speech.

Editorial extensions

If this is right

  • Most existing spoken QA benchmarks are English-only and synthetic; SpokenNativQA offers a human-recorded, multilingual alternative built on real-user questions.
  • ASR error propagation measurably reduces LLM answer quality on everyday queries, and the paper quantifies the drop for four ASR systems.
  • Region-specific ASR helps Arabic while the general Whisper model helps English, so the choice of ASR for a spoken QA pipeline should be language-dependent.
  • Audio-native models such as GPT-4o Audio can match or exceed gold-text performance, supporting the move toward cascade-less spoken LLMs.
  • The released data and scripts let other groups reproduce the benchmark and extend it to further regions and languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the recordings are read speech rather than spontaneous conversation, the measured advantage of audio-native models probably understates its real size on queries full of disfluencies, hesitations, and prosodic cues.
  • A stress test with noisy environments, accented spontaneous speech, and unscripted questions would likely widen the gap between cascaded and audio-native systems.
  • A natural follow-up is to score high-stakes topics such as health separately, since the paper motivates cascade-less systems with exactly those queries.
  • BERTScore is the only evaluation metric used; checking the same rankings with human ratings or additional semantic metrics would show whether the observed gaps are metric-specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces SpokenNativQA, a spoken question-answering dataset built by extracting the Arabic and English (Qatar) test sets from the text-based MultiNativQA dataset and having 12 Arabic and 11 English speakers read the displayed questions aloud. The full recording set contains roughly 33k audio samples (~30 hours), and a test subset of 3,310 samples (one recording per question) is used for evaluation. The authors benchmark four ASR systems (Google, Azure, Whisper large-v3, Fanar), three text LLMs (GPT-4o, Fanar, ALLaM) in a cascaded pipeline, and one audio-input model (GPT-4o Audio), reporting BERTScore F1 in No-ASR and ASR-cascaded settings. The main empirical claims are that the No-ASR setup with gold text questions consistently scores highest, that Whisper is the best ASR for English while Google is best for Arabic, and that GPT-4o Audio surpasses the No-ASR setup for Arabic and matches it for English.

Significance. If the dataset is understood as a benchmark for human-read prompts, it is a useful and reusable resource: it is publicly released on Hugging Face, the experimental scripts are provided through LLMeBench, the questions originate from real users, and the answers are human-curated. The paper also provides the first WER and SQA baseline numbers for this specific Arabic/English everyday-QA material, which will be valuable for future cascaded and end-to-end SQA research. However, the advertised central contribution—evaluation of LLMs in 'real-world conversational settings' with 'naturally spoken' queries—is not established by the collection protocol, and several evaluation claims are not statistically supported. The resource can likely be reframed and strengthened, but the current write-up overstates what the data and experiments demonstrate.

major comments (4)
  1. [Section 2.1 and abstract] The abstract and introduction claim that SpokenNativQA evaluates LLMs in 'real-world conversational settings' and that it incorporates 'speech variability, accents, and linguistic diversity' as well as phenomena such as disfluencies, hesitations, and prosody (Section 1). The collection protocol in Section 2.1, however, asks speakers to 'record audio by reading the question' shown one at a time on a recording platform. This elicits read speech, not spontaneous or conversational speech, and it does not capture the disfluencies, hesitations, restarts, or question prosody named as motivation. The sentence 'We did not impose any restrictions on how to read the question or on the recording environment' addresses recording conditions, not speech style. The paper should either soften the claims to 'human-read prompts' or provide a disfluency/prosody analysis demonstrating that the recordings contain conversational characteristics; as written, the headline claim is not supported by the data-collection protocol.
  2. [Section 4, Table 4] The main findings are based on BERTScore F1 differences that are extremely small, yet no confidence intervals, significance tests, or repeated runs are reported. For example, the Arabic No-ASR average is 0.536 versus 0.531 for Google, and GPT-4o Audio scores 0.55 versus 0.536 for No-ASR; for English, GPT-4o Audio scores 0.62 versus 0.619 for No-ASR. Statements such as 'consistently achieves the highest scores' and 'outperforms' are not justified without uncertainty quantification, especially because the differences are on the order of 0.005–0.014. I recommend per-question bootstrap confidence intervals or paired significance tests for every comparison that supports a conclusion.
  3. [Section 4, 'Overall' paragraph] The claim that 'SQA performance is significantly better for English than for Arabic' is not supported by the reported metric. Arabic BERTScore is computed with AraBERT (v2) while English BERTScore is computed with bert-base-uncased (Section 3.2); these are different embedding spaces, so the F1 numbers are not directly comparable across languages. This cross-language comparison must either use a language-comparable normalization or be removed.
  4. [Section 3.2, Listing 1] Two aspects of the evaluation protocol threaten the validity of the benchmark results. First, the prompt was 'redesigned after several iterations of manual checking to achieve optimal performance' on what appears to be the same test set, with no development set or statement that the prompt was frozen before seeing test results; this creates an overfitting risk. Second, the 'length' variable in the prompt is set from the gold answer length in the dataset, giving the model oracle information about the reference answer length. While the effect on BERTScore may be modest, it is still a form of test-set leakage that does not reflect a real deployment setting. The authors should document how the prompt was selected, whether it was tuned on a development split, and justify or remove the gold-length conditioning.
minor comments (4)
  1. [Table 1] The English row in Table 1 appears to have twelve column entries even though the text says 11 English speakers, and the displayed formatting makes it difficult to tell which speaker (S10?) is missing. Please clarify the speaker numbering and ensure the row sums are legible.
  2. [Section 3.1 and Table 3] The ASR descriptions use undefined abbreviations: 'AR-QA' for Google and Azure is not explained, and the abstract's 'Google location-based ASR' does not match the model names in the table. Please define the exact ASR model variants and versions.
  3. [Abstract and Section 2.2] The abstract says the dataset 'comprises approximately 33k naturally spoken questions and answers,' but the benchmark evaluation uses a test subset of only 3,310 samples (one recording per question). The full 33k count includes roughly ten recordings per question. This distinction should be explicit in the abstract and contributions to avoid overstating the evaluation scale.
  4. [Section 4, findings (i)] The statement that 'ASR performance is affected for English, possibly because all speakers are L2 speakers' is presented as a finding, but no L1/L2 controlled comparison is made. Please label this as a hypothesis and note that English and Arabic speakers differ in many ways besides L1 status.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SpokenNativQA is an empirical dataset and benchmark, not a derivation that reduces to its inputs.

full rationale

The paper's contribution is the construction of a spoken QA dataset and the empirical benchmarking of ASR and LLM systems; there is no formal derivation whose conclusion is encoded in its premises. The text QA pairs are taken from the authors' prior MultiNativQA work, but the audio recordings are new, and the benchmark uses released ASR and LLM APIs with zero-shot prompting and BERTScore, with no parameter fitted to the reported results. The self-citations ([19], [20], [24]) are data/model/tool provenance; the source dataset is released and externally inspectable, so the citations are real evidence rather than a circular justification. The one methodological caveat—the prompt is given the reference answer length ('The value of the length variable is determined by based on the length of the answer in the dataset')—is a test-set-informed prompt choice that could affect absolute scores, but it is applied uniformly across all systems and therefore does not force the paper's relative comparisons. The substantive weakness is external validity, not circularity: Section 2.1 describes a reading task ('Speakers were asked to record audio by reading the question'), so the 'real-world conversational settings' claim is not established by the collection protocol; that is a validity/correctness concern, not a circular derivation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No numeric constants are fitted to data beyond the oracle answer length used in the prompt, and the paper introduces no new physical or conceptual entities. The main unstated premises are about data collection fidelity, metric comparability, and test set tuning.

free parameters (1)
  • Answer length limit in prompt = Gold reference answer word count per test question
    Listing 1 instructs models to limit answers to {length} words, where length is taken from the gold answer in the dataset, giving models oracle length information at inference time.
assumptions (4)
  • domain assumption Read-aloud recordings by recruited speakers approximate naturally spoken everyday queries
    Section 2.1 describes speakers reading displayed questions and submitting audio, with no conversational context or spontaneous formulation, yet the abstract claims real-world conversational settings.
  • domain assumption BERTScore F1 computed with different language-specific encoders is comparable across Arabic and English
    Section 3.2 uses AraBERT for Arabic and bert-base-uncased for English; the paper then directly compares Arabic and English scores, assuming the two embedding spaces are on the same scale.
  • domain assumption The MultiNativQA test set is a valid, high-quality source of gold questions and answers
    Section 2 adopts the Arabic and English test sets from MultiNativQA and inherits its manual annotations; the benchmark results depend on the quality of that inherited data.
  • ad hoc to paper The manually iterated prompt does not overfit the test set
    Section 3.2 describes redesigning the prompt after several iterations of manual checking to achieve optimal performance, with no held-out development set mentioned; the reported results assume this tuning does not inflate scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs." pith.science (2026). https://pith.science/paper/GCGPSGLY

@misc{pith2026250519163,
  author       = {Pith},
  title        = {Pith review of: SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GCGPSGLY}},
  note         = {Machine review of arXiv:2505.19163}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable performance across various disciplines and tasks. However, benchmarking their capabilities with multilingual spoken queries remains largely unexplored. In this study, we introduce SpokenNativQA, the first multilingual and culturally aligned spoken question-answering (SQA) dataset designed to evaluate LLMs in real-world conversational settings. The dataset comprises approximately 33,000 naturally spoken questions and answers in multiple languages, including low-resource and dialect-rich languages, providing a robust benchmark for assessing LLM performance in speech-based interactions. SpokenNativQA addresses the limitations of text-based QA datasets by incorporating speech variability, accents, and linguistic diversity. We benchmark different ASR systems and LLMs for SQA and present our findings. We released the data at (https://huggingface.co/datasets/QCRI/SpokenNativQA) and the experimental scripts at (https://llmebench.qcri.org/) for the research community.

Figures

Figures reproduced from arXiv: 2505.19163 by the authors.

Figure 1
Figure 1. Complete overview of the SpokenNativQA dataset development pipeline and benchmarking experiments. ations to more realistic, noisy, and conversational spoken envi￾ronments. These datasets have facilitated benchmarking LLMs for SQA. However, current efforts in benchmarking LLMs and Multimodal LLMs (MLLMs) for SQA remain relatively lim￾ited [16] compared to text-based benchmarks [17, 18]. More￾over, the majority of exi… view at source ↗
Figure 2
Figure 2. Topic wise distribution for Arabic [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Topic wise distribution for English (Qatar). 2.2. Test Subset For this study, we selected the test subset from the full set. We randomly selected one speaker’s recording for each question. Moreover, we ensured that recordings from all speakers were included in the test subset. Detailed statistics on recording du￾ration for both the full set and the test subset are provided in [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: F1 across different setups and models for Arabic [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: F1 across different setups and models for English. both question and audio inputs. The F1 scores are 0.55 for Ara￾bic and 0.62 for English. It outperforms all other models in both languages, surpassing the No-ASR setup for Arabic and matching its performance for Englis…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 25 canonical work pages

  1. [1]

    Introduction The emergent capabilities of LLMs have significantly increased their adoption among end users, enabling seamless integration into everyday tasks. These models are now widely utilized across diverse domains, from education to professional activ- ities, where their ability to generate human-like responses, pro- vide contextual understanding, an...

    work page Pith review arXiv 2025

  2. [2]

    The dataset consists of questions covering 18 different topics that are culture- and region-specific, as well as everyday questions

    Dataset We adopted the MultiNativQA [19] dataset, which includes QA pairs where queries come from real users and answers were manually edited. The dataset consists of questions covering 18 different topics that are culture- and region-specific, as well as everyday questions. The topics in the dataset include Animals, Business, Clothing, Education, Events,...

    work page

  3. [3]

    answer":

    Methodology As presented in Figure 1, our methodological steps consist of SpokenNativQA dataset development, discussed in the previous section, and benchmarking. The latter part involves employing several open and closed ASR systems to obtain transcriptions for both languages. To understand the impact of ASR error propagation, we benchmark three LLMs and ...

    work page

  4. [4]

    What is Doha known for food?

    Results and Findings In Table 4, we present the performance of different models and compare No-ASR and ASR-cascaded setups. Based on the re- sults, the No-ASR setup, which uses gold questions, consis- tently achieves the highest scores across all models and both languages. The average performance in this setup is 0.536 for Arabic and 0.619 for English. No...

    work page

  5. [5]

    For SQA, it is particularly important to assess how current LLMs and MLLMs perform under noisy, real-world conditions

    Related Work Benchmarks play a crucial role in evaluating AI systems. For SQA, it is particularly important to assess how current LLMs and MLLMs perform under noisy, real-world conditions. A fundamental challenge in this evaluation is the availability of benchmarking datasets. Early efforts, such as Spoken SQuAD [11], introduced ASR-generated spoken quest...

    work page

  6. [6]

    The questions are recorded by L1 speakers for Arabic and L2 speakers for English languages

    Conclusion and Future Work In this study, we introduce a multilingual SQA dataset for ev- eryday queries. The questions are recorded by L1 speakers for Arabic and L2 speakers for English languages. This is due to the nature of the task itself, which targets culture and region- specific QA. We collected ∼ 33k samples for both Arabic and English, where ques...

    work page

  7. [7]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” OpenAI, Tech. Rep., 2023

    2023

  8. [8]

    Sparks of artificial general intelligence: Early experiments with GPT-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y . Zhang, “Sparks of artificial general intelligence: Early experiments with GPT-4,” Microsoft Research, Tech. Rep., 2023

    work page 2023

Show all 36 references
  1. [9]

    LLaMA: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023

    2023 arXiv

  2. [10]

    A frame- work for automatic generation of spoken question-answering data,

    M. ¨U. Menevs ¸e, Y . Manav, E. Arisoy, and A.¨Ozg¨ur, “A frame- work for automatic generation of spoken question-answering data,” in Findings of the ACL: EMNLP 2022 , 2022, pp. 4659– 4666

    2022

  3. [11]

    LibriSQA: A novel dataset and framework for spoken question answering with large language models,

    Z. Zhao, Y . Jiang, H. Liu, Y . Wang, and Y . Wang, “LibriSQA: A novel dataset and framework for spoken question answering with large language models,” IEEE Transactions on Artificial Intelli- gence, 2024

    2024

  4. [12]

    AudioGPT: Understanding and generating speech, music, sound, and talking head,

    R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y . Wu, Z. Hong, J. Huang, J. Liu et al., “AudioGPT: Understanding and generating speech, music, sound, and talking head,” inProc. of the AAAI Conference on AI, vol. 38, no. 21, 2024, pp. 23 802–23 804

    2024

  5. [13]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y . Zhou, and X. Qiu, “SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,” in Findings of the ACL: EMNLP 2023. Singapore: ACL, Dec. 2023, pp. 15 757– 15 773

    2023

  6. [14]

    Qwen-audio: Advancing universal au- dio understanding via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal au- dio understanding via unified large-scale audio-language models,” arXiv:2311.07919, 2023

    2023 arXiv

  7. [15]

    WavLLM: Towards robust and adap- tive speech large language model,

    S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaranet al., “WavLLM: Towards robust and adap- tive speech large language model,”arXiv:2404.00656, 2024

    2024 arXiv

  8. [16]

    GhostT5: generate more features with cheap operations to improve text- less spoken question answering,

    X. Cheng, Z. Zhu, Z. Yao, H. Li, Y . Li, and Y . Zou, “GhostT5: generate more features with cheap operations to improve text- less spoken question answering,” in Proc. Interspeech, 2023, pp. 1134–1138

    2023

  9. [17]

    Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension,

    C.-H. Lee, S.-L. Wu, C.-L. Liu, and H.-y. Lee, “Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension,” Proc. Interspeech 2018, pp. 3459– 3463, 2018

    2018

  10. [18]

    Zero- shot end-to-end spoken question answering in medical domain,

    Y . Labrak, A. Moumen, R. Dufour, and M. Rouvier, “Zero- shot end-to-end spoken question answering in medical domain,” arXiv:2406.05876, 2024

    2024 arXiv

  11. [19]

    HeySQuAD: A spoken question answering dataset,

    Y . Wu, S. Rallabandi, R. Srinivasamurthy, P. P. Dakle, A. Gon, and P. Raghavan, “HeySQuAD: A spoken question answering dataset,” arXiv preprint arXiv:2304.13689, 2023

    2023 arXiv

  12. [20]

    CORAAL QA: A dataset and framework for open domain spontaneous speech question answering from long audio files,

    N. B. Shankar, A. Johnson, C. Chance, H. Veeramani, and A. Al- wan, “CORAAL QA: A dataset and framework for open domain spontaneous speech question answering from long audio files,” in ICASSP 2024-2024 IEEE ICASSP . IEEE, 2024, pp. 13 371– 13 375

    2024

  13. [21]

    Towards data distillation for end-to-end spoken conversational question answer- ing,

    C. You, N. Chen, F. Liu, D. Yang, and Y . Zou, “Towards data distillation for end-to-end spoken conversational question answer- ing,” arXiv preprint arXiv:2010.08923, 2020

    2010 arXiv

  14. [22]

    AudioBench: A universal benchmark for audio large language models,

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “AudioBench: A universal benchmark for audio large language models,” arXiv e-prints, pp. arXiv–2406, 2024

    2024

  15. [23]

    Holistic evaluation of language models,

    P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y . Zhang, D. Narayanan, Y . Wu, A. Kumar et al., “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  16. [24]

    A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,

    Y . Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V . Do, Y . Xu, and P. Fung, “A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,” in Proc. of the 13th International Joint...

    2023

  17. [25]

    Na- tivQA: Multilingual culturally-aligned natural query for llms,

    M. A. Hasan, M. Hasanain, F. Ahmad, S. R. Laskar, S. Upadhyay, V . N. Sukhadia, M. Kutlu, S. A. Chowdhury, and F. Alam, “Na- tivQA: Multilingual culturally-aligned natural query for llms,” arXiv:2407.09823, 2024

    2024 arXiv

  18. [26]

    Fanar: An arabic-centric multimodal generative ai platform,

    F. Team, U. Abbas, M. S. Ahmad, F. Alam, E. Altinisik, E. As- gari, Y . Boshmaf, S. Boughorbel, S. Chawla, S. Chowdhuryet al., “Fanar: An arabic-centric multimodal generative ai platform,” arXiv:2501.13944, 2025

    2025 arXiv

  19. [27]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in ICML. PMLR, 2023, pp. 28 492–28 518

    2023

  20. [28]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “GPT-4 technical report,”arXiv:2303.08774, 2023

    2023 arXiv

  21. [29]

    ALLam: Large language models for arabic and english,

    M. S. Bari, Y . Alnumay, N. A. Alzahrani, N. M. Alotaibi, H. A. Alyahya, S. AlRashed, F. A. Mirza, S. Z. Alsubaie, H. A. Alahmed, G. Alabduljabbar, R. Alkhathran, Y . Almushayqih, R. Alnajim, S. Alsubaihi, M. A. Mansour, S. A. Hassan, D. M. Alrubaian, A. Alammari, Z. Alawami, ...

    2025

  22. [30]

    LLMeBench: A flexi- ble framework for accelerating LLMs benchmarking,

    F. Dalvi, M. Hasanain, S. Boughorbel, B. Mousi, S. Abdaljalil, N. Nazar, A. Abdelali, S. A. Chowdhury, H. Mubarak, A. Ali, M. Hawasly, N. Durrani, and F. Alam, “LLMeBench: A flexi- ble framework for accelerating LLMs benchmarking,” in Proc. of 18th EACL: System Demonstrations,...

    2024

  23. [31]

    BERTScore: Evaluating text generation with BERT,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “BERTScore: Evaluating text generation with BERT,” in ICLR, 2020

    2020

  24. [32]

    AraBERT: Transformer-based model for Arabic language understanding,

    W. Antoun, F. Baly, and H. Hajj, “AraBERT: Transformer-based model for Arabic language understanding,” in Proc. of the 4th Workshop on OSCAT, 2020, pp. 9–15

    2020

  25. [33]

    BERT: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language under- standing,” in Proc. of the 2019 Conference of the NAACL: HLT , Minnesota, USA, 2019

    2019

  26. [34]

    A critical evaluation of evaluations for long-form question answering,

    F. Xu, Y . Song, M. Iyyer, and E. Choi, “A critical evaluation of evaluations for long-form question answering,” in Proc. of the 61st Annual Meeting of the ACL (Volume 1: Long Papers), 2023, pp. 3225–3245

    2023

  27. [35]

    Data-centric improvements for enhancing multi-modal understanding in spoken conversation modeling,

    M. Chen, R. Sun, and S. ¨O. Arık, “Data-centric improvements for enhancing multi-modal understanding in spoken conversation modeling,” arXiv preprint arXiv:2412.15995, 2024

    2024 arXiv

  28. [36]

    Dual: Discrete spoken unit adaptive learning for textless spoken question answering,

    G.-T. Lin, Y .-S. Chuang, H.-L. Chung, S. wen Yang, H.-J. Chen, S. Dong, S.-W. Li, A. Mohamed, H. yi Lee, and L. shan Lee, “Dual: Discrete spoken unit adaptive learning for textless spoken question answering,” in Proceedings of Interspeech 2022 . In- cheon, Korea: ISCA, September 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.