Pith. sign in

REVIEW 4 major objections 7 minor 36 references

Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new parallel benchmark of over 5,000 Dzongkha-English exam questions shows that even the strongest large language models answer the same questions far less accurately in Dzongkha, and that appending an English translation narrows the gap.

desk verdict DZEN is a genuinely useful new parallel Dzongkha–English benchmark, but the headline accuracy gap is not yet established as a capability difference without a contamination check. read the letter →

arxiv 2505.18638 v2 pith:NWW5DO2H submitted 2025-05-24 cs.CL

Md. Tanzib Hosain , Rajan Das Gupta , Md. Kishor Morol This is my paper
classification cs.CL
keywords Dzongkhalow-resourcelanguagesmultilingualquestionansweringLLMbenchmarkchain-of-thoughtpromptingtranslation-augmentedBhutannationalexamsfoundationmodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents DZEN, a parallel corpus of 5,161 multiple-choice questions drawn from Bhutan's national eighth-, tenth-, and twelfth-grade science and mathematics exams, with official English and Dzongkha versions of every question. The authors use it to benchmark GPT-3.5, GPT-4, Claude 2.1, LLaMA-2, and Mistral, arguing that the parallel structure makes the English-Dzongkha comparison fair, and they report a large accuracy gap: GPT-4 drops from roughly 77-87% in English to 55-65% in Dzongkha, while GPT-3.5 falls to about 19-28% in Dzongkha. They also find that chain-of-thought prompting improves reasoning and application questions by 12-22 percentage points while leaving factual questions nearly unchanged, and that appending an English translation to a Dzongkha prompt, even a machine-generated one, raises Dzongkha accuracy across subjects. If the parallel-pair construction is sound, the contribution is a reusable bilingual instrument, a measurement of a genuine low-resource language gap, and a practical prompting recipe for narrowing it.

What carries the argument

The carrying object is DZEN itself: 5,161 four-option multiple-choice questions from Bhutan's national board examinations (55.4% twelfth grade, 36.0% tenth grade, 8.7% eighth grade) in biology, chemistry, physics, mathematics, and general science, each with an official English and Dzongkha version. The cross-lingual comparison rests on the parallel-pair pipeline: each English question is translated to Dzongkha with Google Translate, matched to the official Dzongkha version by cosine similarity of OpenAI text-embedding-ada-002 embeddings, verified by two native Dzongkha speakers, and pairs whose two language versions disagree on the answer key are discarded. Questions are then classified by GPT-4 into three categories, Factual Knowledge, Procedural & Application, and Reasoning, which drive the chain-of-thought analysis. On top of this, the prompting machinery is a zero-shot and k-shot CoT protocol plus a translation-append prompt that attaches an English version while requiring the model to reason and answer in Dzongkha.

What would settle it

Audit a random sample of about 200 DZEN pairs: have two independent bilingual annotators, blind to model results, verify that the English and Dzongkha versions are the same question and that the official answer keys give the same correct option in both languages. If more than about 2-3% of pairs disagree, the reported gap could be mostly label noise. Separately, run membership-inference-style probes on GPT-3.5, GPT-4, and Claude 2.1 to test whether BCSEA exam content appears in their training data; if it does, the absolute accuracies must be re-measured on fresh, non-public questions.

Watch

Extended reading notes

Core claim

The central claim is that DZEN is a valid parallel benchmark, the first Dzongkha dataset covering a wide range of topics and question formats, and that measuring LLMs on it reveals a genuine capability gap between languages. On the same questions, the strongest model (GPT-4) scores roughly 77-87% in English but only 55-65% in Dzongkha, and the gap widens sharply for weaker models; Claude 2.1 is the outlier, trailing GPT-3.5 clearly in English but matching or exceeding it in several Dzongkha subjects. The paper further establishes two prompting effects: chain-of-thought reasoning helps reasoning and application questions by 12-22 points while barely moving factual ones, and giving the model a Dzongkha question with an English translation attached, whether human-made or produced by GPT-3.5 or GPT-4, improves Dzongkha accuracy in every subject tested. The same translation-append strategy transfers to a Dzongkha version of Big-Bench Hard, gaining about 6 percentage points over Dzongkha-only prompting. The authors present the English-Dzongkha gap as a real shortfall in low-resource language capability, not an artifact of the test.

Load-bearing premise

The load-bearing premise is that every English question and its matched Dzongkha version are the same question with the same correct answer in both languages, and that the tested models had not memorized the publicly available exam papers; the paper reports no error rate for its matching step, no agreement measure between the two native-speaker checkers, and no contamination check, so a modest rate of misalignment or memorization would contaminate the measured language gap and every prompting comparison.

Editorial extensions

If this is right

  • If DZEN's parallel pairs are faithful, the reported accuracy gap is a real measurement: the same questions answered in Dzongkha score roughly 20 to 40 points lower than in English, quantifying how far low-resource languages lag in current foundation models.
  • Translation-augmented prompting offers a no-retraining route to better Dzongkha answers, and since LLM-generated translations perform nearly as well as human ones, the recipe applies to languages that lack professional translation resources.
  • Chain-of-thought prompting is category-dependent: it buys 12-22 points on reasoning and application questions but little on factual ones, so factual-heavy subjects need different interventions.
  • Open-source models trail proprietary ones by a wide margin even in English and were not usable in Dzongkha in these tests, which means the gap is also an equity gap for regions that rely on open models.
  • DZEN is a reusable bilingual instrument: the same 5,161 pairs can support later work on Dzongkha generation, translation quality, and automatic answer verification, not just the accuracy comparisons reported here.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Read as a diagnostic rather than a leaderboard, the bilingual pairs could also grade machine translation: mismatches between the official Dzongkha and the Google-translated version would track translation difficulty, not just model capability, and DZEN could be re-purposed as a translation-evaluation set.
  • If contamination of the public exam questions is ever checked and found in the tested models, the absolute accuracies would need to be re-baselined, but the relative findings, CoT helping reasoning more than factual questions and translation append helping Dzongkha, would likely survive because they are comparisons within the same data.
  • Claude 2.1's relatively stronger Dzongkha performance suggests the English-Dzongkha gap is not a uniform low-resource penalty but depends on tokenizer coverage and training mix; comparing tokenization efficiency for Dzongkha across these models would be a cheap test of that explanation.
  • A testable extension of the paper's recipe: apply translation-append prompting to other low-resource languages with official bilingual exam material, such as other South Asian school systems, to see whether the roughly 6-point transfer gain on Big-Bench Hard generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces DZEN, a parallel corpus of 5,161 English and Dzongkha multiple-choice questions taken from Bhutan's national middle- and high-school examinations. The authors categorize the questions as factual, procedural/application, or reasoning; benchmark GPT-3.5, GPT-4, Claude 2.1, LLaMA-2 7B/13B, and Mistral 7B in zero- and few-shot settings; and study chain-of-thought prompting and translation-augmented prompting. The central empirical claims are that there is a significant English--Dzongkha performance gap, that chain-of-thought helps reasoning and application questions more than factual ones, and that appending English translations improves Dzongkha accuracy. The paper also presents a GPT-4-translated Dzongkha version of selected Big-Bench Hard tasks.

Significance. If the parallel corpus is clean and the evaluation protocol is sound, DZEN is a useful new resource: it is one of the first bilingual benchmarks for Dzongkha, it is built from official national examinations, and it targets a genuinely low-resource language. The dataset release, the native-speaker alignment process, and the question-type taxonomy are concrete strengths, and the paper makes falsifiable predictions about the English--Dzongkha gap and about prompting effects. The benchmark's value is currently weakened by the absence of contamination screening, by the lack of uncertainty quantification, and by underspecified annotation procedures. The use of GPT-4 for question categorization and translation is not, by itself, circular; the problem is that these GPT-4 outputs are not independently validated at the scale needed to support the paper's conclusions.

major comments (4)
  1. [§3.1.2, §5.1.1, Table 7] The headline claim of a capability gap between English and Dzongkha is not yet established because no contamination check is reported. The source exams are publicly released (footnote 1), so the English questions are likely to appear in web-scale pretraining data at a much higher rate than the newly translated Dzongkha versions. This asymmetry can produce an accuracy gap in exactly the direction observed in Table 7. Please report a contamination analysis: for example, exact and near-duplicate overlap with Common Crawl, C4, and The Pile, plus API-based memorization probes on a sample of questions. If contamination cannot be fully ruled out, the abstract and Section 5.1 should reframe the claim as a benchmark-specific gap rather than a demonstrated capability gap.
  2. [Tables 7–9, Figures 1–10] The paper reports single accuracy values without sample sizes, confidence intervals, or significance tests, yet the abstract uses the word 'significant.' For paired comparisons on the same questions, McNemar's test or a comparable paired test is appropriate. This matters especially for small subsets: Appendix B.1 uses only 105 questions per subject, and several differences in Table 9 are only a few points (e.g., 34.8 vs. 37.4 in Object Counting). Without intervals, the qualitative claims about which prompting effects 'help' or are 'unchanged' are not statistically supported.
  3. [§3.1.2] The parallel-corpus construction is not fully documented. The paper states that Google Translate plus cosine similarity with OpenAI embeddings was followed by manual checking by two native speakers, but it does not report the cosine-similarity threshold, the number of pairs discarded by the matcher, the number of questions removed for divergent answer keys, or the inter-annotator agreement between the two speakers. If even a small fraction of pairs are misaligned or have language-dependent correct answers, the per-language accuracies in Table 7 are contaminated by label noise. Please report these numbers and, where disagreements occurred, how they were resolved.
  4. [§4.4] The evaluation procedure is underspecified. The paper says model outputs were manually examined to see whether the final response matched the ground truth, but it does not state how many outputs were examined, who performed the examination, what rules were used for partially matching answers, or how disagreements were adjudicated. This is particularly important for Dzongkha, where models may not follow the requested output format. Please provide the annotation protocol and an agreement measure; otherwise the accuracy numbers in Tables 7 and 8 are not reproducible.
minor comments (7)
  1. [§4.2] The statement that open-source models 'don't work well on Dzongkha' is presented as a justification for testing them only in English, but no Dzongkha results are shown to support it; please either include a small Dzongkha sample or explicitly label this as an assumption.
  2. [Table 1] The column headers 'grouped by level' and 'grouped by field' are confusing: the first block groups by subject (Biology, Chemistry, etc.) and the second by grade level (12th, 10th, 8th). Please relabel the blocks accordingly.
  3. [Abstract] The dataset URL in the abstract contains a space ('llm dzongkha evaluation') and should be corrected to 'llm_dzongkha_evaluation'.
  4. [Figures 17 and 20] The captions of Figures 17 and 20 contain the typo '1oth grade'; this should be '10th grade.'
  5. [Appendix E.1] The list of selected Big-Bench Hard tasks mentions 'Multistep Arithmetic' twice; the duplicate should be removed.
  6. [Table 4] Table 4 is difficult to interpret: in the 'w/o' (without translation) condition, rows still list CoT ✓/× even though the condition appears to have no chain-of-thought. Please clarify the experimental design and label rows and columns unambiguously.
  7. [§6.1] The text says there is 'little difference between zero-shot and few-shot settings,' but Figure 4 does not show zero-shot baselines; please include them or revise the wording to match what is actually plotted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DZEN is an external benchmark built from official Bhutanese exams; the empirical comparisons and prompting results do not reduce to their inputs by construction.

full rationale

The paper has no fitted parameters, no derived equations, and no load-bearing self-citation chain. The central artifact, DZEN, is constructed from official Bhutanese national board examinations with an explicit matching procedure (Google Translate, cosine similarity, manual verification by two native Dzongkha speakers) and removal of items with divergent ground truths; the subsequent accuracy measurements are direct evaluations, not predictions from a fitted model. The English-Dzongkha performance gap and the prompting comparisons are empirical observations on this external corpus, so they are not equivalent to their inputs by construction. The use of GPT-4 to categorize questions, correct grammar, or generate translations is an annotation or augmentation tool, not a step that defines the outcome being measured; even if those annotations were imperfect, that would be a measurement-quality concern, not circularity. The paper also discloses limitations in Appendix A, including the removal of figure-based questions and the multiple-choice format. No 'uniqueness theorem' or prior-work-by-the-same-authors citation is invoked to force a conclusion, and no known empirical pattern is merely renamed as a contribution. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the quality of the parallel alignment and ground truth, not on fitted constants. The paper introduces no free parameters and no theoretical entities. Its assumptions are domain assumptions about dataset validity, evaluator reliability, and absence of contamination; all are plausible but none are quantitatively verified in the manuscript.

assumptions (4)
  • domain assumption Official Bhutanese exam answer guides are the correct ground truth for both the English and Dzongkha versions of every question.
    Section 3.1.1 states ground truth was established by collecting physical test papers and using answer guides; no independent verification of answer correctness is described.
  • ad hoc to paper Google Translate plus cosine similarity with OpenAI embeddings, followed by manual checking by two native speakers, produces correct parallel English-Dzongkha question pairs.
    Section 3.1.2 defines the matching pipeline but reports no thresholds, no error rate, and no inter-annotator agreement for the manual check.
  • domain assumption Publicly available national exam questions were not memorized by the evaluated models during training.
    The paper does not test for contamination, despite the questions coming from public board exams, which could leak into LLM training data.
  • domain assumption Manual examination of model outputs correctly identifies whether the final answer matches the ground truth.
    Section 4.4 says outputs were manually examined but does not specify evaluators, instructions, or agreement statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models." pith.science (2026). https://pith.science/paper/NWW5DO2H

@misc{pith2026250518638,
  author       = {Pith},
  title        = {Pith review of: Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NWW5DO2H}},
  note         = {Machine review of arXiv:2505.18638}
}
read the original abstract

In this work, we provide DZEN, a dataset of parallel Dzongkha and English test questions for Bhutanese middle and high school students. The over 5K questions in our collection span a variety of scientific topics and include factual, application, and reasoning-based questions. We use our parallel dataset to test a number of Large Language Models (LLMs) and find a significant performance difference between the models in English and Dzongkha. We also look at different prompting strategies and discover that Chain-of-Thought (CoT) prompting works well for reasoning questions but less well for factual ones. We also find that adding English translations enhances the precision of Dzongkha question responses. Our results point to exciting avenues for further study to improve LLM performance in Dzongkha and, more generally, in low-resource languages. We release the dataset at: https://github.com/kraritt/llm_dzongkha_evaluation.

Figures

Figures reproduced from arXiv: 2505.18638 by the authors.

Figure 1
Figure 1. CoT reasoning average score in few-shot scenarios for the English GPT 3.5 Turbo. Note [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Subject-by-subject CoT performance in English. Note that w/ denotes with and w/o denotes [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance summary in English by question type. Note that w/ denotes with and w/o [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Dzongkha and English k-shot CoT prompted on GPT 3.5. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Impact of CoT reasoning on the GPT-3.5 for DZEN English throughout k-shot. Note that [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Impact of CoT reasoning on GPT-3.5 across k-shot. Note that w/ denotes with and w/o [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: CoT performance summary in Dzongkha by subject. Note that w/ denotes with and w/o [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: CoT performance summary in Dzongkha by question type. Note that w/ denotes with and [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Consequences of adding an English version was produced by LLM in the adding experiment. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Answering questions in Dzongkha is made easier by including an English translation. Note [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Grammatical errors’ effects on GPT 3.5 performance. Note that w/ denotes with and w/o [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: GPT-4 prompt for creating Grammar-corrected questions. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: DZEN dataset categorized questions prompt. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Proprietary models prompt for DZEN zero-shot benchmark. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Proprietary models prompt for DZEN Few-shot benchmark. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Zero-shot append experiment prompt. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Physics sample questions for the 1oth grade in DZEN by category. [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Chemistry sample questions for the 1oth grade in DZEN by category. [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Biology sample questions for the 10th Grade in DZEN by category. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Mathematics sample questions for the 1oth grade in DZEN by category. [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 14 canonical work pages

  1. [1]

    Harnessing ai so that all students benefit: A nonprofit approach for equal access, 2023

    Khan Academy. Harnessing ai so that all students benefit: A nonprofit approach for equal access, 2023. URL https://blog.khanacademy.org/harnessing-ai-so-that-all-students-benefit-a-nonprofit-approach-for-equal-access/. Accessed: 2023-12-15

    work page 2023

  2. [2]

    Do all languages cost the same? tokenization in the era of commercial language models

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 9904--9923, Singapore, 2023. Association for Computational Linguistics

    work page 2023

  3. [3]

    Mega: Multilingual evaluation of generative ai

    Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528, 2023

    arXiv 2023

  4. [4]

    A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023

    arXiv 2023

  5. [5]

    Choi, Kristin E

    Jonathan H. Choi, Kristin E. Hickman, Amy Monahan, and Daniel Schwarcz. Chatgpt goes to law school. Available at SSRN, 2023

    work page 2023

  6. [6]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  7. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  8. [8]

    Khapra, Anoop Kunchukuttan, and Pratyush Kumar

    Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    work page 2023

Show all 36 references
  1. [9]

    Recipe: How to integrate chatgpt into efl writing education

    Jieun Han, Haneul Yoo, Yoonsu Kim, Junho Myung, Minsun Kim, Hyunseung Lim, Juho Kim, Tak Yeon Lee, Hwajung Hong, So-Yeon Ahn, and Alice Oh. Recipe: How to integrate chatgpt into efl writing education. In Proceedings of the Tenth ACM Conference on Learning @ Scale, pp.\ 416--42...

    2023

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  3. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  4. [12]

    Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004, 2023

    2023 arXiv

  5. [13]

    Cosmos qa: Machine reading comprehension with contextual commonsense reasoning

    Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference ...

    2019

  6. [14]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  7. [15]

    Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu

    Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 12359--12374, 2023

    2023

  8. [16]

    Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al

    Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language mode...

    2023

  9. [17]

    Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning

    Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613, 2023

    2023 arXiv

  10. [18]

    Evaluating the logical reasoning ability of chatgpt and gpt-4

    Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439, 2023

    2023 arXiv

  11. [19]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  12. [20]

    Language model tokenizers introduce unfairness between languages

    Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. arXiv preprint arXiv:2305.15425, 2023

    2023 arXiv

  13. [21]

    Xcopa: A multilingual dataset for causal commonsense reasoning

    Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. Xcopa: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 2362--2376, Onl...

    2020

  14. [22]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011

    2011

  15. [23]

    Language models are multilingual chain-of-thought reasoners

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057, 2022

    arXiv

  16. [24]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint ...

    2022 arXiv

  17. [25]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  18. [26]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2019

  19. [27]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  20. [28]

    Aya model: An instruction finetuned open-access multilingual language model

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting...

    2024

  21. [29]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022 a

    2022 arXiv

  22. [30]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022 b

    2022

  23. [31]

    Languages by total number of speakers, 2023

    Adam Zeidan. Languages by total number of speakers, 2023

    2023

  24. [32]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, Florence, Italy, 2019. Association for Com...

    2019

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  26. [34]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  27. [35]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  28. [36]

    X@HQ " H ٝyH f Ac ΁;ޝ SN4 ! B !q޽ b6g ! B gggO޼ygpi ǎ c 9r ,! B !]6 ;vf͚t¥i̛7_ ! Bl#R ! Bd B !

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.