Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis

Md Sazzad Islam; Sheryl Hsu; Shimanto Bhowmik; Tahsin Reasat; Tawsif Tashwar Dipto

REVIEW 4 major objections 3 minor 23 references

Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper sets out to show that LLMs perform worse in Bengali partly because Bengali text is over-tokenized, and that a new eight-dataset Bengali benchmark can measure the gap.

desk verdict Useful Bengali benchmark release with an overclaimed tokenization finding; the resource is worth engaging, but the causal claim needs controls and the ABPT interpretation is backwards. read the letter →

arxiv 2507.23248 v1 pith:Q2HX4YPK submitted 2025-07-31 cs.CL cs.LG

Shimanto Bhowmik , Tawsif Tashwar Dipto , Md Sazzad Islam , Sheryl Hsu , Tahsin Reasat This is my paper · ORCID

classification cs.CLcs.LG

keywords BengaliNLPLLMevaluationtokenizationefficiencybenchmarktranslationmultilingualLLMslow-resourcelanguageserroranalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish why open-weights LLMs answer Bengali prompts worse than English prompts, and to give the community a standard way to measure it. It translates eight established English benchmarks into Bengali, evaluates ten open models, and finds a consistent Bengali-English accuracy gap that is largest for small models and for the Mistral family, while DeepSeek models stay relatively stable. Its central new claim is an inverse relationship between tokenization efficiency and accuracy: Bengali inputs are split into far more tokens per word and per row than English, and models with more fragmented tokenization tend to score lower. If that claim holds, tokenizer design is a partly independent lever for improving low-resource language performance, not just a side effect of model quality.

What carries the argument

The load-bearing object is the translated benchmark suite plus four tokenization metrics. The suite covers eight datasets (CommonsenseQA, OpenbookQA, BoolQ, Winogrande, HellaSwag, ARC, GSM8K-Main, MMLU) in Bengali, translated with a prompted LLM and post-processed into JSON. The metrics are average tokens per row ($\mathrm{ATPR}$), average tokens per word ($\mathrm{ATPW}$), average bytes per token ($\mathrm{ABPT}$), and average normalized sequence length ($\mathrm{ANSL}$); the last is the ratio of a tokenizer's sequence length to a baseline tokenizer's length for the same text. The paper correlates these metrics with its LLM-Judge score across ten models and reads the negative slope as evidence that over-tokenization hurts accuracy.

What would settle it

A reader could settle the claim by taking one model family, comparing models controlled for size, or fine-tuning a single tokenizer on Bengali and re-running the eight benchmarks: if accuracy does not move with token efficiency once model identity is held fixed, the inverse relationship is a confound rather than a mechanism.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that when the same benchmark questions are translated from English to Bengali, ten recent open LLMs lose accuracy consistently, and the size of the loss tracks how inefficiently the model's tokenizer encodes Bengali. Models like Mistral 7B and LLaMA 3.2 3B show large drops and high response-format error rates; larger LLaMA and Qwen models do better; DeepSeek-R1 models show the smallest cross-language score variation. The tokenization analysis reports that Bengali rows use dramatically more tokens than English rows, and that across all models higher average tokens per row and per word correlate with lower LLM-Judge scores. The paper presents this as evidence that excessive tokenization introduces noise and degrades accuracy, while concise per-word tokenization helps.

Load-bearing premise

The paper's headline correlation is computed by pooling all ten models, which differ in size, family, and pretraining data; the claim that tokenizer inefficiency causes worse accuracy assumes those differences are not what actually produce both the token counts and the scores.

Editorial extensions

If this is right

If tokenization inefficiency degrades accuracy, then improving Bengali tokenizers could raise scores without changing model weights or pretraining data.
The released eight-dataset suite gives later work a fixed yardstick, so Bengali-specific models can be compared on the same translated questions.
The Mistral family's large gap and high response-error rates should be treated as a real weakness to diagnose rather than a benchmark artifact.
DeepSeek-R1's stable cross-language scores suggest some training or tokenization choices transfer well to Bengali and are worth copying.
Smaller models show the largest Bengali drop, so the paper predicts that scaling alone will not close the multilingual gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper stops at correlation; a causal test the authors did not run would hold tokenizer fixed while varying pretraining, or hold model fixed while swapping tokenizers, to see if accuracy follows token counts.
Because the Bengali translations were not manually validated, part of the measured gap could be flawed test items; human-checking a random sample would separate model failure from translation failure.
The bytes-per-token gap implies a cost corollary the authors only gesture at: if Bengali needs more tokens per word, then Bengali inference costs more compute per word, so tokenizer gains are also efficiency gains.
The same translation pipeline could be extended to other underrepresented scripts; the paper's inverse-relationship claim predicts the gap will be largest for scripts whose tokenizers fragment most.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper is best read as a resource-generation effort. The authors translated eight established English benchmarks into Bengali, ran ten open-weight LLMs on them, and released the data and code. That is genuinely useful for a language that needs standardized eval sets. They also did a blind comparison of translation services and picked GPT-4o-mini, and the pipeline is described in enough detail to reproduce. The error analysis is a plus.\n\nThe analytical claims, though, are not supported. The headline inverse relationship between tokenization efficiency and accuracy is not established. Figures 7 and 8 pool ten models from 3B to 72B across four families with no control for size, family, pretraining data, or Bengali exposure. The pooled correlation could just be model quality. No within-model or family-stratified analysis is shown. The authors themselves hedge in the contributions bullet ("due to noise"), but the abstract states the correlation as a finding. That is a real gap.\n\nThere is also a mechanical error in the tokenizer section. The ABPT formula is bytes per token; the text says lower ABPT means coarser, more byte-efficient tokenization, which is backwards. Lower ABPT means each token covers fewer bytes, i.e., finer granularity. That reverses the interpretation of Figures 9 and 10.\n\nThe paper's own limitations concede both premises the scores depend on: the Bengali translations were not manually validated, and the LLM judge is untested. With no validation sample, the "high-quality benchmark" claim is unearned. The accuracy numbers may be a useful initial signal, but they are not reliable enough for fine-grained conclusions.\n\nCredit where due: the resource release, cost transparency (~$200), and reproducibility-oriented pipeline are real assets. The paper would benefit from a major revision: add a manual validation subsample, run significance tests and model-controlled partial correlations, fix the ABPT interpretation, and soften the abstract to match the design.\n\nWho is this for? Researchers building low-resource benchmarks and anyone needing a quick multilingual sanity check for Bengali model evaluation. The benchmark release is worth citing once on HuggingFace; the tokenization claim should not be cited in its current form.\n\nRecommendation: send to peer review, but expect major revision. The resource value and the testable empirical questions justify referee time; the overclaiming does not.

Referee Report

4 major / 3 minor

Summary. This paper translates eight English benchmark datasets (HellaSwag, Winogrande, CommonsenseQA, BoolQ, OpenbookQA, ARC, GSM8K, MMLU) into Bengali using GPT-4o-mini, evaluates ten open-weight LLMs on both languages using exact-match accuracy, response adherence, and an LLM judge, and then analyzes tokenization statistics (ATPR, ATPW, ABPT, ANSL) to relate tokenization efficiency to model scores. The authors report a consistent English-Bengali performance gap and claim an inverse relationship between tokenization efficiency and accuracy, alongside a release of the translated datasets and code.

Significance. If the tokenization-accuracy relationship were substantiated, it would provide a concrete, actionable insight into why underrepresented languages underperform in LLMs, and the released benchmarks could support future Bengali evaluation. However, the paper's central quantitative claim is not established by the current analysis, and the benchmark quality is limited by unvalidated machine translation and an untested LLM judge. The release of the datasets and code is nevertheless a useful contribution to a low-resource language community.

major comments (4)

[Section 3.4, Figures 7 and 8] The claimed inverse relationship between tokenization efficiency and LLM accuracy is based on pooled correlations over model-dataset pairs from ten models of different sizes and families (Table 3) with no control for model identity, parameter count, or pretraining data. Since larger, higher-quality models may systematically have more Bengali-efficient tokenizers, the observed correlation could reflect model capability rather than a tokenization effect. No within-model, family-stratified, or partial correlation analysis is reported, and no significance tests accompany the correlations. The contribution bullet in Section 1 hedges this same finding as '(due to noise)', which is inconsistent with the causal phrasing in the abstract. This point must be re-analyzed with appropriate controls before the headline claim can be evaluated.
[Section 3.4.4] The interpretation of ABPT is inverted relative to its formula. With r_i = B_i / l_i, a lower ABPT means each token spans fewer bytes, i.e., finer-grained, more fragmented tokenization, not 'coarser, more byte-efficient tokenization' as stated. The sentence 'lower ABPT means each token encodes more of the original text' is also backwards. Because Figures 9 and 10 are used to compare English and Bengali byte efficiency, this reversal directly contradicts the conclusion that English has higher ABPT and is therefore more byte-efficient. The definition and all downstream interpretations must be corrected.
[Section 5 and Section 2.3] The claim in the contributions of releasing 'a comprehensive suite of high-quality Bengali benchmark datasets' is not supported by the evidence provided. The translation pipeline uses a single automatic model (GPT-4o-mini) and the paper explicitly states that translations 'were not manually validated.' The blind review in Section 2.3 compares three translation services without reporting any quantitative or qualitative scoring results. Similarly, the LLM-judge metric is untested against human annotations, despite being one of the three headline metrics. These gaps directly affect the validity of the performance numbers reported in Tables 4-6, so the benchmark quality claim and the numerical results must be substantially tempered or the data must be validated.
[Section 3.1 and Table 3] The experimental setup is under-specified and internally inconsistent. The text states 'we selected all available open-source multilingual LLaMA models,' but Table 3 includes Qwen, Mistral, and DeepSeek models, and the promised list of specific models is missing from the text. The evaluation is also described as 'without finetuning the Llama family of models,' which does not cover the other families. Please clarify the model selection procedure and list the exact model checkpoints, inference settings (decoding parameters, temperature, number of samples), and the split used for each dataset.

minor comments (3)

[Section 3.4.3 and 3.4.4] The paragraph beginning 'This metric captures the average amount of raw text (in bytes) that each token represents' appears under the ATPW heading but describes ABPT; the discussion of bytes-per-token should be moved to Section 3.4.4.
[Abstract and Section 6] The abstract says the code is available on GitHub and the datasets on Hugging Face, but the full text gives no URLs; please include the actual repository links.
[Section 3.4.5, Figure 6b] In the text, 'with bn and lang datasets ranging between 2-7 tokens per word' appears to contain a typo; 'lang' should likely be 'en' or 'English'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; tokenization metrics and accuracy are measured independently, and self-created benchmarks are standard rather than circular.

full rationale

The paper's derivation chain is not circular. The benchmark datasets are created by translating English benchmarks into Bengali with gpt-4o-mini, then ten open-weight LLMs are evaluated on those datasets using accuracy, RER/RAR, and a separate LLM judge. The tokenization metrics (ATPR, ATPW, ABPT, ANSL) are computed directly from tokenizer outputs on the translated datasets, independently of model accuracy. Accuracy is measured against ground-truth labels or separate judge verdicts; it is not fitted from tokenization statistics, and no tokenization parameter is tuned to reproduce the observed scores. The reported inverse relationship between tokenization efficiency and accuracy is an empirical correlation over pooled model-dataset pairs, not a quantity defined in terms of itself. Confounding by model size or family would be a statistical-validity concern, not circularity. The self-created benchmarks are the standard contribution of a benchmark paper: the authors do not rely on their own prior results, uniqueness theorems, or ansatz-carrying citations to force the conclusions. The ABPT interpretation inconsistency in Section 3.4.4 is an internal error in describing the metric, not a self-referential definition. No load-bearing self-citation, fitted-input-renamed-as-prediction, or definitional equivalence was found. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical findings rest on translation quality, the LLM judge, and an uncontrolled correlation; none of these is independently validated in the paper.

free parameters (2)

Translation temperature = 1.0
The authors increased temperature to 1.0 to reduce repetitive translations (Section 2.4). This hand-chosen setting affects translation quality and therefore all downstream scores.
Number of translated datasets retained for evaluation = 8 of 20
Twenty datasets were translated but only eight were selected for evaluation with no stated selection criterion (Section 2.5). This post-hoc choice shapes all aggregates and can bias the benchmark.

assumptions (4)

domain assumption The LLM judge provides accurate correctness assessments.
Explicitly assumed in Section 3.2 and acknowledged as uncertain in Section 5. All LLM-Judge scores and the tokenization correlations in Figures 7 and 8 depend on it.
domain assumption Machine-translated Bengali datasets without manual validation are valid evaluation benchmarks.
Stated in Section 5 as a limitation; if translations contain systematic errors, reported scores and gaps are unreliable.
domain assumption English and Bengali versions are parallel, so cross-language score differences reflect model competence rather than translation artifacts.
The methodology translates and compares directly (Section 2.2); no validation that meaning is preserved.
ad hoc to paper The correlation between tokenization metrics and scores is not confounded by model size or architecture.
The paper infers an inverse relationship from pooled data in Figures 7 and 8 without controlling for model family or size.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis." pith.science (2026). https://pith.science/paper/Q2HX4YPK

@misc{pith2026250723248,
  author       = {Pith},
  title        = {Pith review of: Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2HX4YPK}},
  note         = {Machine review of arXiv:2507.23248}
}

read the original abstract

Bengali is an underrepresented language in NLP research. However, it remains a challenge due to its unique linguistic structure and computational constraints. In this work, we systematically investigate the challenges that hinder Bengali NLP performance by focusing on the absence of standardized evaluation benchmarks. We then evaluated 10 recent open source Large Language Models (LLMs) in 8 of the translated datasets and performed a comprehensive error analysis to pinpoint their primary failure modes. Our findings reveal consistent performance gaps for Bengali compared to English, particularly for smaller models and specific model families like Mistral. We also identified promising robustness in certain architectures, such as DeepSeek, that maintain more stable performance across languages. Our analysis reveals an inverse relationship between tokenization efficiency and LLM accuracy where models tend to perform worse when inputs are excessively tokenized, whereas more efficient \& concise tokenization results in improved performance. These findings highlight critical areas where current models fall short and underscore the need for improved dataset quality and evaluation methodologies tailored to multilingual contexts. This work will catalyze further research on NLP for underrepresented languages, helping to democratize access to advanced language technologies worldwide. The code and dataset used in this research is publicly available at https://github.com/BengaliAI/bn-llm-benchmark.

Figures

Figures reproduced from arXiv: 2507.23248 by the authors.

**Figure 1.** Methodology Overview 2.1 Dataset Selection To select appropriate datasets, we refer to the methodology used in the white paper by LLaMA, identifying commonly used datasets that align with our research objectives. This approach allowed us to ensure the inclusion of high-quality, diverse and representative text corpora for Bengali language modeling. A summary of the dataset statistics is attached. 2.2 Translation Fo… view at source ↗

**Figure 2.** Dataset Distribution • Missing Entries Due to Multithreading: Some dataset entries were skipped due to parallel processing errors. We resolved this issue by analyzing logs and re-processing the missing translations to ensure dataset completeness. • Decoding Errors: Some dataset entries had decoding errors due to the JSON not being parsed properly . These errors include missing comma(,)delimiters, unclosed quotation… view at source ↗

**Figure 3.** Language-wise score trends and the effect of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: The models sorted by average of the score [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Architecture-wise performance and robustness [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Comparison of tokenization efficiency metrics [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 8.** Figure 8: Scatter plot of tokenization efficiency metrics [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 9.** Figure 9: Comparison of tokenization efficiency met [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 10.** Figure 10: Effect of tokenization efficiency measured [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 9 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Emre Can Acikgoz, Mete Erdogan, and Deniz Yuret. 2024. https://api.semanticscholar.org/CorpusID:269626406 Bridging the bosphorus: Advancing turkish large language models through strategies for low-resource language adaptation and benchmarking . ArXiv, abs/2405.04685

arXiv 2024
[4]

Samiul Alam, Tahsin Reasat, Asif Shahriyar Sushmit, Sadi Mohammad Siddique, Fuad Rahman, Mahady Hasan, and Ahmed Imtiaz Humayun. 2021. A large multi-target dataset of common bengali handwritten graphemes. In International Conference on Document Analysis and Recognition, pages 383--398. Springer

work page 2021
[5]

BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla

Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin Mubasshir, M. Sohel Rahman, Anindya Iqbal, and Rifat Shahriyar. 2021. https://api.semanticscholar.org/CorpusID:230437858 Banglabert: Combating embedding barrier for low-resource language understanding . ArXiv, abs/2101.00204

work page Pith review arXiv 2021
[6]

Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. https://arxiv.org/abs/2402.01035 Getting the most out of your tokenizer for pre-training and domain adaptation . Preprint, arXiv:2402.01035

arXiv 2024
[7]

Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. https://openreview.net/forum?id=yIEyHP7AvH Khayyam challenge (persian MMLU ): Is your LLM truly wise to the persian language? In First Conference on Language Modeling

work page 2024
[8]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3...

arXiv 2024

Show all 23 references

[9]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

2025 arXiv
[10]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 arXiv
[11]

Golam Kabir, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, M Saiful Bari, and Enamul Hoque

M. Golam Kabir, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, M Saiful Bari, and Enamul Hoque. 2023. https://api.semanticscholar.org/CorpusID:262465154 Benllm-eval: A comprehensive evaluation into the potentials and pitfalls of large language models on be...

2023 arXiv
[12]

Mohammed Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad B, Varun G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh Khapra. 2024. https://doi.org/10.18653/v1/2024.acl-long.843 Indicllmsuite: A blueprint for ...

2024 doi
[13]

Yamshchikov

Pierre-Carl Langlais, Carlos Rosas Hinostroza, Mattia Nee, Catherine Arnett, Pavel Chizhov, Eliot Krzystof Jones, Irène Girard, David Mach, Anastasia Stasenko, and Ivan P. Yamshchikov. 2025. https://arxiv.org/abs/2506.01732 Common corpus: The largest collection of ethical data...

2025 arXiv
[14]

Mistral AI Team . 2025. Mistral-Small-24B-Instruct-2501 . Hugging Face Model Card, Apache-2.0 licensed. https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501 (accessed 2025-07-20)

2025
[15]

Shahriar Kabir Nahin, Rabindra Nath Nandi, Sagor Sarker, Quazi Sarwar Muhtaseem, Md Kowsher, Apu Chandraw Shill, Md Ibrahim, Mehadi Hasan Menon, Tareq Al Muntasir, and Firoj Alam. 2025. https://arxiv.org/abs/2502.11187 Titullms: A family of bangla llms with comprehensive bench...

2025 arXiv
[16]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

2025 arXiv
[17]

Nishat Raihan and Marcos Zampieri. 2025. https://arxiv.org/abs/2503.10995 Tigerllm - a family of bangla large language models . Preprint, arXiv:2503.10995

2025 arXiv
[18]

Shahidul Salim, Hasan Murad, Dola Das, and Faisal Ahmed

Md. Shahidul Salim, Hasan Murad, Dola Das, and Faisal Ahmed. 2023. https://api.semanticscholar.org/CorpusID:265056023 Banglagpt: A generative pretrained transformer-based model for bangla language . 2023 International Conference on Information and Communication Technology for ...

2023
[19]

Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, Jonathan H

H M Quamran Hasan Sheikh Shafayat, Minhajur Rahman, Chowdhury Mahim, Rifki Afina, James Putri, Alice Thorne, Oh, Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel C. Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, Jonathan H. Choi,...

2024 arXiv
[20]

Arif Shahriar and Denilson Barbosa. 2024. https://api.semanticscholar.org/CorpusID:269803986 Improving bengali and hindi large language models . In International Conference on Language Resources and Evaluation

2024
[21]

Sam Witteveen and Martin Andrews. 2019. https://api.semanticscholar.org/CorpusID:208092413 Paraphrasing with large language models . In Conference on Empirical Methods in Natural Language Processing

2019
[22]

Dian Yu, Kenji Sagae, and Zhou Yu. 2021. https://api.semanticscholar.org/CorpusID:232307805 Attribute alignment: Controlling text generation from pre-trained language models . In Conference on Empirical Methods in Natural Language Processing

2021
[23]

Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. https:/...

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Emre Can Acikgoz, Mete Erdogan, and Deniz Yuret. 2024. https://api.semanticscholar.org/CorpusID:269626406 Bridging the bosphorus: Advancing turkish large language models through strategies for low-resource language adaptation and benchmarking . ArXiv, abs/2405.04685

arXiv 2024

[4] [4]

Samiul Alam, Tahsin Reasat, Asif Shahriyar Sushmit, Sadi Mohammad Siddique, Fuad Rahman, Mahady Hasan, and Ahmed Imtiaz Humayun. 2021. A large multi-target dataset of common bengali handwritten graphemes. In International Conference on Document Analysis and Recognition, pages 383--398. Springer

work page 2021

[5] [5]

BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla

Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin Mubasshir, M. Sohel Rahman, Anindya Iqbal, and Rifat Shahriyar. 2021. https://api.semanticscholar.org/CorpusID:230437858 Banglabert: Combating embedding barrier for low-resource language understanding . ArXiv, abs/2101.00204

work page Pith review arXiv 2021

[6] [6]

Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. https://arxiv.org/abs/2402.01035 Getting the most out of your tokenizer for pre-training and domain adaptation . Preprint, arXiv:2402.01035

arXiv 2024

[7] [7]

Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. https://openreview.net/forum?id=yIEyHP7AvH Khayyam challenge (persian MMLU ): Is your LLM truly wise to the persian language? In First Conference on Language Modeling

work page 2024

[8] [8]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3...

arXiv 2024