{"id":"2a6c3808-81c0-4fc0-88ee-1b026da7affe","arxiv_id":"2507.23248","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors release eight Bengali benchmarks translated from English and report that models with more fragmented Bengali tokenization tend to score lower.","lead":"This paper translated eight English AI benchmarks into Bengali and ran ten open-source language models on them. It reports that Bengali is tokenized less efficiently and that heavier tokenization is linked to lower model accuracy, while releasing the translated datasets and code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The tokenization-efficiency claim rests on pooled correlations across 10 models of different sizes and families, with no model-identity control, and §3.4.4's ABPT interpretation is internally inverted; the central causal claim is not established.","rationale":"The reader's weakest assumption correctly identifies the load-bearing problem: the tokenization-performance correlation is pooled over heterogeneous models with no controls, so the central claim is confounded by model identity. My reading agrees, and I add that the ABPT metric is described in contradiction with its own formula, further weakening the tokenization analysis. The paper does contribute translated datasets and a reproducible pipeline, and those resources may be useful despite the analytical problems. However, the abstract's strongest claim — that inefficient tokenization causes worse Bengali accuracy — is not supported by the evidence presented, and the limitations section explicitly concedes that the evaluation scores rest on unvalidated translations and a possibly unreliable judge. The REJECT verdict is therefore appropriate, and I see no reason to change it.","tokens_in":12528,"tokens_out":2550,"duration_ms":32666,"concrete_test":"Recompute the ATPR and ATPW versus LLM-Judge correlations from Figures 7 and 8 as partial Spearman correlations controlling for model parameter count (or model-family indicators), and also recompute the same correlations within each model across the eight datasets. If the negative ATPR correlation or the positive ATPW correlation does not survive within-model or partial-correlation analysis, the inverse relationship is a model-quality confound rather than evidence about tokenization efficiency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is an inverse relationship between tokenization efficiency and LLM accuracy. The evidence in Figures 7 and 8 pools model-dataset pairs across ten models spanning 3B–72B parameters and multiple families (Table 3), with no control for model size, family, pretraining data, or Bengali exposure. If larger or better-performing models happen to have more compact Bengali tokenizers, the pooled correlation would reflect model quality rather than a tokenization effect. No within-model, partial-correlation, or family-stratified analysis is reported, so the causal reading 'excessive tokenization degrades accuracy' is unsupported. The contribution bullet itself hedges the same correlation as '(due to noise),' which is inconsistent with the abstract's causal phrasing. Additionally, §3.4.4 defines ABPT as average bytes per token but states that 'lower ABPT indicates coarser, more byte-efficient tokenization.' With the given formula, lower ABPT means each token covers fewer bytes, i.e., finer and more fragmented encoding, so the interpretation of Figures 9 and 10 is backwards. The paper's own Limitations also concede that the Bengali translations were not manually validated and that the LLM judge may be unreliable, leaving no independent support for the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper translates eight English benchmark datasets (HellaSwag, Winogrande, CommonsenseQA, BoolQ, OpenbookQA, ARC, GSM8K, MMLU) into Bengali using GPT-4o-mini, evaluates ten open-weight LLMs on both languages using exact-match accuracy, response adherence, and an LLM judge, and then analyzes tokenization statistics (ATPR, ATPW, ABPT, ANSL) to relate tokenization efficiency to model scores. The authors report a consistent English-Bengali performance gap and claim an inverse relationship between tokenization efficiency and accuracy, alongside a release of the translated datasets and code.","tokens_in":12705,"tokens_out":4615,"duration_ms":50259,"significance":"If the tokenization-accuracy relationship were substantiated, it would provide a concrete, actionable insight into why underrepresented languages underperform in LLMs, and the released benchmarks could support future Bengali evaluation. However, the paper's central quantitative claim is not established by the current analysis, and the benchmark quality is limited by unvalidated machine translation and an untested LLM judge. The release of the datasets and code is nevertheless a useful contribution to a low-resource language community.","major_comments":[{"comment":"The claimed inverse relationship between tokenization efficiency and LLM accuracy is based on pooled correlations over model-dataset pairs from ten models of different sizes and families (Table 3) with no control for model identity, parameter count, or pretraining data. Since larger, higher-quality models may systematically have more Bengali-efficient tokenizers, the observed correlation could reflect model capability rather than a tokenization effect. No within-model, family-stratified, or partial correlation analysis is reported, and no significance tests accompany the correlations. The contribution bullet in Section 1 hedges this same finding as '(due to noise)', which is inconsistent with the causal phrasing in the abstract. This point must be re-analyzed with appropriate controls before the headline claim can be evaluated.","section":"Section 3.4, Figures 7 and 8"},{"comment":"The interpretation of ABPT is inverted relative to its formula. With r_i = B_i / l_i, a lower ABPT means each token spans fewer bytes, i.e., finer-grained, more fragmented tokenization, not 'coarser, more byte-efficient tokenization' as stated. The sentence 'lower ABPT means each token encodes more of the original text' is also backwards. Because Figures 9 and 10 are used to compare English and Bengali byte efficiency, this reversal directly contradicts the conclusion that English has higher ABPT and is therefore more byte-efficient. The definition and all downstream interpretations must be corrected.","section":"Section 3.4.4"},{"comment":"The claim in the contributions of releasing 'a comprehensive suite of high-quality Bengali benchmark datasets' is not supported by the evidence provided. The translation pipeline uses a single automatic model (GPT-4o-mini) and the paper explicitly states that translations 'were not manually validated.' The blind review in Section 2.3 compares three translation services without reporting any quantitative or qualitative scoring results. Similarly, the LLM-judge metric is untested against human annotations, despite being one of the three headline metrics. These gaps directly affect the validity of the performance numbers reported in Tables 4-6, so the benchmark quality claim and the numerical results must be substantially tempered or the data must be validated.","section":"Section 5 and Section 2.3"},{"comment":"The experimental setup is under-specified and internally inconsistent. The text states 'we selected all available open-source multilingual LLaMA models,' but Table 3 includes Qwen, Mistral, and DeepSeek models, and the promised list of specific models is missing from the text. The evaluation is also described as 'without finetuning the Llama family of models,' which does not cover the other families. Please clarify the model selection procedure and list the exact model checkpoints, inference settings (decoding parameters, temperature, number of samples), and the split used for each dataset.","section":"Section 3.1 and Table 3"}],"minor_comments":[{"comment":"The paragraph beginning 'This metric captures the average amount of raw text (in bytes) that each token represents' appears under the ATPW heading but describes ABPT; the discussion of bytes-per-token should be moved to Section 3.4.4.","section":"Section 3.4.3 and 3.4.4"},{"comment":"The abstract says the code is available on GitHub and the datasets on Hugging Face, but the full text gives no URLs; please include the actual repository links.","section":"Abstract and Section 6"},{"comment":"In the text, 'with bn and lang datasets ranging between 2-7 tokens per word' appears to contain a typo; 'lang' should likely be 'en' or 'English'.","section":"Section 3.4.5, Figure 6b"}],"recommendation":"major_revision","confidential_remarks":"The paper's main claim is not currently supported by the analysis, and the ABPT error is a clear internal contradiction. However, the dataset release and the broad evaluation effort could become a useful resource if the authors substantially revise the analysis (within-model or controlled correlations) and either validate a sample of translations or soften the 'high-quality' claim. The paper may be better positioned as a resource/benchmark paper rather than one making a causal tokenization claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is best read as a resource-generation effort. The authors translated eight established English benchmarks into Bengali, ran ten open-weight LLMs on them, and released the data and code. That is genuinely useful for a language that needs standardized eval sets. They also did a blind comparison of translation services and picked GPT-4o-mini, and the pipeline is described in enough detail to reproduce. The error analysis is a plus.\\n\\nThe analytical claims, though, are not supported. The headline inverse relationship between tokenization efficiency and accuracy is not established. Figures 7 and 8 pool ten models from 3B to 72B across four families with no control for size, family, pretraining data, or Bengali exposure. The pooled correlation could just be model quality. No within-model or family-stratified analysis is shown. The authors themselves hedge in the contributions bullet (\"due to noise\"), but the abstract states the correlation as a finding. That is a real gap.\\n\\nThere is also a mechanical error in the tokenizer section. The ABPT formula is bytes per token; the text says lower ABPT means coarser, more byte-efficient tokenization, which is backwards. Lower ABPT means each token covers fewer bytes, i.e., finer granularity. That reverses the interpretation of Figures 9 and 10.\\n\\nThe paper's own limitations concede both premises the scores depend on: the Bengali translations were not manually validated, and the LLM judge is untested. With no validation sample, the \"high-quality benchmark\" claim is unearned. The accuracy numbers may be a useful initial signal, but they are not reliable enough for fine-grained conclusions.\\n\\nCredit where due: the resource release, cost transparency (~$200), and reproducibility-oriented pipeline are real assets. The paper would benefit from a major revision: add a manual validation subsample, run significance tests and model-controlled partial correlations, fix the ABPT interpretation, and soften the abstract to match the design.\\n\\nWho is this for? Researchers building low-resource benchmarks and anyone needing a quick multilingual sanity check for Bengali model evaluation. The benchmark release is worth citing once on HuggingFace; the tokenization claim should not be cited in its current form.\\n\\nRecommendation: send to peer review, but expect major revision. The resource value and the testable empirical questions justify referee time; the overclaiming does not.","headline":"Useful Bengali benchmark release with an overclaimed tokenization finding; the resource is worth engaging, but the causal claim needs controls and the ABPT interpretation is backwards.","tokens_in":13274,"tokens_out":3457,"would_cite":true,"duration_ms":35820,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper sets out to show that LLMs perform worse in Bengali partly because Bengali text is over-tokenized, and that a new eight-dataset Bengali benchmark can measure the gap.","keywords":["Bengali NLP","LLM evaluation","tokenization efficiency","benchmark translation","multilingual LLMs","low-resource languages","Bengali benchmark","error analysis"],"falsifier":"A reader could settle the claim by taking one model family, comparing models controlled for size, or fine-tuning a single tokenizer on Bengali and re-running the eight benchmarks: if accuracy does not move with token efficiency once model identity is held fixed, the inverse relationship is a confound rather than a mechanism.","tokens_in":12278,"feed_emoji":"🧠","tokens_out":6209,"duration_ms":62431,"temperature":0.7,"pith_summary":"The paper tries to establish why open-weights LLMs answer Bengali prompts worse than English prompts, and to give the community a standard way to measure it. It translates eight established English benchmarks into Bengali, evaluates ten open models, and finds a consistent Bengali-English accuracy gap that is largest for small models and for the Mistral family, while DeepSeek models stay relatively stable. Its central new claim is an inverse relationship between tokenization efficiency and accuracy: Bengali inputs are split into far more tokens per word and per row than English, and models with more fragmented tokenization tend to score lower. If that claim holds, tokenizer design is a partly independent lever for improving low-resource language performance, not just a side effect of model quality.","feed_headline":"Bengali answers drop when tokenizers over-fragment text","feed_subtitle":"Ten open models scored on eight translated Bengali benchmarks; tokenizer granularity tracks the accuracy gap.","key_machinery":"The load-bearing object is the translated benchmark suite plus four tokenization metrics. The suite covers eight datasets (CommonsenseQA, OpenbookQA, BoolQ, Winogrande, HellaSwag, ARC, GSM8K-Main, MMLU) in Bengali, translated with a prompted LLM and post-processed into JSON. The metrics are average tokens per row ($\\mathrm{ATPR}$), average tokens per word ($\\mathrm{ATPW}$), average bytes per token ($\\mathrm{ABPT}$), and average normalized sequence length ($\\mathrm{ANSL}$); the last is the ratio of a tokenizer's sequence length to a baseline tokenizer's length for the same text. The paper correlates these metrics with its LLM-Judge score across ten models and reads the negative slope as evidence that over-tokenization hurts accuracy.","core_discovery":"On the paper's own terms, the discovery is that when the same benchmark questions are translated from English to Bengali, ten recent open LLMs lose accuracy consistently, and the size of the loss tracks how inefficiently the model's tokenizer encodes Bengali. Models like Mistral 7B and LLaMA 3.2 3B show large drops and high response-format error rates; larger LLaMA and Qwen models do better; DeepSeek-R1 models show the smallest cross-language score variation. The tokenization analysis reports that Bengali rows use dramatically more tokens than English rows, and that across all models higher average tokens per row and per word correlate with lower LLM-Judge scores. The paper presents this as evidence that excessive tokenization introduces noise and degrades accuracy, while concise per-word tokenization helps.","pith_inferences":["The paper stops at correlation; a causal test the authors did not run would hold tokenizer fixed while varying pretraining, or hold model fixed while swapping tokenizers, to see if accuracy follows token counts.","Because the Bengali translations were not manually validated, part of the measured gap could be flawed test items; human-checking a random sample would separate model failure from translation failure.","The bytes-per-token gap implies a cost corollary the authors only gesture at: if Bengali needs more tokens per word, then Bengali inference costs more compute per word, so tokenizer gains are also efficiency gains.","The same translation pipeline could be extended to other underrepresented scripts; the paper's inverse-relationship claim predicts the gap will be largest for scripts whose tokenizers fragment most."],"forward_implications":["If tokenization inefficiency degrades accuracy, then improving Bengali tokenizers could raise scores without changing model weights or pretraining data.","The released eight-dataset suite gives later work a fixed yardstick, so Bengali-specific models can be compared on the same translated questions.","The Mistral family's large gap and high response-error rates should be treated as a real weakness to diagnose rather than a benchmark artifact.","DeepSeek-R1's stable cross-language scores suggest some training or tokenization choices transfer well to Bengali and are worth copying.","Smaller models show the largest Bengali drop, so the paper predicts that scaling alone will not close the multilingual gap."],"supporting_citations":[{"why":"Defines the tokenization efficiency metrics (ATPR, ATPW, ABPT, ANSL) that the paper uses to measure Bengali over-tokenization.","marker":"Dagan et al., 2024"},{"why":"Supplies the LLaMA model family used in the evaluations and the white-paper dataset selection approach the paper follows.","marker":"Grattafiori et al., 2024"},{"why":"Introduces Mistral 7B, the model family the paper identifies as consistently underperforming in Bengali.","marker":"Jiang et al., 2023"},{"why":"Introduces DeepSeek-R1, the model family the paper finds most robust across English and Bengali.","marker":"Guo et al., 2025"},{"why":"Documents the Qwen2.5 models whose mixed English-Bengali scores appear in every comparison.","marker":"Qwen et al., 2025"},{"why":"Establishes the prior Bengali LLM evaluation context and motivates the need for standardized benchmarks.","marker":"Kabir et al., 2023"}],"fun_headline_variants":["Tokenizer bloat skews Bengali LLM scores","Bengali token cost predicts LLM accuracy drop","Efficient tokenization closes Bengali-English gap","Mistral and LLaMA stumble on Bengali token load","DeepSeek holds steady as Bengali tokenizer inflation hurts others"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline correlation is computed by pooling all ten models, which differ in size, family, and pretraining data; the claim that tokenizer inefficiency causes worse accuracy assumes those differences are not what actually produce both the token counts and the scores.","fun_headline_variants_meta":{"raw":{"variants":["Tokenizer bloat skews Bengali LLM scores","Bengali token cost predicts LLM accuracy drop","Efficient tokenization closes Bengali-English gap","Mistral and LLaMA stumble on Bengali token load","DeepSeek holds steady as Bengali tokenizer inflation hurts others"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1786,"prompt_tokens":929,"completion_tokens":857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":781}},"tokens_in":545,"tokens_out":857,"duration_ms":7516,"temperature":1.0,"reasoning_tokens":781,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:55:09.125354+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the claim by taking one model family, comparing models controlled for size, or fine-tuning a single tokenizer on Bengali and re-running the eight benchmarks: if accuracy does not move with token efficiency once model identity is held fixed, the inverse relationship is a confound rather than a mechanism.","supporting_citations":[],"review_version":1}