Pith. sign in

REVIEW 4 major objections 3 minor 53 references

TASE: Token Awareness and Structured Evaluation for Multilingual Language Models

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TASE benchmark shows humans far ahead of LLMs on token-level tasks

desk verdict A plausible multilingual token-level benchmark with a real human–LLM gap, but the abstract alone can't verify the tasks actually measure token awareness; deserves peer review if the full paper addresses construct validity. read the letter →

arxiv 2508.05468 v1 pith:FO6AXAZL submitted 2025-08-07 cs.CL

Chenzhuo Zhao , Xinda Wang , Yue Huang , Junting Lu , Ziqian Liu This is my paper
classification cs.CL
keywords TASEtokenawarenessstructuralunderstandingbenchmarkmultilingualevaluationtoken-levelreasoningLLMsyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TASE, a 35,927-instance benchmark of 10 tasks in Chinese, English, and Korean designed to measure token-level awareness and structural understanding. After evaluating over 30 leading LLMs, the authors report that human performance significantly outpaces every model tested, even those that excel at high-level semantic tasks. This suggests a persistent blind spot in current models: precise control over individual tokens and grammatical structure does not come automatically from training on broad text. The benchmark also includes a synthetic data generation pipeline, and a Qwen2.5-14B model trained with GRPO shows that the tasks can be used to drive improvement.

What carries the argument

The central object is the TASE benchmark itself: a 35,927-instance evaluation set organized into ten tasks under 'token awareness' and 'structural understanding,' built using a scalable synthetic data generation pipeline. The benchmark is the instrument that carries the argument—it defines what 'token-level awareness' means operationally and measures the gap between humans and models. Additionally, the GRPO-trained Qwen2.5-14B model shows the pipeline can be used for targeted training, making TASE not just a test but a source of training signal.

What would settle it

Give a strong LLM the same 35,927 TASE instances but with input converted to individual characters or sub-token units (for example, Korean text split into jamo, or English text space-separated into single characters) and ask it to answer without additional training. If its scores jump to near-human levels on character counting and token alignment while human scores stay the same, the claimed 'token-level weakness' is an artifact of tokenizer granularity rather than a fundamental reasoning deficiency. If the gap persists, the paper's interpretation holds.

Watch

Extended reading notes

Core claim

The central claim is that large language models, despite their strong performance on semantic reasoning, systematically fail at fine-grained token-level and structural tasks. TASE operationalizes this claim with ten tasks in two categories—token awareness and structural understanding—covering character counting, token alignment, syntactic parsing, and length constraint satisfaction across three languages. The reported human-model gap is presented as evidence of a real deficiency in token-level reasoning, not an artifact of task difficulty. The paper further demonstrates that a custom model trained on TASE's synthetic pipeline can make progress, indicating the benchmark can serve as both a di

Load-bearing premise

The load-bearing premise is that TASE's ten tasks genuinely measure token-level awareness and structural understanding in a fair, cross-lingually comparable way that cannot be solved by memorization or surface heuristics.

Editorial extensions

If this is right

  • Applications that require exact token-level constraints, such as code generation with strict line limits or formatted data output, will likely remain unreliable even as semantic capabilities improve.
  • Cross-lingual claims about model competence must be qualified: passing English token tasks does not imply passing Chinese or Korean ones, so evaluation needs to be language-specific.
  • The synthetic data pipeline offers a concrete path to improve token-level skills, meaning these weaknesses may be trainable rather than intrinsic.
  • TASE can function as a diagnostic for future models, giving developers a way to track progress on low-level language understanding separate from high-level benchmark scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported aggregate gap may hide large variation across the ten tasks; a natural extension is to analyze which tasks and which languages contribute the most to the human-model difference.
  • Because the tasks are generated synthetically, models could exploit generator-specific patterns rather than learning general token awareness; this hypothesis is testable by varying the generator's randomization and checking score stability.
  • The human-model gap on character counting and alignment tasks may partly reflect tokenizer granularity rather than reasoning limits—a model given byte-level or character-segmented input could plausibly close the gap, which would reframe the finding as an interface issue.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces TASE, a multilingual benchmark for evaluating LLMs' token-level awareness and structural understanding. TASE comprises 10 tasks in two categories across Chinese, English, and Korean, with 35,927 evaluation instances and a synthetic data generation pipeline used to train a Qwen2.5-14B model via GRPO. The authors evaluate more than 30 LLMs and report that human performance significantly exceeds current LLM performance, underscoring persistent weaknesses in token-level reasoning. The code and dataset are said to be publicly available. The present submission, however, contains only the abstract; no methods, results, or appendices are supplied for review.

Significance. If the central claim holds, TASE would be a useful diagnostic resource for a class of linguistic tasks on which LLMs underperform relative to humans, complementing existing high-level semantic benchmarks. The explicit cross-lingual design (Chinese, English, Korean) and the public release of data and code are commendable. The benchmark could inform future work on tokenization, structural reasoning, and cross-lingual generalization. The significance is, however, contingent on the construct validity of the tasks and the comparability of the human baseline; neither can be assessed from the abstract alone.

major comments (4)
  1. [Abstract (benchmark description)] The central construct 'token awareness' is never defined. The listed tasks mix character-level counting, token alignment, and syntactic structure parsing; these may measure different abilities. In English, character counting is not a token-level task under BPE tokenization, and token alignment requires a token definition that differs across Chinese, English, and Korean. The abstract provides no task instructions, no examples, and no evidence that the tasks are not answerable via surface heuristics (e.g., Unicode code point counts or public tokenizer ID matching). These details are load-bearing for the headline human--LLM gap.
  2. [Abstract (human baseline)] The claim that 'human performance significantly outpaces' LLMs requires a rigorously collected, comparable human baseline. The abstract reports no protocol: number of participants, language backgrounds, time limits, display format, whether participants had access to token definitions, and how multiple judgments were adjudicated. If humans were given unlimited time and full access to the underlying strings while models received a single greedy generation, the gap could be an artifact of effort or prompting rather than a real cognitive difference. Without this information, the headline comparison is uninterpretable.
  3. [Abstract (evaluation protocol and leakage)] The abstract reports results for 'over 30 leading commercial and open-source LLMs' but no evaluation details: sampling temperature, number of runs, prompt template, answer extraction, or handling of non-determinism. The reported Qwen2.5-14B model was trained on a synthetic data pipeline, and if the 35,927-instance evaluation set is generated from the same pipeline, the trained model's evaluation scores may be partly circular. The abstract does not describe any holdout procedure or distributional separation between training and evaluation sets. A detailed contamination and leakage analysis is required before the central claim can be credited.
  4. [Full text] The provided manuscript consists only of the abstract. There are no methods, equations, tables, figures, or appendices, and no link to a paper PDF or supplementary material is given beyond a GitHub URL. Consequently, all load-bearing premises—task construction, label quality, cross-lingual comparability, model evaluation, and statistical significance—are unsupported in the reviewed artifact. This is not a request for stylistic polishing; it is a claim that the experimental evidence is not available for assessment.
minor comments (3)
  1. [Abstract] The model name 'O3' should likely be written 'o3' to match common convention; consider consistency for 'Claude 4' and 'Gemini 2.5 Pro'.
  2. [Abstract] The abbreviation TASE is not expanded; please provide the full name or a brief etymology.
  3. [General] No references are included in the provided text. Since the paper positions itself against existing LLM benchmarks, a related-work section with citations is expected in the full manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the human–LLM gap is an external measurement, and the abstract separates the evaluation set from the training pipeline.

full rationale

The paper's central claim is an empirical result: human performance significantly outpaces current LLMs on the TASE benchmark. This claim is not derived from any fitted parameter or from the benchmark's construction in a circular way. The abstract explicitly distinguishes the 35,927-instance evaluation set from the synthetic data generation pipeline used for training, so the evaluation is not used to fit model weights, and the human baseline is an independent measurement. There are no self-citations, no imported uniqueness theorems, and no equations that reduce a predicted quantity to an input by definition. The only potential concern is construct validity—whether the tasks truly measure token-level reasoning—which is a question of experimental design, not circularity. Hence, no circular steps are identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

Abstract-only review. The central claim rests on the benchmark's integrity and the evaluation protocol, none of which can be inspected. No fitted numerical parameters are visible from the abstract; the benchmark's design choices (task difficulty, rubric, model prompt settings) are unenumerated. TASE is an evaluation instrument, not a postulated physical entity, so the invented-entities ledger is empty.

assumptions (3)
  • domain assumption The 35,927 benchmark instances are correctly labeled and unambiguous.
    The abstract asserts the evaluation set size but gives no annotation or quality-control details; if labels are wrong, scores are meaningless.
  • domain assumption Model evaluation was fair and comparable across all 30+ models (prompting, decoding, scoring).
    Cross-model comparisons of this kind are sensitive to prompt templates and sampling settings; the abstract does not describe the protocol.
  • domain assumption The human performance baseline was collected under conditions comparable to the model evaluation.
    The headline claim is a human-model gap; without a fair human baseline the comparison is uninterpretable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TASE: Token Awareness and Structured Evaluation for Multilingual Language Models." pith.science (2026). https://pith.science/paper/FO6AXAZL

@misc{pith2026250805468,
  author       = {Pith},
  title        = {Pith review of: TASE: Token Awareness and Structured Evaluation for Multilingual Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FO6AXAZL}},
  note         = {Machine review of arXiv:2508.05468}
}
read the original abstract

While large language models (LLMs) have demonstrated remarkable performance on high-level semantic tasks, they often struggle with fine-grained, token-level understanding and structural reasoning--capabilities that are essential for applications requiring precision and control. We introduce TASE, a comprehensive benchmark designed to evaluate LLMs' ability to perceive and reason about token-level information across languages. TASE covers 10 tasks under two core categories: token awareness and structural understanding, spanning Chinese, English, and Korean, with a 35,927-instance evaluation set and a scalable synthetic data generation pipeline for training. Tasks include character counting, token alignment, syntactic structure parsing, and length constraint satisfaction. We evaluate over 30 leading commercial and open-source LLMs, including O3, Claude 4, Gemini 2.5 Pro, and DeepSeek-R1, and train a custom Qwen2.5-14B model using the GRPO training method. Results show that human performance significantly outpaces current LLMs, revealing persistent weaknesses in token-level reasoning. TASE sheds light on these limitations and provides a new diagnostic lens for future improvements in low-level language understanding and cross-lingual generalization. Our code and dataset are publicly available at https://github.com/cyzcz/Tase .

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 17 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    AI, .; :; Young, A.; Chen, B.; Li, C.; Huang, C.; Zhang, G.; Zhang, G.; Wang, G.; Li, H.; Zhu, J.; Chen, J.; Chang, J.; Yu, K.; Liu, P.; Liu, Q.; Yue, S.; Yang, S.; Yang, S.; Xie, W.; Huang, W.; Hu, X.; Ren, X.; Niu, X.; Nie, P.; Li, Y.; Xu, Y.; Liu, Y.; Wang, Y.; Cai, Y.; Gu, Z.; Liu, Z.; and Dai, Z. 2025. Yi: Open Foundation Models by 01.AI. arXiv:2403.04652

    arXiv 2025

  5. [5]

    D.; and Stuart, A

    Bhattacharya, B.; Gibbons, J. D.; and Stuart, A. 2002. Median of the p Value Under the Alternative Hypothesis. The American Statistician, 56(3): 225--230. Discusses behavior/distribution of p -values under both null and alternative

    work page 2002

  6. [6]

    Boucher, N.; Shumailov, I.; Anderson, R.; and Papernot, N. 2022. Bad characters: Imperceptible nlp attacks. In 2022 IEEE Symposium on Security and Privacy (SP), 1987--2004. IEEE

    work page 2022

  7. [7]

    H.; Choi, E.; Collins, M.; Garrette, D.; Kwiatkowski, T.; Nikolaev, V.; and Palomaki, J

    Clark, J. H.; Choi, E.; Collins, M.; Garrette, D.; Kwiatkowski, T.; Nikolaev, V.; and Palomaki, J. 2020. Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics, 8: 454--470

    work page 2020

  8. [8]

    Comanici, G.; Bieber, E.; Schaekermann, M.; Pasupat, I.; Sachdeva, N.; Dhillon, I.; Blistein, M.; Ram, O.; Zhang, D.; Rosen, E.; et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261

    arXiv 2025

Show all 53 references
  1. [9]

    R.; Schwenk, H.; and Stoyanov, V

    Conneau, A.; Lample, G.; Rinott, R.; Williams, A.; Bowman, S. R.; Schwenk, H.; and Stoyanov, V. 2018. XNLI: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053

    2018 arXiv

  2. [10]

    Davies, M. 2010. The Corpus of Contemporary American English as the first reliable monitor corpus of English. Literary and linguistic computing, 25(4): 447--464

    2010

  3. [11]

    Edman, L.; Schmid, H.; and Fraser, A. 2024. CUTE: Measuring LLMs' Understanding of Their Tokens. arXiv preprint arXiv:2409.15452

    2024 arXiv

  4. [12]

    Efrat, A.; Honovich, O.; and Levy, O. 2022. Lmentry: A language model benchmark of elementary language tasks. arXiv preprint arXiv:2211.02069

    2022 arXiv

  5. [13]

    Fenogenova, A.; Chervyakov, A.; Martynov, N.; Kozlova, A.; Tikhonova, M.; Akhmetgareeva, A.; Emelyanov, A.; Shevelev, D.; Lebedev, P.; Sinev, L.; et al. 2024. MERA: A comprehensive LLM evaluation in Russian. arXiv preprint arXiv:2401.04531

    2024 arXiv

  6. [14]

    Fu, T.; Ferrando, R.; Conde, J.; Arriaga, C.; and Reviriego, P. 2023. Why Do Large Language Models (LLMs) Struggle to Count Letters? CoRR, abs/2412.18626

    2023 arXiv

  7. [15]

    Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Hiraoka, T.; and Inui, K. 2025. Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters. arXiv:2506.10641

    2025 arXiv

  9. [17]

    T.; Liang, W.; Sun, F.-Y.; and Haber, N

    Hua, T.; Hua, H.; Xiang, V.; Klieger, B.; Truong, S. T.; Liang, W.; Sun, F.-Y.; and Haber, N. 2025. ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code. arXiv:2506.02314

    2025 arXiv

  10. [18]

    Huang, J.; Zhao, Q.; and Wu, Y. 2013. Design of embedded multi-language lattice font based on QT. In Proceedings 2013 International Conference on Mechatronic Sciences, Electric Engineering and Computer (MEC), 2252--2255. IEEE

    2013

  11. [19]

    Y.; and You, Y

    Jia, Q.; Yue, X.; Huang, S.; Qin, Z.; Liu, Y.; Lin, B. Y.; and You, Y. 2024. Visual perception in text strings. arXiv preprint arXiv:2410.01733

    2024

  12. [20]

    Kim, K. 2002. New canonical decomposition and composition processes for Hangeul. Computer Standards & Interfaces, 24(1): 69--82

    2002

  13. [21]

    S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

    Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2023. Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916

    2023 arXiv

  14. [22]

    Kostikova, A.; Wang, Z.; Bajri, D.; Pütz, O.; Paaßen, B.; and Eger, S. 2025. LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models. arXiv:2505.19240

    2025

  15. [23]

    D.; Van Nguyen, C.; Ngo, N

    Lai, V. D.; Van Nguyen, C.; Ngo, N. T.; Nguyen, T.; Dernoncourt, F.; Rossi, R. A.; and Nguyen, T. H. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv preprint arXiv:2307.16039

    2023 arXiv

  16. [24]

    Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212

    2023 arXiv

  17. [25]

    Li, J.; and Zhou, J. 2007. Chinese character structure analysis based on complex networks. Physica A: Statistical Mechanics and its Applications, 380: 629--638

    2007

  18. [26]

    Liang, Y.; Duan, N.; Gong, Y.; Wu, N.; Guo, F.; Qi, W.; Gong, M.; Shou, L.; Jiang, D.; Cao, G.; et al. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint arXiv:2004.01401

    2020 arXiv

  19. [27]

    Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  20. [28]

    Liu, X.; Cheng, K.; Luo, Y.; Duh, K.; and Matsumoto, Y. 2013. A hybrid Chinese spelling correction using language model and statistical machine translation with reranking. In Proceedings of the seventh SIGHAN workshop on chinese language processing, 54--58

    2013

  21. [29]

    Ma, W.; Cui, Y.; Si, C.; Liu, T.; Wang, S.; and Hu, G. 2020. CharBERT: Character-aware pre-trained language model. arXiv preprint arXiv:2011.01513

    2020 arXiv

  22. [30]

    Pearson, K. 1895. Note on Regression and Inheritance in the Case of Two Parents. Proceedings of the Royal Society of London, 58(347-352): 240--242

  23. [31]

    J.; and Green, T

    Pepicello, W. J.; and Green, T. A. 1984. Language of riddles: new perspectives. The Ohio State University Press

    1984

  24. [32]

    Qwen; :; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin,...

    2025 arXiv

  25. [33]

    Ren, D.; Cai, Y.; Li, W.; Xia, R.; Li, Z.; and Li, Q. 2019. Solving chinese character puzzles based on character strokes. In CCF International Conference on Natural Language Processing and Chinese Computing, 303--313. Springer

    2019

  26. [34]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  27. [35]

    Shibata, Y.; Kida, T.; Fukamachi, S.; Takeda, M.; Shinohara, A.; Shinohara, T.; and Arikawa, S. 1999. Byte pair encoding: A text compression scheme that accelerates pattern matching

    1999

  28. [36]

    I.; Ngui, J

    Singh, S.; Romanou, A.; Fourrier, C.; Adelani, D. I.; Ngui, J. G.; Vila-Suero, D.; Limkonchotiwat, P.; Marchisio, K.; Leong, W. Q.; Susanto, Y.; et al. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. arXiv preprint arX...

    2024 arXiv

  29. [37]

    A.; Abid, A.; Fisch, A.; Brown, A

    Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research

    2023

  30. [38]

    Student. 1908. The probable error of a mean. Biometrika, 1--25

    1908

  31. [39]

    W.; Chowdhery, A.; Le, Q

    Suzgun, M.; Scales, N.; Sch \"a rli, N.; Gehrmann, S.; Tay, Y.; Chung, H. W.; Chowdhery, A.; Le, Q. V.; Chi, E. H.; Zhou, D.; et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  32. [40]

    Wan, Z.; Wang, X.; Liu, C.; Alam, S.; Zheng, Y.; Liu, J.; Qu, Z.; Yan, S.; Zhu, Y.; Zhang, Q.; Chowdhury, M.; and Zhang, M. 2024. Efficient Large Language Models: A Survey. arXiv:2312.03863

    2024 arXiv

  33. [41]

    Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32

    2019

  34. [42]

    Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  35. [43]

    Wang, B.; Liu, Z.; Huang, X.; Jiao, F.; Ding, Y.; Aw, A.; and Chen, N. F. 2023. Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. arXiv preprint arXiv:2309.04766

    2023 arXiv

  36. [44]

    Wang, D.; Li, Y.; Jiang, J.; Ding, Z.; Luo, Z.; Jiang, G.; Liang, J.; and Yang, D. 2025. Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization. arXiv:2405.17067

    2025 arXiv

  37. [45]

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903

    2023 arXiv

  38. [46]

    J.; and Matsuura, K

    Willmott, C. J.; and Matsuura, K. 2005. Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate research, 30(1): 79--82

    2005

  39. [47]

    Xu, N.; and Ma, X. 2025. LLM The Genius Paradox: A Linguistic and Math Expert’s Struggle with Simple Word-based Counting Problems. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 3344--3370. Orig...

    2025 arXiv

  40. [48]

    Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  41. [49]

    Yehudai, G.; Kaplan, H.; Ghandeharioun, A.; Geva, M.; and Globerson, A. 2024. When Can Transformers Count to n? arXiv preprint arXiv:2407.15160

    2024

  42. [50]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  43. [51]

    Zhang, X.; Li, C.; Zong, Y.; Ying, Z.; He, L.; and Qiu, X. 2023. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474

    2023 arXiv

  44. [52]

    Zhang, Y.; and He, Z. 2024. Large Language Models Can Not Perform Well in Understanding and Manipulating Natural Language at Both Character and Word Levels? In Findings of the Association for Computational Linguistics: EMNLP 2024, 11826--11842

    2024

  45. [53]

    Zhou, J.; Lu, T.; Mishra, S.; Brahma, S.; Basu, S.; Luan, Y.; Zhou, D.; and Hou, L. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.