REVIEW 2 cited by
Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Training of a tokenizer plays an important role in the performance of deep learning models. This research aims to understand the performance of tokenizers in five state-of-the-art (SOTA) large language models (LLMs) in the Assamese language of India. The research is important to understand the multi-lingual support for a low-resourced language such as Assamese. Our research reveals that the tokenizer of SUTRA from Two AI performs the best with an average Normalized Sequence Length (NSL) value of 0.45, closely followed by the tokenizer of GPT-4o from Open AI with an average NSL value of 0.54, followed by Gemma 2, Meta Llama 3.1, and Mistral Large Instruct 2407 with an average NSL value of 0.82, 1.4, and 1.48 respectively.
Forward citations
Cited by 2 Pith papers
-
A Breadth-First Catalog of Text Processing, Speech Processing and Multimodal Research in South Asian Languages
A breadth-first survey of recent South Asian language NLP, speech, and multimodal research, organized with LLM-based classification and clustering.
-
Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
A single-sentence evaluation claims SUTRA's tokenizer is most token-efficient across 14 of India's 22 official languages, based on one example text per language.
Discussion (0). Continue with ORCID to comment.