A single-sentence evaluation claims SUTRA's tokenizer is most token-efficient across 14 of India's 22 official languages, based on one example text per language.
Future applications of generative large language models: A data-driven case study on ChatGPT,
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2024 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
A single-sentence evaluation claims SUTRA's tokenizer is most token-efficient across 14 of India's 22 official languages, based on one example text per language.