pith. sign in

arxiv: 2305.16307 · v3 · pith:RBBNZTDFnew · submitted 2023-05-25 · 💻 cs.CL

IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages

Jay Gala , Pranjal A. Chitale , Raghavan AK , Varun Gumma , Sumanth Doddapaneni , Aswanth Kumar , Janki Nawale , Anupama Sujatha
show 6 more authors
Ratish Puduppully Vivek Raghavan Pratyush Kumar Mitesh M. Khapra Raj Dabre Anoop Kunchukuttan
This is my paper
classification 💻 cs.CL
keywords languagesmodelsindiaworkparallelscheduledtranslationbenchmarks
0
0 comments X
read the original abstract

India has a rich linguistic landscape with languages from 4 major language families spoken by over a billion people. 22 of these languages are listed in the Constitution of India (referred to as scheduled languages) are the focus of this work. Given the linguistic diversity, high-quality and accessible Machine Translation (MT) systems are essential in a country like India. Prior to this work, there was (i) no parallel training data spanning all 22 languages, (ii) no robust benchmarks covering all these languages and containing content relevant to India, and (iii) no existing translation models which support all the 22 scheduled languages of India. In this work, we aim to address this gap by focusing on the missing pieces required for enabling wide, easy, and open access to good machine translation systems for all 22 scheduled Indian languages. We identify four key areas of improvement: curating and creating larger training datasets, creating diverse and high-quality benchmarks, training multilingual models, and releasing models with open access. Our first contribution is the release of the Bharat Parallel Corpus Collection (BPCC), the largest publicly available parallel corpora for Indic languages. BPCC contains a total of 230M bitext pairs, of which a total of 126M were newly added, including 644K manually translated sentence pairs created as part of this work. Our second contribution is the release of the first n-way parallel benchmark covering all 22 Indian languages, featuring diverse domains, Indian-origin content, and source-original test sets. Next, we present IndicTrans2, the first model to support all 22 languages, surpassing existing models on multiple existing and new benchmarks created as a part of this work. Lastly, to promote accessibility and collaboration, we release our models and associated data with permissive licenses at https://github.com/AI4Bharat/IndicTrans2.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

    cs.CL 2026-05 unverdicted novelty 7.0

    BrahmicTokenizer-131K is a 131K-vocab tokenizer constructed via script-prune crop and linear-programming retrofit to o200k_base, achieving 26.7% fewer tokens on Indic text while matching o200k_base on English fertilit...

  2. MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

    cs.CL 2026-04 unverdicted novelty 7.0

    MORPHOGEN is a new multilingual benchmark for testing LLMs on gender-aware morphological generation via rewriting first-person sentences to the opposite gender in French, Arabic, and Hindi.

  3. BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources

    cs.CL 2026-04 unverdicted novelty 7.0

    A unified survey that consolidates Indian NLP resources by task, language, domain, and modality while identifying gaps in coverage and generalization.

  4. SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR

    cs.CL 2026-05 unverdicted novelty 6.0

    SCRIBE is a new diagnostic evaluation framework for Indic ASR that provides categorical error rates via sandhi-tolerant alignment and domain vocabulary injection, with released models and human-validated alignment to ...

  5. SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR

    cs.CL 2026-05 unverdicted novelty 5.0

    New categorical error decomposition framework SCRIBE for Indic ASR evaluation, with released rich transcription models for Hindi, Malayalam, and Kannada.

  6. Structure-Preserving Document Translation via Multi-Stage LLM Pipeline: A Case Study in Marathi

    cs.CL 2026-06 unverdicted novelty 4.0

    A multi-stage LLM pipeline for structure-preserving Marathi-to-English translation of government PDFs using layout-aware OCR and HTML reconstruction.

  7. Neural Machine Translation for Low-Resource Tangkhul--English

    cs.CL 2026-06 unverdicted novelty 4.0

    ByT5-large fine-tuned on 38k Tangkhul-English pairs achieves BLEU 39.97, chrF++ 58.07, BERTScore 0.8104 and COMET 0.7302 on a 3.8k-sentence held-out test set.

  8. AgriGov: A Structured Multilingual Dataset Curation for Indian Government Schemes for Farmers

    cs.CL 2026-06 unverdicted novelty 4.0

    AgriGov is a new structured trilingual dataset of ~8000 sentence pairs from 50 Indian farmer schemes, created via scraping, MT pipeline, and corpus augmentation for NLP tasks.