Pith. sign in

REVIEW 6 major objections 6 minor 38 references

IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding

T0 review · 6 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IndicMMLU-Pro translates the MMLU-Pro benchmark into nine Indic languages, reports that GPT-4o is the best of thirteen tested models with top accuracy of 44.8% in Hindi, and argues that back-translation checks and expert proofreading…

desk verdict A genuinely useful dataset release buried under an overclaimed validation story: the benchmark deserves referee time, but only with the missing QA evidence and the acknowledged translation errors addressed. read the letter →

arxiv 2501.15747 v2 pith:XSJOPABR submitted 2025-01-27 cs.CL cs.AI

Sankalp KJ , Ashutosh Kumar , Laxmaan Balaji , Nikunj Kotecha , Vinija Jain , Aman Chadha , Sreyoshi Bhaduri This is my paper · ORCID
classification cs.CLcs.AI
keywords IndiclanguagesmultilingualbenchmarkMMLU-Promachinetranslationlargelanguagemodelsevaluationnaturalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IndicMMLU-Pro is a benchmark that takes the English MMLU-Pro multiple-choice dataset and translates it into nine Indian languages — Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu — using the IndicTrans2 machine translation model. The paper argues that with back-translation quality checks, expert proofreading, and embedding-based similarity scores, the translated dataset preserves the meaning and difficulty of the original while adapting to Indian linguistic and cultural contexts. It then reports baseline accuracy for thirteen multilingual and Indic-specific models, finding that GPT-4o leads every language with scores between 38.46% and 44.80%, and that most other models stay near chance level. If the benchmark is sound, it gives researchers a standardized, publicly available way to measure progress in Indic language understanding.

What carries the argument

The load-bearing object is the translation-and-back-translation validation loop. IndicTrans2 produces the nine Indic datasets from English MMLU-Pro; quality is asserted by translating a sample back to English and thresholding five automatic metrics (chrF++ above 0.5, BLEU/SacreBLEU 25–30, METEOR 0.5–0.6, TER below 0.4–0.5), then scoring 9,000 sentence pairs by 13 expert reviewers on semantic accuracy, fluency, and stylistic appropriateness, and finally measuring LaBSE embedding cosine similarity between languages. This loop is what the paper relies on to claim the dataset maintains the integrity of the original MMLU-Pro.

What would settle it

Take a random sample of questions from Bengali, Punjabi, Kannada, Telugu, Urdu, and Marathi, have native speakers independently identify the intended answer from the translated options, and compare with the English key; if a substantial fraction change or become unanswerable — as the printed Tamil example already shows 'embryological origin' becoming 'thematic appearance' — the dataset's integrity claim fails for those languages.

Watch

Extended reading notes

Core claim

The central claim is that a high-quality, publicly available Indic-language equivalent of MMLU-Pro can be built by machine translation rather than by native human authoring. The paper shows the full pipeline: MMLU-Pro is translated by IndicTrans2 into nine languages, a subset is back-translated to English and scored with chrF++, BLEU, METEOR, TER, and SacreBLEU, thirteen native experts proofread 9,000 sentence pairs on meaning, fluency, and style, and LaBSE cosine similarity between Indic and English items is used to confirm semantic alignment. On the resulting benchmark, GPT-4o is the strongest model in all nine languages, with Hindi the highest at 44.80% accuracy, while older multilingual encoders cluster near 10–13%. The paper interprets these numbers as baseline results and a performance tier structure for future Indic LLM work.

Load-bearing premise

The benchmark's value rests on the assumption that IndicTrans2's translations into all nine languages preserve each question's meaning, difficulty, and correct answer; the paper directly verifies this for only Hindi, Gujarati, and Tamil.

Editorial extensions

If this is right

  • GPT-4o's 38.46–44.80% range across the nine languages becomes the baseline that future Indic LLMs must beat.
  • The large gap between GPT-4o and the 10–13% scores of IndicBERT, MuRIL, XLM-RoBERTa, and similar models quantifies how far current open and Indic-specific models lag on multitask understanding.
  • Because the dataset preserves MMLU-Pro's 14 categories and structure, model rankings on IndicMMLU-Pro can be compared directly with English MMLU-Pro rankings to isolate language-specific degradation.
  • The public Hugging Face release lets any team reproduce baselines or add new models without building a translation pipeline.
  • The absence of complete quality metrics for six of the nine languages means a full-dataset quality statement will require the promised future publication of those numbers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports quality metrics for only three of nine languages, so a natural extension is to back-translate and expert-check the other six; until then, statements about the whole dataset's quality are extrapolations.
  • Because translation errors can change the correct option, cross-language accuracy differences (for example, Tamil's lower scores) may reflect translation artifacts as much as model capability — a confound the benchmark does not decompose.
  • A diagnostic extension would pair English MMLU-Pro with IndicMMLU-Pro for the same models: the per-category score drop in each language would map where translation or reasoning fails.
  • The dataset could also serve as a stress test for machine translation quality itself, since answer-key consistency across languages is a measurable, high-stakes signal of semantic preservation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces IndicMMLU-Pro, a nine-language (Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, Urdu) benchmark created by translating the English MMLU-Pro dataset with IndicTrans2 while retaining the original multiple-choice structure and inherited gold labels. It describes a quality-assurance pipeline based on back-translation metrics (chrF++, BLEU, METEOR, TER, SacreBLEU), LaBSE cosine similarity, and an expert proofreading exercise, and it reports zero-shot baseline accuracy for a range of multilingual and Indic-specific models. The central claim is that this systematic pipeline ensures a high-quality multilingual benchmark preserving MMLU-Pro's integrity.

Significance. If the translation-fidelity claim were fully supported, this would be a useful public benchmark filling a clear gap for Indic-language evaluation, with broad model coverage and a plausible template for adapting MMLU-Pro. The authors make the dataset publicly available on Hugging Face, describe their process transparently, and include a wider range of baseline models than many prior Indic benchmark papers. However, as submitted, the load-bearing validation is incomplete: reported QA metrics cover only three of nine languages, expert proofreading scores are never presented, and the paper's own examples contain meaning-changing translation errors. The current version therefore does not yet establish the benchmark's validity for the comprehensive claims made in the title, abstract, and Figure 1 caption.

major comments (6)
  1. [§3.3, Table 3] The quality assessment that underpins the 'high-quality benchmark' claim is reported for only three of nine languages. The text explicitly states that metrics for Bengali, Punjabi, Kannada, Telugu, Urdu, and Marathi are missing and 'being compiled.' Because the dataset is released for all nine languages and the abstract and Figure 1 caption assert comprehensive coverage, the central claim is not supported for six languages. Complete per-language metrics and, ideally, per-item pass/fail counts under the stated thresholds are needed.
  2. [§2.1, §3.3] The back-translation QA is circular with respect to the translation model: forward translation and back-translation both use IndicTrans2. Any systematic translation errors that IndicTrans2 makes in both directions will be invisible to chrF++, BLEU, METEOR, TER, and SacreBLEU. The independent check described in §2.6 (expert proofreading of 9,000 sentence pairs by 13 reviewers) would break this circularity, but no scores, no per-language results, and no inter-annotator agreement are reported. As a result, the manuscript does not currently provide the independent evidence needed to support the claim that meaning and answer labels are preserved.
  3. [Figure 4; Figures 8–9] The paper's own published examples contradict the semantic-fidelity claim. Figure 4 translates 'embryological origin' into Tamil as 'thematic appearance,' and Figures 8 and 9 render 'pupil of your eye' as 'eyelid' in both Gujarati and Tamil. These are referent-changing errors rather than stylistic shifts; for a multiple-choice benchmark whose gold labels are inherited from the English source, such errors can invalidate individual items. The authors need to audit the released dataset for these and similar errors and either correct the translations or explicitly remove/repair the affected items; aggregate back-translation scores do not address the validity of individual items.
  4. [Table 4] The LaBSE cosine similarities (~0.91–0.93) are too coarse to establish meaning preservation. A sentence about 'thematic appearance' can be close in embedding space to one about 'embryological origin,' particularly when the surrounding context is similar, so these scores do not detect the referent-changing errors shown in Figures 4, 8, and 9. The table should be presented as a coarse supplementary signal, not as evidence that answer-carrying content is preserved.
  5. [§7.2, Tables 1–2] The baseline protocol for MuRIL and XLM-RoBERTa uses a non-standard pairwise sequence-classification scoring heuristic (question paired with each option separately, then argmax over the resulting scores), which the appendix itself concedes is 'not the standard way' and 'may not be as accurate or reliable' as a properly fine-tuned multiple-choice model; IndicBART is scored with a generation-length heuristic. Because §3.2 and §5 draw tier conclusions and cross-model comparisons from these numbers, the baselines should either be replaced by a consistent fine-tuned multiple-choice protocol or be explicitly labeled as heuristic scores and excluded from comparative claims such as 'XLM-RoBERTa significantly outperforms other models.'
  6. [Figure 1, §2.2] Figure 1 and §2.2 state that 'only translations meeting quality thresholds proceed to the final dataset,' but the manuscript reports no information about how many items failed the thresholds in each language, whether thresholds were applied per item or per aggregate corpus, or how rejected items were handled. Without this information, the dataset construction process is not reproducible, and the claim that the thresholds enforce quality cannot be verified.
minor comments (6)
  1. [§2.2, §3.3] The threshold lists give BLEU as '>25-30%' and METEOR as '50-60%,' but the reported scores are on a 0–1 scale (0.54–0.59 BLEU, 0.52–0.56 METEOR); the units should be harmonized to avoid ambiguity.
  2. [Table 2 caption] The caption contains the misspellings 'Dravidan' and 'Telegu,' and the table does not explain why certain model–language combinations are marked N/A.
  3. [§1, §2] The introduction lists 'task taxonomy' as a contribution, but no explicit task taxonomy section or enumeration is provided in the paper; either add it or soften the claim.
  4. [§3.3] The text says that data for the other six languages 'is missing from Table 2,' but the intended reference is Table 3 (the back-translation metrics table), not Table 2 (model accuracy).
  5. [Figure 6] The axis label '#' is ambiguous; clarifying that it represents a count would improve readability.
  6. [§2.5, Tables 1–2] The model list in §2.5 includes Airavata, OpenHathi, TamilLlama, and MahaMarathi, but Tables 1–2 report results for only a subset of these models; the availability or applicability conditions for each model should be stated in one place.

Circularity Check

1 steps flagged · score 5.0 of 10

Translation QA is self-referential: back-translation uses the same IndicTrans2 model that generated the benchmark, leaving the central 'high-quality / maintains integrity' claim without independent support.

  1. other [Section 2.2 (Quality Assurance) and Section 3.3 (Dataset Quality Assessment); Figure 1 caption]
    "In order to heuristically assert the quality of Indic language datasets, we convert the dataset from the respective Indic language back to English again by leveraging IndicTrans2 as in Section 2.1."

    Section 2.1 creates IndicMMLU-Pro by translating MMLU-Pro with IndicTrans2; Section 3.3 then asserts quality by converting those Indic questions back to English with the same IndicTrans2 and scoring similarity to the original. The reported chrF++/BLEU/METEOR/TER/SacreBLEU values therefore measure round-trip consistency inside one fixed translator, not whether the Indic text preserves the English meaning or the inherited gold answer. Systematic errors that persist through both passes cancel and are invisible to the metrics.

full rationale

The derivation chain here is not a mathematical derivation; the claimed output is a 'high-quality' benchmark that 'maintains the integrity of the original MMLU-Pro' (Figure 1 caption). The only quantitative support for that claim is back-translation metrics computed with the same model used for forward translation (Sections 2.1, 2.2, 3.3). That validation loop is circular by construction. The circularity is partial, not total: the baseline accuracy scores (Tables 1-2) are computed on the released artifact and are independently reproducible; external human evaluation of the dataset would be a valid non-circular check. However, Section 3.3 itself concedes that metrics for Bengali, Punjabi, Kannada, Telugu, Urdu, and Marathi are missing, and Section 2.6 reports an expert proofreading exercise but publishes no scores or agreement statistics. The published examples in Figures 8-9 ('pupil' rendered as 'eyelid' in Gujarati and Tamil) and Figure 4 ('embryological origin' to 'thematic appearance') are consistent with the concern that meaning-changing errors pass the pipeline. These are correctness risks, and the missing expert scores prevent them from being resolved. No load-bearing self-citation chain or fitted-parameter-as-prediction pattern was found; the central circular step is the same-system back-translation QA.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim of a high-quality benchmark rests on the translation fidelity of IndicTrans2 and on validation procedures that are either circular (back-translation with the same model), partial (metrics for 3/9 languages), or unreported (expert scores). No new physical or mathematical entities are introduced.

free parameters (1)
  • Back-translation quality thresholds = chrF++ > 0.5, BLEU > 25-30, METEOR 0.5-0.6, TER < 0.4-0.5, SacreBLEU > 25-30
    Chosen by hand in Section 2.2 and 3.3 as acceptance thresholds for translation quality; no justification for specific values, and it is unclear whether they were applied to filter items.
assumptions (4)
  • domain assumption IndicTrans2 translations preserve the meaning, difficulty, and answer keys of MMLU-Pro questions for all nine languages.
    The entire benchmark relies on the forward translation being faithful; Section 2.1. Examples in Figures 4, 8, 9 show that this assumption fails for at least some items (e.g., 'embryological origin' becomes 'thematic appearance' in Tamil).
  • domain assumption Back-translation with the same translation model is a valid proxy for translation quality.
    Section 2.2 and 3.3 use IndicTrans2 for both forward and back translation; this cannot detect consistent errors made by the model.
  • domain assumption The 1,000 sampled sentence pairs per language are representative of the full dataset.
    Section 2.6 states 9,000 sentence pairs were proofread (1,000 per language), but the selection details and results are not reported, and the sample may not represent the full test split.
  • domain assumption Automated metrics (chrF++, BLEU, METEOR, TER, SacreBLEU) correlate with human-judged translation quality.
    Used in Section 3.3 as evidence of quality, but no human correlation analysis is provided for this data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding." pith.science (2026). https://pith.science/paper/XSJOPABR

@misc{pith2026250115747,
  author       = {Pith},
  title        = {Pith review of: IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSJOPABR}},
  note         = {Machine review of arXiv:2501.15747}
}
read the original abstract

Known by more than 1.5 billion people in the Indian subcontinent, Indic languages present unique challenges and opportunities for natural language processing (NLP) research due to their rich cultural heritage, linguistic diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) across Indic languages, building upon the MMLU Pro (Massive Multitask Language Understanding) framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique challenges and opportunities presented by the linguistic diversity of the Indian subcontinent. This benchmark encompasses a wide range of tasks in language comprehension, reasoning, and generation, meticulously crafted to capture the intricacies of Indian languages. IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. This paper outlines the benchmarks' design principles, task taxonomy, and data collection methodology, and presents baseline results from state-of-the-art multilingual models.

Figures

Figures reproduced from arXiv: 2501.15747 by the authors.

Figure 1
Figure 1. IndicMMLU-Pro Dataset Construction and Evaluation Pipeline. The diagram illustrates the end-to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The original text sample, its Hindi translation, and the corresponding back-translated text [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 5
Figure 5. Model Accuracy Across Different Languages [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: Average Question and Option Lengths in # [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Additional examples showcasing the machine translation workflow, including the original text samples, [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Additional examples showcasing the machine translation workflow, including the original text samples, [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Additional examples showcasing the machine translation workflow, including the original text samples, [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 17 canonical work pages

  1. [1]

    Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022. Indicxnli: Evaluating multilingual inference for indian languages. arXiv preprint arXiv:2204.08776

    work page Pith review arXiv 2022

  2. [2]

    Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, et al. 2023. Megaverse: Benchmarking large language models across languages, modalities, models and tasks. arXiv preprint arXiv:2311.07463

    work page Pith review arXiv 2023

  3. [3]

    Tejaswini Ananthanarayana, Nikunj Kotecha, Priyanshu Srivastava, Lipisha Chaudhary, Nicholas Wilkins, and Ifeoma Nwogu. 2021 a . Dynamic cross-feature fusion for american sign language translation. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1--8. IEEE

    work page 2021

  4. [4]

    Tejaswini Ananthanarayana, Priyanshu Srivastava, Akash Chintha, Akhil Santha, Brian Landy, Joseph Panaro, Andre Webster, Nikunj Kotecha, Shagan Sah, Thomastine Sarchet, et al. 2021 b . Deep learning methods for sign language translation. ACM Transactions on Accessible Computing (TACCESS), 14(4):1--30

    work page 2021

  5. [5]

    Abhinand Balachandran. 2023. Tamil-llama: A new tamil language model based on llama 2. arXiv preprint arXiv:2311.05845

    arXiv 2023

  6. [6]

    Charith Chandra Sai Balne, Sreyoshi Bhaduri, Tamoghna Roy, Vinija Jain, and Aman Chadha. 2024. Parameter efficient fine tuning: A comprehensive analysis across applications. arXiv preprint arXiv:2404.13506

    arXiv 2024

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  8. [8]

    Rewina Bedemariam, Natalie Perez, Sreyoshi Bhaduri, Satya Kapoor, Alex Gil, Elizabeth Conjar, Ikkei Itoku, David Theil, Aman Chadha, and Naumaan Nayyar. 2025. Potential and perils of large language models as judges of unstructured textual data. arXiv preprint arXiv:2501.08167

    arXiv 2025

Show all 38 references
  1. [9]

    Sreyoshi Bhaduri, Satya Kapoor, Alex Gil, Anshul Mittal, and Rutu Mulkar. 2024 a . Reconciling methodological paradigms: Employing large language models as novice qualitative research assistants in talent management research. arXiv preprint arXiv:2408.11043

    2024 arXiv

  2. [10]

    Sreyoshi Bhaduri, Kenneth Ohnemus, Jess Blackburn, Anshul Mittal, Yan Dong, Savannah Laferriere, Robert Pulvermacher, Marina Dias, Alex Gil, Shahriar Sadighi, et al. 2024 b . (multi-disciplinary) teamwork makes the (real) dream work: Pragmatic recommendations from industry for...

    2024

  3. [11]

    Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2020. https://arxiv.org/abs/2010.12821 Rethinking embedding coupling in pre-trained language models . Preprint, arXiv:2010.12821

    2020 arXiv

  4. [12]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . Prepri...

    2020 arXiv

  5. [13]

    Khapra, and Pratyush Kumar

    Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra, and Pratyush Kumar. 2021. https://arxiv.org/abs/2109.02903 Indicbart: A pre-trained model for natural language generation of indic languages . Preprint, arXiv:2109.02903

    2021 arXiv

  6. [14]

    Dipto Das, Shion Guha, Jed R Brubaker, and Bryan Semaan. 2024. The``colonial impulse" of natural language processing: An audit of bengali sentiment analysis tools and their identity-based biases. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1--18

    2024

  7. [15]

    Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M

    Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. https://arxiv.org/abs/2305.16307 Indictrans2...

    2023 arXiv

  8. [16]

    Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain, Mohammed Safi Ur Rahman Khan, Diptesh Kanojia, Ratish Puduppully, Mitesh M Khapra, Raj Dabre, Rudra Murthy, Anoop Kunchukuttan, et al. 2024. Airavata: Introducing hindi instruction-tuned llm. arXiv preprint arXiv:2401.15006

    2024 arXiv

  9. [17]

    Carolin Holtermann, Paul R \"o ttger, Timm Dill, and Anne Lauscher. 2024. Evaluating the elementary multilingual capabilities of large language models with multiq. arXiv preprint arXiv:2403.03814

    2024 arXiv

  10. [18]

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080

    2020 arXiv

  11. [20]

    Sachin Joshi et al. 2022. Indiccorp: A large-scale dataset for indic languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    2022

  12. [21]

    Khapra, and Pratyush Kumar

    Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020 a . IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages . In Findings of EMNLP

    2020

  13. [23]

    Divyanshu Kakwani et al. 2020 c . Indicglue: A benchmark for evaluating indic language understanding. arXiv preprint arXiv:2004.09095

    2020 arXiv

  14. [24]

    Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. https://arxiv.org/abs/2103.107...

    2021 arXiv

  15. [25]

    Sankalp KJ, Vinija Jain, Sreyoshi Bhaduri, Tamoghna Roy, and Aman Chadha. 2024 a . Decoding the diversity: A review of the indic ai research landscape. arXiv preprint arXiv:2406.09559

    2024 arXiv

  16. [26]

    Sankalp KJ, Ashutosh Kumar, Laxmaan Balaji, Nikunj Kotecha, Vinija Jain, Aman Chadha, and Sreyoshi Bhaduri. 2024 b . IndicMMLU-Pro: A Comprehensive Benchmark for Evaluating Multilingual AI Models across Indic Languages . https://huggingface.co/datasets/LinguaLift/IndicMMLU-Pro...

    2024

  17. [27]

    Aman Kumar, Himani Shrotriya, Prachi Sahu, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Amogh Mishra, Mitesh M Khapra, and Pratyush Kumar. 2022. Indicnlg benchmark: Multilingual datasets for diverse nlg tasks in indic languages. arXiv preprint arXiv:2203.05437

    2022 arXiv

  18. [28]

    Anoop Kunchukuttan. 2020. https://github.com/anoopkunchukuttan/indic_nlp_library The indicnlp library

    2020

  19. [29]

    James Lane. 2019. The 10 most spoken languages in the world. Babbel Magazine, 6(09)

    2019

  20. [30]

    Yaobo Liang et al. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint arXiv:2004.01401

    2020 arXiv

  21. [31]

    Tammy Mackenzie, Leslie Salgado, Sreyoshi Bhaduri, Victoria Kuketz, Solenne Savoia, and Lilianny Virguez. 2024. Beyond the algorithm: Empowering ai practitioners through liberal education. In 2024 ASEE Annual Conference & Exposition

    2024

  22. [32]

    Mounika Marreddy, Subba Reddy Oota, Lakshmi Sireesha Vakada, Venkata Charan Chinni, and Radhika Mamidi. 2022. Am i a resource-poor language? data sets, embeddings, models and analysis for four different nlp tasks in telugu language. ACM Transactions on Asian and Low-Resource L...

    2022

  23. [33]

    Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946--1958

    2017

  24. [34]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation

    2002

  25. [35]

    Maja Popovi \'c . 2017. https://doi.org/10.18653/v1/W17-4770 chr F ++: words helping character n-grams . In Proceedings of the Second Conference on Machine Translation, pages 612--618, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  26. [36]

    Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771

    2018 arXiv

  27. [37]

    Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223--231

    2006

  28. [38]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust and...

    2024 arXiv

  29. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.