Pith. sign in

REVIEW 6 major objections 6 minor 46 references

PARAM-1 BharatGen 2.9B Model

T0 review · 6 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a 2.9-billion-parameter model pretrained on a corpus that is 25% Hindi plus English can serve both as a competent general-purpose model and as a state-of-the-art open baseline for India-centric benchmarks…

desk verdict A 2.9B Hindi-English model whose headline SOTA claims are contradicted by its own tables, and whose advertised tokenizer was not used; not ready for review. read the letter →

arxiv 2507.13390 v1 pith:6DSQTVML submitted 2025-07-16 cs.CL cs.LG

Kundeshwar Pundalik , Piyush Sawarkar , Nihar Sahoo , Abhishek Shinde , Prateek Chanda , Vedant Goswami , Ajay Nagpal , Atul Singh
show 14 more authors
Viraj Thakur Vijay Dewane Aamod Thakur Bhargav Patel Smita Gautam Bhagwan Panditi Shyam Pawar Madhav Kotcha Suraj Racha Saral Sureka Pankaj Singh Rishi Bal Rohit Saluja Ganesh Ramakrishnan
This is my paper
classification cs.CLcs.LG
keywords PARAM-1Hindi-EnglishbilingualmodelIndia-centricLLM25%HindipretrainingallocationMILUbenchmarkSANSKRITItokenizerfertilityinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PARAM-1 is a 2.9-billion-parameter, decoder-only, text-only language model trained from scratch on a bilingual corpus of Hindi and English, with a deliberate 25% Hindi allocation, and then instruction-tuned on a curated bilingual dataset. The paper's central claim is that this design yields a model that is both a competent general-purpose model and a robust baseline for India-centric applications, achieving state-of-the-art results among open models on MILU, MMLU-Hindi, and SANSKRITI and surpassing the Indic-focused Sarvam-1 on several Indian benchmarks. The authors argue this shows linguistic diversity should be embedded at pretraining level through data, tokenization, and evaluation, rather than deferred to post-hoc fine-tuning. A sympathetic reader would care because, if the claim holds, it offers a concrete recipe for small models to serve linguistically diverse regions efficiently.

What carries the argument

The load-bearing mechanism is the data mixture and training curriculum rather than a new architecture. The paper specifies three design levers: a 25% Hindi corpus allocation (1.52 trillion of 5 trillion tokens), a three-phase pretraining schedule—bootstrap on the full corpus, factual-preservation on a 2-trillion-token bilingual corpus with 30% parallel data, and long-context adaptation on a 500-billion-token corpus with many documents above 2,048 tokens—and bilingual instruction tuning on two SFT sets (roughly 1 million and 473,000 pairs) filtered by a strict scoring rubric. The tokenizer described for tokenization fairness is a SentencePiece BPE with a 128K vocabulary and byte fallback, although a footnote in Section 3 states that PARAM-1 itself was trained with the Nemotron tokenizer rather than the in-house one described there.

What would settle it

Run the released PARAM-1 checkpoint on the MILU Hindi and English subsets and on MMLU-Hindi under the paper's stated prompts and decoding settings. The central claim is falsified if the reproduced scores land near Table 4's values (MILU 30.17/36.3) instead of the Section 8.2 prose values (48.3/49.7), or if the margin over Sarvam-1 disappears. A second check is to audit the released data recipe and training logs to see whether the corpus was 5 trillion tokens as in Section 5.1.1 or tens of billions as in Section 6.1.

Watch

Extended reading notes

Core claim

PARAM-1 is presented as a 2.9-billion-parameter, decoder-only, text-only transformer trained from scratch on a bilingual corpus of Hindi and English, with 1.52 trillion Hindi tokens out of a total 5 trillion, and then instruction-tuned on curated bilingual datasets. The paper's central claim is that this recipe yields a model that is simultaneously a competent general-purpose model and a robust baseline for India-centric applications, achieving state-of-the-art results among open models on MILU, MMLU-Hindi, and SANSKRITI and outperforming the Indic-focused Sarvam-1 by roughly 6 points on MILU in both Hindi and English. The authors present this as evidence that linguistic diversity should be built into pretraining through corpus allocation, tokenization, and culturally aligned evaluation, rather than added later through fine-tuning.

Load-bearing premise

The central claim rests on the accuracy of the paper's reported benchmark numbers and training-data description; the paper itself contains discrepancies—MILU is given as 48.3/49.7 in the prose and 30.17/36.3 in Table 4, ARC-Challenge few-shot is 52.9% in prose but 54.4% in the table, and the pretraining corpus is described in one section as 5 trillion tokens and in another as tens of billions—so if those numbers do not reproduce, the claimed state-of-the-art India-centric performance collapses.

Editorial extensions

If this is right

  • A 2.9B model with deep Hindi representation could serve as an efficient open baseline for Indian knowledge and cultural tasks, reducing the compute needed for India-centric deployment.
  • The 25% Hindi allocation becomes a concrete design target for future Indic models, implying that broad multilinguality is not required for strong India-focused performance.
  • The bilingual instruction-tuning pipeline, filtered to a perfect rubric score, would be a reusable recipe for culturally aligned alignment.
  • If the reported results hold, they suggest that pretraining-stage representation, rather than post-hoc fine-tuning, is the place to secure performance in underrepresented languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports MILU prose scores of 48.3/49.7 and table scores of 30.17/36.3; I infer that the headline 'state-of-the-art' claim is provisional until the released checkpoint reproduces one set of numbers.
  • The comparison against Sarvam-1, which was trained on multiple Indic languages, hints that depth in one major language may transfer better than breadth across many, but the paper runs no ablation that isolates this factor.
  • A testable extension is to train a sibling model with 12.5% Hindi and 12.5% of a second Indic language, to separate the effect of Hindi depth from the effect of the overall 25% Indic allocation.
  • The discrepancy between the 5-trillion-token pretraining claim and the 'tens of billions' figure in the infrastructure section could be settled by auditing the released data recipe and training logs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces PARAM-1, a 2.9B-parameter English-Hindi decoder-only transformer pretrained from scratch on a claimed 5T-token corpus with a 25% Hindi allocation, a custom tokenizer (Section 3), three progressive pretraining phases, and bilingual SFT. It reports English and Indic benchmark results in Tables 3 and 4, and Section 8.2 claims state-of-the-art performance among open models on MILU, MMLU-Hindi, and SANSKRITI. The paper also reports tokenizer fertility comparisons, Prometheus-based generation quality, and toxicity evaluations.

Significance. If the reported results were reliable, the paper would offer a useful design point for India-centric bilingual modeling: a relatively small dense model with a deliberate 25% Hindi data allocation and fertility-aware tokenization, compared directly against several open baselines. The tokenizer fertility comparison in Table 1 is a useful artifact. However, the manuscript is dominated by internal contradictions between prose claims and tables, and no code, model weights, or evaluation harness are released. As submitted, the central claims of competence and SOTA performance are not supported by a consistent empirical record.

major comments (6)
  1. [§8.2, Table 4] Section 8.2 claims PARAM-1 scores 48.3% (Hindi) and 49.7% (English) on MILU, surpassing SARVAM-1 by approximately 6 points in both languages, but Table 4 lists PARAM-1 MILU scores of 30.17 (Hi) and 36.3 (En), with SARVAM-1 at 28.48 (Hi) and 32.12 (En) and QWEN-3B at 33.6 (Hi) and 49.84 (En). According to the only tabulated numbers, PARAM-1 is not state-of-the-art on MILU, and the claimed ~6-point advantage is not supported. This directly contradicts the central 'state-of-the-art among open models' statement.
  2. [§8.1, Table 3] Section 8.1 states that PARAM-1 achieves 52.9% (few-shot) on ARC-Challenge, outperforming SARVAM-1 (44.8%) and QWEN-2.5 3B (50.4%), but Table 3 reports SARVAM-1 at 54.4% and QWEN-3B at 57.08% for this column. The same paragraph claims MMLU few-shot 35.2% versus SARVAM-1 28.7%, while Table 3 gives 46.0% for PARAM-1 and 47.7% for SARVAM-1. The prose comparisons and table entries disagree on both the model's own score and the baseline scores, so the reader cannot determine which set of numbers constitutes the reported result.
  3. [§8.3] Section 8.3 says that in English, PARAM-1 achieves an overall score of 3.259, 'slightly ahead of' SARVAM-1 (3.318). The numeric value 3.259 is lower than 3.318, so this sentence is internally inconsistent and contradicts the surrounding claim of consistent gains over SARVAM-1. If the intended ordering was the opposite, the text should state it explicitly and use the correct scores.
  4. [§3, §6.2] Section 3 describes a custom SentencePiece BPE tokenizer (BharatGen-128K v1) and presents its fertility results in Table 1, but the final paragraph of that section states that 'PARAM-1 was trained using the Nemotron tokenizer [31].' Section 6.2, in contrast, says the NeMo tokenizer customization 'helped us to use or inhouse multilingual tokenizer.' These statements are mutually exclusive. If the Nemotron tokenizer was used, Section 3 and Table 1 do not describe the model's tokenizer and the tokenizer-fairness claim is unfounded; if the in-house tokenizer was used, the note in Section 3 is false. This is a load-bearing inconsistency for the paper's tokenization-fairness contribution.
  5. [§2.1, §5.1.1, §6.1] Section 2.1 and Section 5.1.1 describe a 5 trillion-token multilingual corpus (3.48T English + 1.52T Hindi) for pretraining, while Section 6.1 says the training ran 'over tens of billions of tokens using hundreds of H100 GPUs.' These statements differ by roughly two orders of magnitude and cannot both be true. The additional Phase-2 (2T tokens) and Phase-3 (500B tokens) corpora further compound the discrepancy. Without a consistent statement of the actual training data volume, the training narrative is not reproducible.
  6. [Table 4] In Table 4, the HellaSwag-Hindi scores for PARAM-1 are 71.4 (zero-shot) and 73.4 (few-shot), which are exactly identical to the English HellaSwag scores reported for PARAM-1 in Table 3. In contrast, every other baseline shows a substantial drop when moving from English to Hindi HellaSwag (e.g., QWEN-3B from 73.6 to 32.9; SARVAM-1 from 66.9 to 42.9). The identical values strongly suggest the Hindi HellaSwag evaluation was not performed separately or was misreported, which materially weakens the claimed cross-lingual evaluation evidence.
minor comments (6)
  1. [Section 7] The text says 'LogiQA [16]' but reference [16] is the TriviaQA paper; a correct citation for LogiQA is needed.
  2. [Section 6.3] The description of LLAMA-3.2-3B as 'a pruned and distilled variant of the LLaMA 3.1 70B variant' is inaccurate for the dense 3B model; this characterization should be corrected or removed.
  3. [Tables 3 and 4] Tables 3 and 4 report single evaluation runs without standard errors, confidence intervals, or any statement of the number of repetitions; for claimed improvements of 1–3 points (e.g., SANSKRITI in Table 4), this is insufficient to establish that the differences are meaningful.
  4. [Introduction and Section 8] The introduction promises evaluations on IndicQA, code-mixed reasoning, and socio-linguistic robustness tasks, but no results for these tasks are reported in Section 8; either add the results or adjust the introduction to describe what is actually evaluated.
  5. [Table 4 and Section 8.2] Section 8.2 claims that PARAM-1 'outperforms SARVAM-1 in all four question types' on SANSKRITI, but Table 4 reports only an overall SANSKRITI score and no per-type breakdown, so this claim is unverifiable from the presented results.
  6. [Section 7] The Hindi HellaSwag evaluation is referenced as 'its Hindi adaptation' without citing a dataset or describing the adaptation procedure, making the results difficult to reproduce.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation is claimed, so no circularity; internal benchmark inconsistencies are evidentiary problems, not circular steps.

full rationale

PARAM-1 is an empirical pretraining and instruction-tuning report; its central claims are benchmark measurements after training, not quantities derived from an equation whose inputs already contain the outputs. The only formal objects in the paper (the next-token cross-entropy loss in Section 5 and the fertility-update equations in Section 3.2) are training or optimization definitions, and they are never used to predict the scores in Tables 3 and 4. No fitted parameter is renamed as a prediction: the model weights are the fitted objects, and the reported scores are direct evaluations of those weights on external benchmarks. The self-citations to Udaan [27] and IndicQA [38] are data or benchmark sources rather than load-bearing derivations; IndicQA appears in the SFT corpus but no IndicQA result is reported, and the headline India-centric SOTA claim rests on MILU, MMLU-Hindi, and SANSKRITI, none of which are authored by this team. Using DeepSeek-V3 or Qwen-32B-IT to generate or filter instruction data is a data-construction choice, not a circular evaluation loop, because the final measurements are on the trained model against fixed public benchmarks. The prose-vs-table discrepancies (e.g., MILU 48.3/49.7 vs Table 4's 30.17/36.3, ARC-Challenge 52.9 vs Table 3's 54.4, and 5T tokens vs 'tens of billions' in Section 6.1) and the admission that PARAM-1 was actually trained with the Nemotron tokenizer despite the BharatGen-128K v1 tokenizer analysis are serious evidentiary and consistency problems, but they are correctness risks rather than circular reductions. No step in the paper's argument reduces, by the paper's own equations or by self-citation, to its own inputs, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's scientific contribution rests on hand-chosen data mixtures and filters rather than a derivation with fixed constants; no new entities are postulated.

free parameters (5)
  • Indic corpus allocation ratio = 1.52T/5T (30.4%), described as 25%
    Chosen by hand to ensure equity; the central claim depends on this mixture, but no ablation is reported.
  • Momentum factor mu (tokenizer fertility) = not specified
    Defined in Section 3.2's update rule; hand-chosen smoothing for the fertility rebalancing, which is not used in the final model.
  • Smoothing constant epsilon = not specified
    Used in Section 3.2 to avoid division by zero; not applied to the deployed tokenizer.
  • Phase data ratios = PT2: 20% PT1, 30% parallel, 50% new; PT3: 20% reuse, 30% parallel, 50% long docs
    Hand-selected curriculum proportions; no sensitivity analysis provided.
  • SFT quality threshold = 5 out of 5 on all four Qwen-32B-IT rubric dimensions
    Retaining only perfect-scored instruction pairs is a hand-chosen filter that determines the SFT dataset and hence reported chat performance.
assumptions (4)
  • domain assumption Increasing Indic-language pretraining data improves Indic-task performance
    Core premise of the design; asserted but not demonstrated with controlled ablations.
  • domain assumption Fertility score is a meaningful proxy for tokenizer quality
    Used to justify the tokenizer design in Section 3, yet the deployed model uses the Nemotron tokenizer.
  • domain assumption Benchmark scores reflect real-world competence
    The paper equates scores on MMLU, MILU, SANSKRITI etc. with a 'competent general-purpose model'; no human evaluation is reported.
  • ad hoc to paper The described 5T-token corpus was actually used
    Section 6.1 says 'tens of billions of tokens', contradicting the 5T claim; the training narrative is unverifiable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PARAM-1 BharatGen 2.9B Model." pith.science (2026). https://pith.science/paper/6DSQTVML

@misc{pith2026250713390,
  author       = {Pith},
  title        = {Pith review of: PARAM-1 BharatGen 2.9B Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6DSQTVML}},
  note         = {Machine review of arXiv:2507.13390}
}
read the original abstract

Large Language Models (LLMs) have emerged as powerful general-purpose reasoning systems, yet their development remains dominated by English-centric data, architectures, and optimization paradigms. This exclusionary design results in structural under-representation of linguistically diverse regions such as India, where over 20 official languages and 100+ dialects coexist alongside phenomena like code-switching and diglossia. We introduce PARAM-1, a 2.9B parameter decoder-only, text-only language model trained from scratch with an explicit architectural and linguistic focus on Indian diversity. PARAM-1 is trained on a bilingual dataset consisting of only Hindi and English, constructed with a strong focus on fact-rich, high-quality content. It is guided by three core principles: equitable representation of Indic languages through a 25% corpus allocation; tokenization fairness via a SentencePiece tokenizer adapted to Indian morphological structures; and culturally aligned evaluation benchmarks across IndicQA, code-mixed reasoning, and socio-linguistic robustness tasks. By embedding diversity at the pretraining level-rather than deferring it to post-hoc alignment-PARAM-1 offers a design-first blueprint for equitable foundation modeling. Our results demonstrate that it serves as both a competent general-purpose model and a robust baseline for India-centric applications.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages

  1. [31]

    Nemotron-4 15b technical report.arXiv preprint arXiv:2402.16819, 2024

    Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, et al. Nemotron-4 15b technical report.arXiv preprint arXiv:2402.16819, 2024

    arXiv 2024

  2. [1]

    Winogrande: An adversarial winograd schema challenge at scale. 2019

    2019

  3. [2]

    Gqa: Train- ing generalized multi-query transformer models from multi-head checkpoints, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Train- ing generalized multi-query transformer models from multi-head checkpoints, 2023. https://arxiv.org/abs/2305. 13245

    work page 2023

  4. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [4]

    Llama-nemotron: Efficient reasoning models, 2025

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier De- lalleau, Zi...

    arXiv 2025

  6. [5]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  7. [6]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    arXiv 2018

  8. [7]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.https://www.databricks

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.https://www.databricks. com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm

    work page 2023

Show all 46 references
  1. [8]

    Ultrafeedback: Boosting language models with high-quality feedback, 2023

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023

    2023

  2. [9]

    Language Identification on Short Textual Data

    Yexin Cui. Language Identification on Short Textual Data. PhD thesis, 2020. 24

    2020

  3. [11]

    Distributed, combined cpu and gpu profiling within hpx using apex

    Patrick Diehl, Gregor Daiss, Kevin Huck, Dominic Marcello, Sagiv Shiber, Hartmut Kaiser, Juhan Frank, Geoffrey C Clayton, and Dirk Pflüger. Distributed, combined cpu and gpu profiling within hpx using apex. arXiv preprint arXiv:2210.06437, 2022

    arXiv

  4. [12]

    Enhancing chat language models by scaling high-quality instructional conversations, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023. https://arxiv.org/abs/ 2305.14233

    2023 arXiv

  5. [13]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection, 2022

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection, 2022. https://arxiv.org/abs/2203. 09509

    2022

  7. [15]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  8. [16]

    triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017

    2017 arXiv

  9. [17]

    Adapting multilingual llms to low-resource languages using continued pre-training and synthetic corpus

    Raviraj Joshi, Kanishk Singla, Anusha Kamath, Raunak Kalani, Rakesh Paul, Utkarsh Vaidya, Sanjay Singh Chauhan, Ni- ranjan Wartikar, and Eileen Long. Adapting multilingual llms to low-resource languages using continued pre-training and synthetic corpus. arXiv preprint arXiv:24...

    arXiv

  10. [18]

    Mistral–a journey towards reproducible language model training, 2021

    Siddharth Karamcheti, Laurel Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Dan Jurafsky, et al. Mistral–a journey towards reproducible language model training, 2021

    2021

  11. [19]

    Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra. Indicllmsuite: A blueprint for creating pre-training and fi...

    2024 arXiv

  12. [20]

    Prometheus: Inducing fine-grained evaluation capability in language models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491, 2023

    arXiv

  13. [21]

    Nemo: a toolkit for building ai applications using neural modules

    Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al. Nemo: a toolkit for building ai applications using neural modules. arXiv preprint arXiv:1909.09577, 2019. 25

    1909 arXiv

  14. [22]

    Mi- randa, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Mi- randa, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ro- nan Le Bras, Oyvind Tafjord, Chris Wilhelm...

    2025

  15. [23]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/datasets/Open-Orca/OpenOrca, 2023

    2023

  16. [24]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. https: //arxiv.org/abs/2109.07958

    2022 arXiv

  17. [25]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  18. [26]

    Llm360: Towards fully transparent open-source llms

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Rob...

    2023

  19. [27]

    Udaan-machine learning based post-editing tool for document translation

    Ayush Maheshwari, Ajay Ravindran, Venkatapathy Subramanian, and Ganesh Ramakrishnan. Udaan-machine learning based post-editing tool for document translation. In Proceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 2...

    2023

  20. [28]

    Dictdis: Dictionary constrained disambiguation for improved nmt

    Ayush Maheshwari, Preethi Jyothi, and Ganesh Ramakrishnan. Dictdis: Dictionary constrained disambiguation for improved nmt. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10991–11004, 2024

    2024

  21. [29]

    Sanskriti: A comprehensive benchmark for evaluating language models’ knowledge of indian culture, 2025

    Arijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka, and Sriparna Saha. Sanskriti: A comprehensive benchmark for evaluating language models’ knowledge of indian culture, 2025. https://arxiv.org/abs/2506.15355

    2025

  22. [30]

    The lambada dataset: Word prediction requiring a broad discourse context,

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context,

  23. [32]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlí ˇcek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  24. [33]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  25. [34]

    Samanantar: The largest publicly available parallel corpora collection for 11 indic languages

    Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan Ak, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Divyanshu Kakwani, Navneet Kumar, et al. Samanantar: The largest publicly available parallel corpora collection for 11 indic languages. Transa...

    2022

  26. [35]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3...

    2020

  27. [36]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  28. [37]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  29. [38]

    Indic qa benchmark: A multilingual benchmark to evaluate question answering capability of llms for indic languages, 2025

    Abhishek Kumar Singh, Vishwajeet kumar, Rudra Murthy, Jaydeep Sen, Ashish Mittal, and Ganesh Ramakrishnan. Indic qa benchmark: A multilingual benchmark to evaluate question answering capability of llms for indic languages, 2025. https: //arxiv.org/abs/2407.13522. 26

    2025 arXiv

  30. [39]

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset. arXiv preprint arXiv:2412.02595, 2024

    2024 arXiv

  31. [40]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. https://arxiv.org/abs/2104.09864

    2023 arXiv

  32. [41]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

  33. [42]

    Dclm framework: understanding collaboration in open-ended tabletop learning environments

    Mike Tissenbaum, Matthew Berland, and Leilah Lyons. Dclm framework: understanding collaboration in open-ended tabletop learning environments. International Journal of Computer-Supported Collaborative Learning, 12(1):35–64, 2017

    2017

  34. [43]

    Milu: A multi-task indic language understanding benchmark, 2025

    Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen. Milu: A multi-task indic language understanding benchmark, 2025. https://arxiv.org/abs/2411.02538

    2025 arXiv

  35. [44]

    Smith, Hannaneh Hajishirzi, and Daniel Khashabi

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby ...

    2022 arXiv

  36. [45]

    code": <1-5>,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 27 A Data Filtering Filtering Prompt You are an evalu...

    2019

  37. [2016]

    https://arxiv.org/abs/1606.06031

    arXiv

  38. [2024]

    https://arxiv.org/abs/2408.00118

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.