Pith. sign in

REVIEW 5 major objections 7 minor 19 references

A new open-source framework with six India-specific benchmarks tests multilingual accuracy, caste and regional bias, Digital Public Infrastructure safety, multilingual refusal, jailbreak resistance, and cultural knowledge, reporting that Sa

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:35 UTC pith:VZ3CVDBQ

load-bearing objection Useful open benchmark for Indian LLM safety, but the empirical results are not reliable enough to support the headline claims; the framework and datasets deserve attention, the numbers don't. the 5 major comments →

arxiv 2607.25375 v1 pith:VZ3CVDBQ submitted 2026-07-28 cs.CL

Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context

Abhishek Kumar Singh , Shrey Nag , Sachita , Lipi Goel , Rajeshwar Singh Janwar This is my paper
classification cs.CL
keywords LLM evaluationIndian languagesmultilingual benchmarkssocial biascasteDPI safetyjailbreakcultural knowledge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that standard English- and Western-centric benchmarks do not surface the safety and fairness failures that matter when large language models are deployed across India's 22 languages and 1.4 billion people. To close this gap, it introduces Inspect India Evals, an open framework of six benchmarks covering multilingual factual accuracy in 16 Indian languages, Indian social bias including caste, safety around Digital Public Infrastructure (Aadhaar, UPI, Bhashini), multilingual safety refusal, multi-turn jailbreak resistance, and rubric-scored Indian cultural knowledge. Testing five open-weight models, the paper claims that Sarvam-M 24B and Gemma 2 27B come out on top with roughly 80% on the composite India Fairness Index, and that Sarvam-M beats larger 32B models on cultural knowledge and DPI safety. The paper also finds a sharp trade-off: the reasoning-focused DeepSeek-R1 14B scores highest on multilingual MMLU but lowest on DPI safety and cultural knowledge, showing that strong reasoning does not imply safe or context-aware behavior in Indian deployments.

Core claim

The central claim is that India-specific evaluation is both necessary and feasible, and that the proposed six-benchmark framework measures dimensions of LLM behaviour that standard benchmarks miss. Using this framework, the authors report that two open-weight models—Sarvam-M 24B and Gemma 2 27B—are the safest and most accurate for Indian deployment, each reaching about 80% on the composite India Fairness Index. Sarvam-M 24B outperforms even larger 32B models on Indian cultural knowledge (60% vs 30% for Qwen 2.5 32B) and on DPI safety compliance (100%), while Gemma 2 27B shows the highest unbiased accuracy on the Indian bias benchmark (100%). All models refused direct harmful prompts in India

What carries the argument

The load-bearing mechanism is the set of six benchmark modules plus the composite India Fairness Index (IFI), which averages four normalized sub-scores: multilingual MMLU accuracy, BharatBBQ unbiased accuracy, multilingual safety refusal rate, and DPI safety compliance. The BharatBBQ module adapts the US-centered bias methodology to 13 Indian demographic axes, using ambiguous versus disambiguated question pairs to compute a stereotype-consistent error rate; the DPI module classifies queries by risk level and expected behaviour to separate under-refusal (complying with fraud assistance) from over-refusal (refusing legitimate Aadhaar/UPI guidance); and the cultural knowledge module uses rubric

Load-bearing premise

The framework's validity rests on the assumption that the newly created evaluation items—the translated MMLU questions and the BharatBBQ bias probes—are accurate and culturally correct; the paper reports no human validation of these items, so if translations are flawed or stereotypes mislabeled, the reported model differences would be artifacts of the dataset rather than real capability or safety gaps.

What would settle it

Take a random sample of the translated Multilingual MMLU items and have native speakers independently back-translate them to English; if the back-translation changes the correct answer for more than a small fraction (say >5%) of items, then the reported language-wise accuracies do not measure what the paper claims.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the framework is valid, choosing models for Indian public-sector and consumer deployments should weigh India-specific safety and bias scores as heavily as general reasoning scores; a model like DeepSeek-R1 that leads multilingual MMLU but fails DPI safety would not be suitable for Aadhaar/UPI-facing chatbots.
  • Specialized Indic fine-tuning appears to pay off: Sarvam-M 24B, a model tuned for Indian languages, beats general-purpose 32B models on cultural knowledge and DPI safety, supporting further investment in Indic-language alignment rather than just parameter scaling.
  • The 100% refusal on direct harmful prompts in five Indian languages is not sufficient: multi-turn jailbreak and domain-specific DPI tests reveal a wide range (40–80% and 20–100%), so deployment evaluations should include adversarial and domain-specific probes.
  • The India Fairness Index gives procurement bodies a single comparable safety/fairness score, which could standardize how Indian government agencies assess AI vendors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct extension, which the paper itself recommends, is to run the suite at N≥50 per task; only DPI safety reached statistical significance at N=5, so a larger run would confirm or overturn the reported ranking.
  • Because cultural knowledge scores rely on a single LLM judge (Llama 3.1 8B), an immediate testable extension is to score the same responses with a second judge model or human raters to measure judge-dependent variance.
  • The framework's modular design could be transferred to other multilingual, non-Western societies by swapping the demographic axes and the digital-infrastructure module, for example for African or Southeast Asian languages.
  • The observed vulnerability of the chain-of-thought reasoning model (DeepSeek-R1) to jailbreaks suggests a testable hypothesis: other reasoning-specialized models should show similar drops in jailbreak resistance compared to instruction-tuned models of the same scale.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper presents Inspect India Evals, an open-source evaluation framework built on the UK AISI Inspect AI platform, comprising six benchmarks: Multilingual MMLU across 16 languages, BharatBBQ bias probes, DPI safety, multilingual safety refusal, jailbreak resistance, and Indian cultural knowledge scored by LLM-as-judge. Five open-weight models (Llama 3.1 8B, DeepSeek-R1 14B, Sarvam-M 24B, Gemma 2 27B, Qwen 2.5 32B) are evaluated with 5 samples per task-model, and results feed an India Fairness Index (IFI). The paper claims Sarvam-M and Gemma 2 are top, with Sarvam-M beating larger 32B models on cultural knowledge and DPI safety.

Significance. If the evaluation items are valid, the framework addresses a genuine gap: India-specific multilingual safety and fairness measurement, with a reproducible, open pipeline that does not require commercial API access. The IFI is transparently defined. However, the empirical findings are not yet reliable: sample sizes are too small, the cultural-knowledge judge is an evaluated model with no human validation, translation quality is not demonstrated, and several reporting inconsistencies appear. The paper's contribution is best read as a framework plus a preliminary pilot, not as a definitive model ranking.

major comments (5)
  1. [§4.2 and Table 2] §4.2 states 'limit=5 samples per dataset module per model,' giving 150 total runs. Yet Table 2 reports per-language percentages for 16 languages for each of five models; e.g., DeepSeek-R1 14B shows 66.7% on Assamese and 100% on Bengali and several other languages, which requires far more than 5 total MMLU samples per model. Either the sampling protocol is misdescribed or the table is not derived from the stated protocol. This makes the headline Multilingual MMLU ranking (DeepSeek 80.6%) impossible to interpret and undermines the reproducibility claim.
  2. [§3.7, §5.8] Indian Cultural Knowledge is scored with Llama 3.1 8B as LLM-as-judge; §5.8 concedes 'a single judge model can also have its own blind spots baked in.' Since Llama 3.1 8B is itself one of the five evaluated models (scoring 20% on the same benchmark), the relative ranking, especially Sarvam-M 24B at 60% vs DeepSeek-R1 14B at 10%, may reflect judge bias or rubric interpretation rather than cultural knowledge. No human-annotation agreement, second-judge comparison, or example items are reported. This is the load-bearing support for the abstract's 'beating larger 32B models on Indian cultural knowledge' claim.
  3. [§5.4 and Table 2] For cultural_knowledge, Sarvam-M (3/5) vs DeepSeek-R1 (0/5) is reported with p=0.222. Fisher's exact test on these counts gives a two-sided p of 0.167 (one-sided p=0.083), so the reported p-value is incorrect. More broadly, only 1 of 5 tested tasks reaches p<0.05, and §5.8 acknowledges the scale limitation; nevertheless, the Abstract and §5.1 state definitive rankings ('came out on top', 'beating larger 32B models'). The conclusions should be explicitly limited to preliminary directional evidence.
  4. [§3.2–3.3] The Multilingual MMLU and BharatBBQ datasets were 'developed with reference to native speaker consultation,' but no translation quality checks, back-translation, pilot validation, inter-annotator agreement, or sample items are provided. Without this, language-specific accuracy and bias scores may reflect translation artifacts or cultural misjudgements rather than model capability. This is particularly important because the per-language MMLU percentages in Table 2 are the basis for claims about language-specific weaknesses.
  5. [Abstract vs §5.7/Table 3] The abstract states that both Sarvam-M 24B and Gemma 2 27B score 80% on the composite India Fairness Index, but Table 3 reports IFI values of 76.6% and 83.1%, respectively. Section 3.9 excludes Cultural Knowledge from IFI, but Table 1's 'Overall Mean' includes it; the abstract and §5.1 conflate these two different averages. This discrepancy in the headline number should be corrected.
minor comments (7)
  1. [Section numbering] The section numbering skips from 3.7 to 3.9; there is no §3.8.
  2. [§3.5] The text says '50 harmful prompts' translated into English, Hindi, Tamil, Telugu, and Bengali (250 prompts) but later says 'All 200 prompts are classified as High risk.' Clarify the actual count.
  3. [Table 2 caption] The caption notes 'Results are directional; re-evaluation at n ≥ 50 is recommended,' but the p-values are labeled two-sided and the cultural_knowledge p-value is arithmetically incorrect (see major comment).
  4. [Language naming] The abstract says 'sixteen Indian languages,' but the list in §3.2 includes English as a baseline; consider wording such as '16 languages including English.'
  5. [Figures] Figures lack proper captions and numbering (e.g., 'Figure- Illustrates...' and 'Figure: Performance Heatmap Matrix'). Reference them consistently in the text.
  6. [References] Reference [13] cites 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding' for mBERT, but the mBERT model is not described in that paper; cite the appropriate multilingual model release.
  7. [§4.2] The phrase 'full-scale corpus (N=825+)' is undefined; specify what constitutes a sample and how the total is computed.

Circularity Check

0 steps flagged

No construction-level circularity; only acknowledged LLM-as-judge validity caveat.

full rationale

The paper is a benchmark construction and evaluation report; it contains no fitted parameters, no derived prediction from fitted inputs, and no self-citation chain that carries a load-bearing uniqueness claim. The India Fairness Index is explicitly defined as an unweighted mean of four sub-scores (§3.9), and the reported IFI values in Table 3 are arithmetic consequences of that definition, not independent predictions. The only self-referential element is the use of Llama 3.1 8B as the LLM-as-judge for Module 6 (Indian Cultural Knowledge), where the same model is also one of the five evaluated models (§3.1, §5.8). The paper itself flags this: 'a single judge model can also have its own blind spots baked in' and recommends cross-checking. This is a validity/measurement limitation (the judge's blind spots could distort relative cultural-knowledge scores), but it is not a circularity by construction: the judge is not fitted to any target score, no equation reduces the claimed ranking to the judge's prior outputs, and the rubric scoring is an external measurement step rather than an analytical derivation. Accordingly, no step meets the standard for circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central empirical rankings rest on datasets built by the authors and a judge model chosen for cost, not validated against human labels. The IFI is a hand-defined composite. These choices are transparent but untested, which is why the correctness risk is medium.

free parameters (2)
  • IFI component weights = 1/4 each
    India Fairness Index is defined as the unweighted mean of four sub-scores (§3.9). Equal weighting is chosen by hand without justification or sensitivity analysis.
  • Cultural knowledge binarization threshold = 0.50
    §5.4 binarizes continuous rubric scores at 0.50 prior to Fisher's exact test; this threshold is arbitrary and not validated.
axioms (4)
  • domain assumption BBQ ambiguity/disambiguation methodology is a valid measure of social bias
    Adopted from Parrish et al. [4] and extended to Indian categories without validating the new BharatBBQ items against human judgments.
  • ad hoc to paper Translated MMLU items preserve meaning and difficulty across 16 Indian languages
    §3.2 states questions were translated into native scripts, but no translation quality metric, back-translation check, or human verification is reported.
  • ad hoc to paper LLM-as-judge (Llama 3.1 8B) yields rubric grades that agree with human judgment
    §3.7 uses local Llama 3.1 8B as judge for cultural knowledge; no inter-annotator agreement or comparison with human scoring is provided, and §5.8 itself calls for cross-checking.
  • domain assumption DPI risk labels (Low/Medium/High and Answer/Refuse expectations) are correct
    §3.4 classifies prompts by risk level and expected behavior; no legal, policy, or inter-annotator validation is reported.
invented entities (1)
  • India Fairness Index (IFI) no independent evidence
    purpose: Composite 0–1 score for model selection in Indian deployment contexts
    §3.9 defines IFI as an equal-weight mean of four sub-scores; no external validation shows that IFI correlates with real-world safety, fairness, or deployment risk.

pith-pipeline@v1.3.0-alltime-deepseek · 12901 in / 11055 out tokens · 99200 ms · 2026-08-01T02:35:35.478767+00:00 · methodology

0 comments
read the original abstract

India is a vast nation of over 1.4 billion people, varied by hundreds of diverse and locally specific traditions and cultures and 22 officially recognized languages. Large language models (LLMs) are now being deployed on a massive scale throughout the mainland as well as in remote villages. However, the common benchmarks - MMLU, BIG-Bench, and TruthfulQA are almost exclusively English- and Western-centric. They do not identify those safety, fairness, and accuracy failures unique to the Indian context. That is the gap Inspect India Evals seeks to fill. It is an open-source framework built on top of UK AISI's Inspect AI platform. It has six benchmarks: Multilingual MMLU across sixteen Indian languages, BharatBBQ (our adaptation of BBQ for Indian social bias), a safety evaluation for Digital Public Infrastructure, a multilingual safety test using harmful prompts in Indian languages, a multi-turn jailbreak resistance test, and an Indian cultural knowledge benchmark scored using LLM-as-judge rubrics. In this study, we tested five open-weight models ranging from 8B to 32B parameters. Sarvam-M 24B and Gemma 2 27B came out on top, both scoring 80% on the composite India Fairness Index, with Sarvam-M even beating larger 32B models on Indian cultural knowledge and DPI safety compliance. All models scored 100% refusal on Multilingual Safety, whereas DPI safety varied from 20% to 100%. The framework is public. It's built to work with the UK AISI registry. Anyone can reproduce or extend this work.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 6 linked inside Pith

  1. [1]

    Hendrycks, D., et al. (2020). Measuring Massive Multitask Language Understanding. ICLR 2021

    2020

  2. [2]

    Zellers, R., et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence? ACL 2019

    2019

  3. [3]

    Lin, S., et al. (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022

    2021

  4. [4]

    Parrish, A., et al. (2021). BBQ: A Hand-Built Bias Benchmark for Question Answering. ACL Findings 2022

    2021

  5. [5]

    UK AI Safety Institute. (2024). Inspect AI: A Framework for Large Language Model Evaluations. https://github.com/UKGovernmentBEIS/inspect_ai

    2024

  6. [6]

    R., et al

    Costa-jussà, M. R., et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672

    Pith/arXiv arXiv 2022

  7. [7]

    Xue, L., et al. (2021). mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. NAACL 2021

    2021

  8. [8]

    Kakwani, D., et al. (2020). IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages. EMNLP Findings 2020

    2020

  9. [9]

    Perez, E., et al. (2022). Red Teaming Language Models with Language Models. arXiv:2202.03286

    Pith/arXiv arXiv 2022

  10. [10]

    Ganguli, D., et al. (2022). Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arXiv:2209.07858

    Pith/arXiv arXiv 2022

  11. [11]

    Yong, Z., et al. (2023). Low-Resource Languages Jailbreak GPT-4. arXiv:2310.02446

    Pith/arXiv arXiv 2023

  12. [12]

    Srivastava, A., et al. (2022). Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. arXiv:2206.04615

    Pith/arXiv arXiv 2022

  13. [13]

    W., Lee, K., & Toutanova, K

    Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (mBERT). NAACL 2019

    2019

  14. [14]

    M., Glavaš, G., Majewska, O., Liu, Q., Vulić, I., & Korhonen, A

    Ponti, E. M., Glavaš, G., Majewska, O., Liu, Q., Vulić, I., & Korhonen, A. (2020). XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning. EMNLP 2020

    2020

  15. [15]

    R., Schwenk, H., & Stoyanov, V

    Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., & Stoyanov, V . (2018). XNLI: Evaluating Cross-lingual Sentence Representations. EMNLP 2018

    2018

  16. [16]

    Zhao, J., Wang, T., Yatskar, M., Ordonez, V ., & Chang, K. W. (2018). Gender Bias in Coreference Resolution: Evaluation and Debias Methods (WinoBias). NAACL 2018

    2018

  17. [17]

    Nadeem, M., Bethke, A., & Reddy, S. (2021). StereoSet: Measuring Stereotypical Bias in Pretrained Language Models. ACL 2021

    2021

  18. [18]

    Weber, M., Domeniconi, G., Chen, J., Weidele, D. K. I., Bellei, C., Robinson, T., & Leiserson, C. E. (2019). Anti-Money Laundering in Bitcoin: Experimenting with Graph Convolutional Networks for Financial Fraud Detection. KDD Workshop 2019

    2019

  19. [19]

    Kunchukuttan, A., et al. (2020). AI4Bharat: Open-source Resources and Models for Indic Natural Language Processing. arXiv:2005.00085

    Pith/arXiv arXiv 2020