REVIEW 6 cited by
PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Evaluation of multilingual Large Language Models (LLMs) is challenging due to a variety of factors -- the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and the lack of local, cultural nuances in translated benchmarks. In this work, we study human and LLM-based evaluation in a multilingual, multi-cultural setting. We evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations and find that models such as GPT-4o and Llama-3 70B consistently perform best for most Indic languages. We build leaderboards for two evaluation settings - pairwise comparison and direct assessment and analyze the agreement between humans and LLMs. We find that humans and LLMs agree fairly well in the pairwise setting but the agreement drops for direct assessment evaluation especially for languages such as Bengali and Odia. We also check for various biases in human and LLM-based evaluation and find evidence of self-bias in the GPT-based evaluator. Our work presents a significant step towards scaling up multilingual evaluation of LLMs.
Forward citations
Cited by 6 Pith papers
-
FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes
A new India-focused benchmark shows that popular LLMs exhibit measurable negative bias against marginalized Indian identities and frequently reinforce caste, religion, region, and tribe stereotypes.
-
QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs
QUENCH introduces a generation-based quiz benchmark with masked entities and rationales, and documents a consistent Indic-versus-non-Indic performance gap across seven LLMs.
-
BRoverbs -- Measuring how much LLMs understand Portuguese proverbs
BRoverbs lets researchers test whether language models understand Portuguese proverbs; commercial models nearly master it, small models often guess randomly.
-
One world, one opinion? The superstar effect in LLM responses
Across ten languages, LLMs consistently name a small set of figures such as Einstein, Shakespeare, and Turing for each profession, revealing a 'superstar effect' that may narrow cultural representation.
-
Analysis of Indic Language Capabilities in LLMs
A desk-research review finds that LLM performance is strongest for Hindi, Bengali, Marathi, Telugu, and Tamil, and recommends prioritizing these five languages for safety benchmarks.
-
Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
A single-sentence evaluation claims SUTRA's tokenizer is most token-efficient across 14 of India's 22 official languages, based on one example text per language.
Discussion (0). Continue with ORCID to comment.