Pith. sign in

REVIEW 4 cited by

SEA-HELM: Southeast Asian Holistic Evaluation of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.14301 v2 pith:QHOILLGG submitted 2025-02-20 cs.CL cs.AI

SEA-HELM: Southeast Asian Holistic Evaluation of Language Models

Yosephine Susanto , Adithya Venkatadri Hulagadri , Jann Railey Montalan , Jian Gang Ngui , Xian Bin Yong , Weiqi Leong , Hamsawardhini Rengarajan , Peerat Limkonchotiwat
show 2 more authors
Yifan Mai William Chandra Tjhi
This is my paper
classification cs.CL cs.AI
keywords sea-helmevaluationlanguagesmodelsasianbenchmarkscapabilitiesholistic
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprising five core pillars: (1) NLP Classics, (2) LLM-specifics, (3) SEA Linguistics, (4) SEA Culture, (5) Safety. SEA-HELM currently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

    cs.CL 2026-06 unverdicted novelty 7.0

    SEATauBench is the first agent benchmark for SEA languages, finding that performance holds for language-only changes but degrades sharply with full domain localization.

  2. Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala

    cs.CL 2026-01 unverdicted novelty 7.0

    Language models degrade over 300 times in performance on Romanized Sinhala versus Unicode, with model size showing no correlation to script robustness.

  3. JT-SAFE-V2: Safety-by-Design Foundation Model with World-Context Data

    cs.AI 2026-05 unverdicted novelty 4.0

    JT-Safe-V2 is a safety-by-design LLM that reports SOTA scores on both capability and safety benchmarks while Safe-MoMA cuts inference cost over 30 percent.

  4. Evaluation of Chunking Strategies for Effective Text Embedding in Low-Resource Language on Agricultural Documents

    cs.CL 2026-05 unverdicted novelty 3.0

    Recursive character-based chunking at 300 characters outperforms Sentence-Based, Khmer-Aware, and LLM-Based methods on L2 distance, answer relevance, and Khmer IoU in a 5-fold evaluation on 18 Khmer agricultural QA pairs.