REVIEW 19 cited by
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
To be effectively and safely deployed to global user populations, large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them. We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability, specifically measuring their ability to judge social acceptability across varying levels of cultural norm specificity, from abstract values to explicit social norms. As an instantiation of our framework, we create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries. Through comprehensive experiments on NormAd-Eti, we find that LLMs struggle to accurately judge social acceptability across these varying degrees of cultural contexts and show stronger adaptability to English-centric cultures over those from the Global South. Even in the simplest setting where the relevant social norms are provided, the best LLMs' performance (< 82\%) lags behind humans (> 95\%). In settings with abstract values and country information, model performance drops substantially (< 60\%), while human accuracy remains high (> 90\%). Furthermore, we find that models are better at recognizing socially acceptable versus unacceptable situations. Our findings showcase the current pitfalls in socio-cultural reasoning of LLMs which hinder their adaptability for global audiences.
Forward citations
Cited by 19 Pith papers
-
Trusting sovereign language models as scientific instruments: evidence from Portugal's AMALIA
Portugal's 9B-parameter national language model AMALIA agrees with human annotators on coding moral authority but fails a construct-validity test showing it reaches correct codes via surface correlates rather than the...
-
PLURAL: A Global Dataset for Value Alignment
Synthetic preference data generated from the Integrated Values Survey preserves cross-country value differences and enables DPO fine-tuning that improves LLM cultural alignment across five countries.
-
XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad
XCR-Bench provides 4,100+ parallel sentences with 1,098 culture-specific items mapped to Hall's Triad, and shows LLMs struggle most with deeper, semi-visible cultural norms.
-
Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Across nine Asian languages, multilingual LLMs favor Western cultural entities in 30-40% of culturally grounded contexts, with model-specific sentiment biases and extraction accuracy gaps.
-
CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis
A taxonomy-guided retrieval-augmented framework generates CultureSynth-7, a multilingual cultural QA benchmark, and its evaluation of 14 LLMs suggests cultural competence emerges around 3B parameters.
-
EtiCor++: Towards Understanding Etiquettical Bias in LLMs
A new English etiquette corpus and bias metrics show that LLMs over-prefer Western norms and under-predict etiquettes from low-resource regions.
-
CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesis
A multilingual critique-data training paradigm with a knowledge-unit reward improves LLM cultural alignment on several benchmarks, but its headline benchmark is evaluated with the same LLM-judged metric used to select...
-
Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity
Fair-PP contributes a synthetic persona-anchored preference dataset for social equity and a reweighted DPO/SFT alignment method that outperforms baselines on LLM-similarity tests.
-
PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian
PerCul is a Persian cultural story-comprehension benchmark on which the best LLMs lag human performance by 11.3 to 21.3 percentage points.
-
Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness
The paper argues that LLMs should be evaluated and built for meta-cultural competence rather than static knowledge of specific cultures, and gives a first, illustrative measurement of one component.
-
When One LLM Drools, Multi-LLM Collaboration Rules
A position paper that introduces a four-level taxonomy of multi-LLM collaboration (API, text, logit, weight) and argues it is essential for reliability, pluralism, and democratization.
-
On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena
Arab cultural entities that double as everyday Arabic words are harder for language models to recognize, especially when tokenized as single tokens.
-
CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
CultureVerse is a 188-country, 19k-concept visual QA benchmark, and fine-tuning open VLMs on it improves cultural accuracy, but the main evaluation shares concepts between training and test sets.
-
SafeWorld: Geo-Diverse Safety Alignment
This paper introduces a geo-diverse cultural and legal safety benchmark and shows that a DPO-trained 7B model can outperform GPT-4o on it, with caveats about the GPT-4-based evaluation loop.
-
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
This paper quantifies Western-centric bias in MMLU, releases Global-MMLU across 42 languages with human-verified translations, and shows model rankings shift on culturally sensitive versus agnostic subsets.
-
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
INCLUDE is a multilingual benchmark of 197,243 exam questions from local sources that evaluates how well LLMs handle regional and cultural knowledge.
-
Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales
A benchmark of 809 yokai questions shows Japanese-centric LLMs, particularly Llama-3-based continual pretraining models, outperform English-centric models on Japanese folktale knowledge.
-
Musical ethnocentrism in Large Language Models
LLMs like ChatGPT and Mixtral show a strong Western bias when asked to name top musical contributors and to rate the musical cultures of countries.
-
A Survey on Human-Centric LLMs
A review that sorts existing evidence on how well large language models imitate individual human skills and collective social dynamics into one taxonomy.
Discussion (0). Continue with ORCID to comment.