Pith. sign in

REVIEW 5 major objections 5 minor 54 references

EtiCor++: Towards Understanding Etiquettical Bias in LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Across five leading LLMs, etiquette region identification consistently over-selects North America and Europe and under-selects low-resource regions such as Latin America, Middle East and Africa, and East Asia, even when the model is wrong.

desk verdict Useful corpus and tasks, but the paper's central 'all models' bias claim is contradicted by its own tables. read the letter →

arxiv 2506.08488 v1 pith:H37E4WUV submitted 2025-06-10 cs.CL cs.AIcs.CY

Ashutosh Dwivedi , Siddhant Shivdutt Singh , Ashutosh Modi This is my paper
classification cs.CLcs.AIcs.CY
keywords etiquetticalbiasculturalinLLMsetiquettecorpusregionidentificationLLMevaluationmetricssensitivityEtiCor++
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a resource, EtiCor++, of 47,720 English-language etiquette statements grouped into five world regions (East Asia, Middle East and Africa, Indian Subcontinent, Latin America, North America and Europe) and into four social contexts (dining, travel, visits, business). On top of the corpus it defines three evaluation tasks — Etiquette Sensitivity, Region Identification, and Etiquette Generation — plus seven metrics for measuring what the authors call etiquettical bias. Running five widely used LLMs through these tasks, the paper reports a consistent pattern: models perform best on North America and Europe, rarely choose Latin America, Middle East and Africa, or East Asia when identifying a region, and when they hallucinate they disproportionately fall back to North America and Europe. The claim matters because LLMs are increasingly used as everyday cultural advisers; if these models systematically misread etiquette from large parts of the world, their advice is least reliable precisely for users from those regions.

What carries the argument

The central object is EtiCor++ itself: 47,720 region-labeled etiquette sentences, each tagged with one of four activity groups and a Positive/Negative acceptability label, covering East Asia, Middle East and Africa, Indian Subcontinent, Latin America, and North America and Europe. The argument is carried by seven algorithmic bias metrics computed against this corpus: Preference Score and Bias For Region Score (how often a model picks a region overall and when wrong), Pairwise Regions Bias Score (confusion between regions), Generation Alignment Score (entailment vs contradiction in generated etiquettes), Odds Ratio (dominant themes via POS analysis), and two Incremental Option Testing variants (Closeness/Consistency/Option Sensitivity and Distancing) that probe how model choices shift as region options are added. An auxiliary inter-region correlation algorithm (SBERT similarity plus RoBERTa-MNLI polarity) motivates the option ordering used in the incremental tasks.

What would settle it

Collect an independent etiquette corpus from in-region sources (e.g., local cultural guides and anthropologists' descriptions), with its own region distribution, and re-run the six bias metrics on the same five LLMs. If the NE over-preference and low-resource under-preference disappear or reverse when the baseline distribution changes, then the reported 'bias against low-resource regions' is an artifact of EtiCor++'s particular regional mix rather than a stable property of the models.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that etiquettical bias is not a quirk of one model but a common trend across all five evaluated LLMs: in the Region Identification task, the Preference Score for North America and Europe exceeds the region's share of EtiCor++ for every model, while Latin America, Middle East and Africa, and East Asia are consistently under-selected. The Bias For Region Score sharpens the claim: when a model is wrong about an etiquette's region, it overwhelmingly names North America and Europe. The paper also finds that Etiquette Sensitivity accuracy is highest for NE and lowest for low-resource cultures, and that models generate stereotyped content — e.g., India described with 'traditional', 'spicy', 'diverse' and NE with 'smart', 'egalitarian', 'independent' — as revealed by Odds Ratio analysis. Finally, Incremental Option Testing shows accuracy falls as more region choices are added, indicating the bias is not an artifact of forced choice.

Load-bearing premise

The bias metrics define 'expected' behavior as matching EtiCor++'s own regional distribution D(R) and label any deviation as bias; the whole argument therefore presupposes that EtiCor++ is an unbiased sample of the world's etiquette knowledge and that its region labels are correct ground truth.

Editorial extensions

If this is right

  • LLM-based assistants will give least accurate etiquette guidance for Latin America, Middle East/Africa, and East Asia, and will default to North American/European norms when uncertain.
  • When models invent an etiquette detail, they tend to attribute it to Western regions, which can reinforce a single-culture default in downstream applications.
  • Simply adding more answer choices to a multiple-choice etiquette query does not fix the bias; accuracy drops as options grow, so user-facing systems should not rely on option sets alone.
  • Generation Alignment Scores around 0.5 or lower for most models mean generated etiquette advice frequently contradicts itself across regions, making it unreliable for cross-cultural content creation.
  • The Odds Ratio results show the bias is not just about choosing wrong regions but about producing stereotyped content (e.g., 'traditional' for India, 'smart' for Europe), so evaluation of cultural sensitivity should include generated text, not just classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the NE over-preference is driven by the geographical skew of pretraining data, then rebalancing pretraining or fine-tuning data with more etiquette content from low-resource regions should shrink the PS and BFS gaps; this is a testable mitigation the paper does not attempt.
  • The fact that GAS is high for India on several models while classification accuracy on India is middling suggests consistency and accuracy can diverge, so future benchmarks should measure them separately and not treat fluent generation as cultural competence.
  • A multilingual EtiCor++ would let researchers test whether the English-only prompt and English-only corpus contribute to the Western preference; the authors list this as future work.
  • The pairwise BSP confusion matrix could be used as a diagnostic map: regions that models confuse (e.g., NE vs MEA) may be the best starting points for targeted data collection or debiasing, since they mark where the model's etiquette knowledge is thinnest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. EtiCor++ is a resource paper that introduces a corpus of 47,720 English etiquette sentences annotated with five world regions (EA, MEA, India, LA, NE), four activity types (Dining, Travel, Visits, Business), and positive/negative acceptability labels. The paper proposes three evaluation tasks—Etiquette Sensitivity, Region Identification, and Etiquette Generation—along with Incremental Option Testing and seven bias metrics. Five LLMs (GPT-4o, Gemini-1.5, Llama-3.1, Gemma-2, Phi-3.5) are evaluated, and the headline empirical claim is that LLMs are systematically biased against low-resource regions and over-prefer North America/Europe (NE), especially when the models are wrong.

Significance. The paper's main strengths are the scale of the released resource, the extension of EtiCor to additional countries and regions, and the breadth of evaluation tasks and open-weights models with repeated runs. If the resource and metrics were validated, EtiCor++ would be a useful addition to cultural-bias evaluation. However, the headline cross-model bias claim is not currently supported by the paper's own tables: at least two models over-prefer INDIA and EA. Moreover, the bias metrics' reference distribution is internal to the corpus, and the GAS thresholds and correlation proxy are unvalidated. These issues are central to the paper's contribution, so the empirical conclusions need substantive re-analysis before publication.

major comments (5)
  1. [§5, Table 4 and Table 15] The claim that 'The bias against low-resource regions is a common trend across all models' and that, when wrong, models 'overwhelmingly select NE' is contradicted by the reported scores. In Table 4, PS(R)−D(R) is positive for INDIA in Llama (+5.16), Gemma (+17.7), and Phi (+11.06), and positive for EA in GPT (+2.6) and Llama (+9.97). BFS also contradicts NE dominance: Gemma's largest BFS is INDIA (47.2 vs NE 35.4), and Llama's is EA (35.6 vs NE 25.5). Table 15 shows that when the true region is MEA, Gemma's most frequent incorrect answer is INDIA (59.3% vs NE 31.9%) and Llama's is EA (39.3% vs NE 26.8%). At least two of the five models therefore do not follow the stated common trend. The central empirical claim should be restated in a way that is consistent with Tables 4 and 15, or supported by an additional analysis that defines the claimed trend more precisely.
  2. [§4, Preference Score (PS(R)) definition] The bias interpretation of PS(R) depends on treating the corpus's label distribution D(R) as the expected neutral baseline. D(R) is not an external ground truth; it is the distribution of EtiCor++, a corpus built by scraping English-language travel and government websites. Under the proposed definition, a uniform-random model would show systematic positive and negative deviations from D(R) and would be labeled biased even though it has no cultural preference. The paper should either provide evidence that D(R) approximates an independent distribution of world etiquettes or reframe PS deviations as dataset-relative tendencies. The word 'bias' as currently operationalized conflates the model's preferences with the resource's construction.
  3. [§4, Generation Alignment Score and Algorithm 1] GAS uses two free thresholds—cosine similarity 0.55 and NLI contradiction/entailment 0.90—both 'selected via initial experiments,' but no sensitivity analysis is reported. Because the GAS values in Table 6 are close to the 0.5 interpretation boundary for several model–region pairs, small threshold changes could flip conclusions about whether a model is consistent or contradictory. Similarly, Algorithm 1's semantic-similarity-plus-MNLI correlation is used to order options in Incremental Option Testing, and the conclusions in Section 5 depend on that ordering, yet the correlation proxy is never checked against human judgments or any external validation. The paper should report threshold sensitivity and validate the correlation proxy, or explicitly state that the incremental-testing results are conditional on this unvalidated ordering.
  4. [§5 and App. F.1] GPT-4o and Gemini are evaluated on only 200 samples per region (1,000 total), while Llama, Gemma, and Phi are evaluated on the full corpus, yet Table 4 combines all five models in the cross-model 'common trend' claim without confidence intervals for GPT and Gemini. The larger sampling variance for the two closed models is not accounted for in the comparison. In addition, abstentions are simply excluded from the sensitivity task (up to about 1,986 for Llama in Table 14), and no analysis is given for how exclusion changes accuracy or the bias metrics. Please report results on comparable subsamples with confidence intervals and show that the headline conclusions are robust to abstention handling.
  5. [App. D, Region Identification prompt] The RI prompt lists the options as 'East Asia, India, Middle East, South America, Europe,' while the dataset regions are EA, Indian Subcontinent, MEA, LA, and NE. The PS, BFS, and BSP metrics treat the model's answer as one of the dataset's region labels, but the prompt uses mismatched names ('South America' vs 'Latin America', 'India' vs 'Indian Subcontinent', 'Middle East' vs 'MEA'). This introduces a systematic measurement confound in all Region Identification results. The prompt should use exactly the dataset's region names, or the authors should verify that the models map the prompt options to the intended regions without ambiguity.
minor comments (5)
  1. [§4, BSP description] In the paragraph defining Pairwise Regions Bias Score, the text 'BSS is not a symmetric metric' appears to be a typo for 'BSP'; please correct and also clarify the denominator to make explicit that it counts all incorrect predictions for region R.
  2. [§3, Table 2] The text says 'one of the 4 four social activities,' which contains a redundant numeral; the table header '# Travel # Business # Visits # Dining' also drops the word 'Etiquettes' and would be clearer with explicit column names.
  3. [§5, Region Identification paragraph] There are minor typos: 'Llamahave' should be 'Llama have' and 'Pairwise Resion Bias Score' should be 'Pairwise Region Bias Score.'
  4. [App. B.3] Krippendorff's α is reported as 0.91, but the number of annotators, their background, and the annotation instructions are not described; brief details would help readers interpret the reliability estimate.
  5. [Figure 1] Figure 1 shows regions on a world map but the five region names are not marked in the figure; a legend or region labels would make the figure self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: model outputs are measured externally and the PS/BFS baselines are stated analytical choices, not fitted predictions.

full rationale

EtiCor++'s central bias measurements are not algebraically forced by the corpus. The Preference Score is defined as PS(R)=... and 'bias' is operationalized as PS(R)-D(R), with D(R) the corpus's regional share; this is an explicit, externally stated baseline rather than a parameter fitted from the target claim. The model's PS values come from independent LLM responses to region-identification queries, so the empirical finding that some regions are selected more/less than D(R) is not equivalent to the metric's input. BFS uses a uniform 20% baseline, and the pairwise and incremental metrics are computed from recorded model choices, not from the metric definitions themselves. The GAS thresholds (0.55 similarity, 0.90 NLI) are disclosed hyperparameters chosen during development, and the resulting scores are model-generated. Self-citations to Dwivedi et al. (2023) and Adilazuarda et al. (2024) provide the prior EtiCor corpus, task definition, and a survey; they are provenance and context, and the bias conclusions do not rest on an unverified claim from those works. The reliance on D(R) and the regional groupings is a validity/design assumption, and the Section 5 'all models' generalization is arguably inconsistent with Tables 4/15, but these are correctness concerns, not equation-level circularity. The derivation chain therefore contains no step in which a stated prediction reduces by construction to its own input.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central bias claim rests on several untested assumptions: the corpus's representativeness, the region taxonomy, the validity of the GAS thresholds, and the choice of D(R) as baseline. The paper acknowledges some of these in its Limitations.

free parameters (2)
  • GAS similarity threshold = 0.55
    Used to filter generated etiquettes before NLI; selected via initial experiments (Section 4).
  • GAS NLI threshold = 0.90
    Used to filter entailment/contradiction in the Generation Alignment Score; selected via initial experiments (Section 4).
assumptions (5)
  • domain assumption English-language web sources are representative of etiquette norms for all regions
    The corpus is scraped only from English-language websites, which likely over-represents Western sources; the authors acknowledge this in Limitations.
  • domain assumption The five-region grouping (EA, MEA, IN, LA, NE) is a meaningful taxonomy of world etiquette
    Regions are grouped by cultural similarity, but Africa is merged with the Middle East due to data scarcity, and Russia is placed in NE (Section 3).
  • ad hoc to paper Semantic similarity plus MNLI is a valid proxy for cultural correlation between etiquettes
    Algorithm 1 uses SBERT cosine similarity and RoBERTa-MNLI entailment to approximate correlation; the authors note it is 'the simplest possible approximate measure' and 'prone to errors' (Section 3, Limitations).
  • domain assumption The corpus's region labels are correct ground truth for the Region Identification task
    The RI task expects a single region per etiquette, but etiquettes can be shared across cultures and labeling was done by the authors (Section 3).
  • ad hoc to paper The expected distribution D(R) in the dataset is the appropriate neutral baseline for bias measurement
    PS and BFS compare model choices to D(R); a uniform-random model would deviate from D(R) and appear biased, so this baseline is not assumption-free (Section 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of EtiCor++: Towards Understanding Etiquettical Bias in LLMs." pith.science (2026). https://pith.science/paper/H37E4WUV

@misc{pith2026250608488,
  author       = {Pith},
  title        = {Pith review of: EtiCor++: Towards Understanding Etiquettical Bias in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H37E4WUV}},
  note         = {Machine review of arXiv:2506.08488}
}
read the original abstract

In recent years, researchers have started analyzing the cultural sensitivity of LLMs. In this respect, Etiquettes have been an active area of research. Etiquettes are region-specific and are an essential part of the culture of a region; hence, it is imperative to make LLMs sensitive to etiquettes. However, there needs to be more resources in evaluating LLMs for their understanding and bias with regard to etiquettes. In this resource paper, we introduce EtiCor++, a corpus of etiquettes worldwide. We introduce different tasks for evaluating LLMs for knowledge about etiquettes across various regions. Further, we introduce various metrics for measuring bias in LLMs. Extensive experimentation with LLMs shows inherent bias towards certain regions.

Figures

Figures reproduced from arXiv: 2506.08488 by the authors.

Figure 1
Figure 1. Regions covered under EtiCor++ across the majority of societies/regions) as well as localized (specific to a society/region). LLMs have been trained on almost the entire internet’s data (Villalobos et al., 2024) and have very likely picked up information about etiquettes in various societies. However, it remains to be evaluated if LLMs are able to understand intricate and subtle differences in social norms across cu… view at source ↗
Figure 2
Figure 2. Region wise Correlation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Odds Ratio analysis of etiquettes generated by Llama-3.1 for Europe vs India. The figure shows the words [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Distancing for Correct Option at Start 2 3 4 5 Number of Options 1.6 1.5 1.4 1.3 1.2 1.1 1.0 Closeness Gemma Llama Phi GPT-4O Gemini [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Closeness for Correct Option at End option as it is closest correlation-wise as well as meaning-wise to the correct choice. Closeness: The closeness trend in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Region-wise Correlation for General Eti [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 13
Figure 13. Figure 13: Distribution for ChatGPT4o Model [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 11
Figure 11. Figure 11: Distribution for Gemini Model [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Distribution for Gemma Model [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 8
Figure 8. Figure 8: Odds Ratio analysis of etiquettes generated by Llama-3.1 for Europe vs India. The figure shows the words [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Odds Ratio analysis of etiquettes generated by Llama-3.1 for East Asia vs Middle East Africa. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Odds Ratio analysis of etiquettes generated by Phi-3.5-mini for Europe vs Latin America. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 15
Figure 15. Figure 15: Distribution for Llama Model G Model Output Examples [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 13 canonical work pages

  1. [1]

    Mitchell Abrams and Matthias Scheutz. 2022. https://doi.org/10.18653/v1/2022.naacl-main.1 Social norms guide reference resolution . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1--11, Seattle, United States. Association for Computational Linguistics

    work page doi:10.18653/v1/2022.naacl-main.1 2022

  2. [2]

    Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O ' Neill, Ashutosh Modi, and Monojit Choudhury. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.882 Towards measuring and modeling `` culture '' in LLM s: A survey . In Proceedings of the 2024 Conference on Empirical Methods in Natural Lang...

    doi:10.18653/v1/2024.emnlp-main.882 2024

  3. [3]

    Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, and Monojit Choudhury. 2024. https://aclanthology.org/2024.lrec-main.560 Ethical reasoning and moral value alignment of LLM s depend on the language we prompt them in . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024...

    work page 2024

  4. [4]

    Alghamdi, Reem Masoud, Deema Alnuhait, Afnan Y

    Emad A. Alghamdi, Reem Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2025. https://aclanthology.org/2025.coling-main.579/ A ra T rust: An evaluation of trustworthiness for LLM s in A rabic . In Proceedings of the 31st International Conference on Computational Linguistics, pages 8664--8679, Abu Dhabi, UAE. Association for Co...

    2025

  5. [5]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. https://doi.org/10.18653/v1/2024.acl-long.671 Investigating cultural alignment of large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404--12422, Bangkok, Thailand. Association for Compu...

    doi:10.18653/v1/2024.acl-long.671 2024

  6. [6]

    Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hannaneh Hajishirzi, and Yejin Choi. 2022. https://doi.org/10.18653/v1/2022.naacl-main.439 Aligning to social norms and values in interactive narratives . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages...

    doi:10.18653/v1/2022.naacl-main.439 2022

  7. [7]

    Somnath Banerjee, Sayan Layek, Hari Shrawgi, Rajarshi Mandal, Avik Halder, Shanu Kumar, Sagnik Basu, Parag Agrawal, Rima Hazra, and Animesh Mukherjee. 2025. https://arxiv.org/abs/2410.12880 Navigating the cultural kaleidoscope: A hitchhiker's guide to sensitivity in large language models . Preprint, arXiv:2410.12880

    arXiv 2025

  8. [8]

    Yang Trista Cao, Anna Sotnikova, Hal Daum \'e III, Rachel Rudinger, and Linda Zou. 2022. https://doi.org/10.18653/v1/2022.naacl-main.92 Theory-grounded measurement of U . S . social stereotypes in E nglish language models . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language ...

    doi:10.18653/v1/2022.naacl-main.92 2022

Show all 54 references
  1. [9]

    Yong Cao, Min Chen, and Daniel Hershcovich. 2024. https://aclanthology.org/2024.findings-eacl.63 Bridging cultural nuances in dialogue agents through cultural value surveys . In Findings of the Association for Computational Linguistics: EACL 2024, pages 929--945, St. Julian ' ...

    2024

  2. [10]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  3. [11]

    Dipto Das, Shion Guha, and Bryan Semaan. 2023. https://doi.org/10.18653/v1/2023.c3nlp-1.8 Toward cultural bias evaluation datasets: The case of B engali gender, religious, and national identity . In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3N...

    2023 doi

  4. [12]

    Sunipa Dev, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prabhakaran. 2024. Building socio-culturally inclusive stereotype resources with community engagement. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red...

    2024

  5. [13]

    Xuan Long Do, Kenji Kawaguchi, Min-Yen Kan, and Nancy Chen. 2025. https://aclanthology.org/2025.coling-main.172/ Aligning large language models with human opinions through persona selection and value -- belief -- norm reasoning . In Proceedings of the 31st International Confer...

    2025

  6. [14]

    Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. 2022. https://doi.org/10.1145/3554727 A survey of natural language generation . ACM Comput. Surv., 55(8)

    2022 doi

  7. [15]

    Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.428 E ti C or: Corpus for analyzing LLM s for etiquettes . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6921--6931, Si...

    2023 doi

  8. [16]

    Yi Fung, Tuhin Chakrabarty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.941 NORMSAGE : Multi-lingual multi-cultural norm discovery from conversations on-the-fly . In Proceedings of the 2023 Conference on Empirical Meth...

    2023 doi

  9. [17]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. https://doi.org/10.1162/coli_a_00524 Bias and fairness in large language models: A survey . Computational Linguistics, 50(3):1097--1179

    2024 doi

  10. [18]

    Gemini. 2024. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models . Preprint, arXiv:2312.11805

    2024 arXiv

  11. [19]

    Google. 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . Preprint, arXiv:2408.00118

    2024 arXiv

  12. [20]

    Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders S gaard. 2022. https://doi.org/10.186...

    2022 doi

  13. [21]

    Jing Huang and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.509 Culturally aware natural language inference . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591--7609, Singapore. Association for Computational Linguistics

    2023 doi

  14. [22]

    Akshita Jha, Aida Mostafazadeh Davani, Chandan K Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. 2023. https://doi.org/10.18653/v1/2023.acl-long.548 S ee GULL : A stereotype benchmark with broad geo-cultural coverage leveraging generative models . In Proceedings of...

    2023 doi

  15. [23]

    Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi

    Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. https://arxiv.org/abs/2110.07574 Delphi: Towards machine ethics and norms . CoRR, abs/2110.07574

    2021 arXiv

  16. [24]

    A. Koch, R. Dotsch, C. Unkelbach, and H. Alves. 2016. https://doi.org/10.1037/pspa0000046 The abc of stereotypes about groups: agency/socioeconomic success, conservative–progressive beliefs, and communion. Journal of Personality and Social Psychology, 110:675--709

    2016 doi

  17. [25]

    Grgur Kovac, Masataka Sawayama, R \' e my Portelas, C \' e dric Colas, Peter Ford Dominey, and Pierre - Yves Oudeyer. 2023. https://doi.org/10.48550/ARXIV.2307.07870 Large language models as superpositions of cultural perspectives . CoRR, abs/2307.07870

    2023 doi arXiv

  18. [26]

    Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024 a . Culturellm: Incorporating cultural differences into large language models. In Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS)

    2024

  19. [27]

    Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024 b . Culturepark: Boosting cross-cultural understanding in large language models. In Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS)

    2024

  20. [28]

    Xuelin Liu, Pengyuan Liu, and Dong Yu. 2025. https://aclanthology.org/2025.coling-main.317/ What`s the most important value? INVP : IN vestigating the value priorities of LLM s through decision-making in social scenarios . In Proceedings of the 31st International Conference on...

    2025

  21. [29]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  22. [30]

    Meta. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  23. [31]

    Microsoft. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . Preprint, arXiv:2404.14219

    2024 arXiv

  24. [32]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. https://doi.org/10.18653/v1/2021.acl-long.416 S tereo S et: Measuring stereotypical bias in pretrained language models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inte...

    2021 doi

  25. [33]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.154 C row S -pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  26. [34]

    Ryan, Alan Ritter, and Wei Xu

    Tarek Naous, Michael J. Ryan, Alan Ritter, and Wei Xu. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.862 Having beer after prayer? measuring cultural bias in large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024 doi

  27. [35]

    Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. https://doi.org/10.1145/3543507.3583535 Extracting cultural commonsense knowledge at scale . In Proceedings of the ACM Web Conference 2023, WWW '23, page 1907–1917, New York, NY, USA. Association for ...

    2023

  28. [36]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  29. [37]

    Shramay Palta and Rachel Rudinger. 2023. https://doi.org/10.18653/v1/2023.findings-acl.631 FORK : A bite-sized test set for probing culinary cultural biases in commonsense reasoning models . In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952--99...

    2023 doi

  30. [38]

    Saurabh Kumar Pandey, Harshit Budhiraja, Sougata Saha, and Monojit Choudhury. 2025. https://aclanthology.org/2025.coling-demos.21/ CULTURALLY YOURS : A reading assistant for cross-cultural content . In Proceedings of the 31st International Conference on Computational Linguisti...

    2025

  31. [39]

    Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, and Xia Song. 2023. https://doi.org/10.18653/v1/2023.acl-long.856 Beyond E nglish-centric bitexts for better multilingual language representation learning . In Proceedings of the 61st...

    2023 doi

  32. [40]

    Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. https://arxiv.org/abs/2404.12464 Normad: A framework for measuring the cultural adaptability of large language models . Preprint, arXiv:2404.12464

    2024 arXiv

  33. [41]

    Nils Reimers and Iryna Gurevych. 2019 a . https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confere...

    2019 doi

  34. [42]

    Nils Reimers and Iryna Gurevych. 2019 b . https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  35. [43]

    Hari Shrawgi, Prasanjit Rath, Tushar Singhal, and Sandipan Dandapat. 2024. https://aclanthology.org/2024.eacl-long.111 Uncovering stereotypes in large language models: A task complexity-based approach . In Proceedings of the 18th Conference of the European Chapter of the Assoc...

    2024

  36. [44]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie - Yan Liu. 2020. https://proceedings.neurips.cc/paper/2020/hash/c3a690be93aa602ee2dc0ccab5b7b67e-Abstract.html Mpnet: Masked and permuted pre-training for language understanding . In Advances in Neural Information Processing S...

    2020

  37. [45]

    Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. arXiv preprint arXiv:1904.01172

    2019 arXiv

  38. [46]

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning (ICML)

    2024

  39. [47]

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.243 `` kelly is a warm person, joseph is a role model '' : Gender biases in LLM -generated reference letters . In Findings of the Associat...

    2023 doi

  40. [48]

    Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, and Kai-Wei Chang. 2023 b . https://doi.org/10.18653/v1/2023.findings-emnlp.648 Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems . In Findings of the Association for Computational...

    2023 doi

  41. [49]

    Shaoyang Xu, Yongqi Leng, Linhao Yu, and Deyi Xiong. 2024. https://arxiv.org/abs/2410.12971 Self-pluralising culture alignment for large language models . Preprint, arXiv:2410.12971

    2024 arXiv

  42. [50]

    Damin Zhang, Yi Zhang, Geetanjali Bihani, and Julia Rayz. 2025. https://aclanthology.org/2025.coling-main.529/ Hire me or not? examining language model`s behavior with occupation attributes . In Proceedings of the 31st International Conference on Computational Linguistics, pag...

    2025

  43. [51]

    Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao,...

    2024

  44. [52]

    Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.acl-long.429 N orm B ank: A knowledge bank of situational social norms . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V...

    2023 doi

  45. [53]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.