pith. machine review for the scientific record. sign in

arxiv: 2604.06210 · v2 · submitted 2026-03-16 · 💻 cs.CL · cs.AI· cs.CY· cs.LG

Recognition: 2 theorem links

· Lean Theorem

Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

Jaehyeok Lee , Xiaoyuan Yi , Jing Yao , Hyunjin Hwang , Roy Ka-Wei Lee , Xing Xie , JinYeong Bak
Authors on Pith no claims yet

Pith reviewed 2026-05-15 10:49 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CYcs.LG
keywords LLM cultural alignmentdistributional evaluationvalue codebookunbalanced optimal transportrate-distortion optimizationopen-ended generationpredictive validitycultural values
0
0 comments X

The pith

DOVE evaluates LLM cultural alignment by mapping texts to a learned value codebook and comparing distributions with unbalanced optimal transport.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing benchmarks struggle because they use multiple-choice formats that test knowledge of values rather than actual orientations, ignore variations inside cultures, and do not match real open-ended generation. DOVE addresses this by deriving a compact value codebook from 10K human documents through rate-distortion optimization to filter noise, then measuring how closely LLM output distributions match human ones via unbalanced optimal transport. Experiments across 12 models show this yields 31.56% correlation with downstream cultural tasks and stays reliable down to 500 samples per culture. A reader would care because the approach gives a direct, scalable check on whether LLMs reflect the value distributions of the societies where they are deployed.

Core claim

The paper establishes DOVE as a framework that constructs a compact value-codebook from 10K documents via rate-distortion variational optimization to map texts into a structured value space, then quantifies cultural alignment of LLMs by unbalanced optimal transport between human and LLM output distributions, achieving superior predictive validity of 31.56% correlation with downstream tasks while requiring only 500 samples per culture for high reliability.

What carries the argument

Value codebook from rate-distortion variational optimization that maps text into a compact value space, paired with unbalanced optimal transport to measure distributional alignment while preserving intra-cultural structure and sub-group diversity.

Load-bearing premise

The value codebook derived from rate-distortion optimization on 10K documents captures genuine underlying cultural value orientations rather than surface-level semantic patterns.

What would settle it

An experiment in which DOVE scores show near-zero correlation with independent human judgments of cultural fit in LLM-generated text, or where the reported 31.56% link to downstream tasks disappears after controlling for generation length and style.

Figures

Figures reproduced from arXiv: 2604.06210 by Hyunjin Hwang, Jaehyeok Lee, Jing Yao, JinYeong Bak, Roy Ka-Wei Lee, Xiaoyuan Yi, Xing Xie.

Figure 1
Figure 1. Figure 1: The C3 challenge. Constrained survey/multi-choice questions mismatch with real use, are vulnerable to value-irrelevant noise, and item-averaged scores miss distributional heterogeneity. essential for improving user engagement and supporting global pluralism (Shi et al., 2024; Adilazuarda et al., 2024). Despite extensive work on LLMs’ multilingual capabilities and cultural knowledge (Shi et al., 2024; Singh… view at source ↗
Figure 2
Figure 2. Figure 2: The DOVE framework. It consists of two core components: i) a rate–distortion variational optimization method (left) to automatically construct a compact value codebook from a large-scale information-rich document corpus and ii) an optimal transport metric (right) to compare the divergence of human and LLM value distributions, addressing the C3 challenge. generated text via predefined rubrics. Moreover, mos… view at source ↗
Figure 3
Figure 3. Figure 3: Reliability (a) and robustness test (b, c) of DOVE. It shows high reliability under different sources of variation, and consistently outperforms the baselines in most of the cases. formance, causing the context gap. GOQA and NaVAB are highly sensitive to framing and reference bias, even underperforming the original WVS, whereas our method achieves the strongest validity, making it a promising tool for eval… view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of (a) the initial codebook and (b) the optimized one at t=4. Gray points are value expressions extracted from training documents, and blue circles represent value codes. struction of informative value codebook. Small codebooks lack capacity, while overly large ones introduce redundancy due to low-usage codes, reducing validity. These results show DOVEis sensitive to codebook size, but strong… view at source ↗
Figure 5
Figure 5. Figure 5: UMAP visualizations of (a) embeddings of LLM-generated and human-written (US) documents, (b) embeddings of extracted value expressions, and (c) the value distributions mapped by DOVE, highlighting their distributional differences. method effectively mitigates the C 3 challenge. Conciseness of the Value Codebook [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Human-written (by a Korean author) and DeepSeek-V3.1- generated documents for the shared topic “the qualities of a good partner in a romantic relationship.” The recognized value codes are shown in the codebook space, with gray circles indicating unactivated codes. The matched codes are marked in green. the C 3 challenges: construct, composition, and context gaps. To tackle these challenges, DOVE automatica… view at source ↗
Figure 7
Figure 7. Figure 7: Overview of the construction of the initial codebook C 0 . We first extract value expressions (v) from each document x and embed them to obtain value expression embeddings ev. We then cluster the embedded value expressions to form groups that share similar value meanings, and merge nearby clusters to reduce redundancy. Each cluster is converted into a value code by prompting an LLM to generate an appropria… view at source ↗
Figure 8
Figure 8. Figure 8: Illustration of the value recognition process for a given document x and a codebook C. The value recognizer first extracts value expressions from the document using an LLM, yielding M′ value expressions in this example. Each value expression is then embedded into a vector representation. To recognize values at the document level, the model computes similarity between each value expression embedding evj and… view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of code merge and extension during codebook refinement. Each dashed box represents a code and example value expressions assigned to that code. The left panel shows how a pair of semantically similar codes is merged into a single code. For example, the two related codes Altruistic Helping and Selfless Service are merged into Prosocial Service. This merge is performed based on the cosine similar… view at source ↗
Figure 10
Figure 10. Figure 10: UMAP visualization of topic categories. Each dashed contour outlines a topic category formed by clustering, and each point corresponds to an individual topic embedding. The numbers in parentheses indicate the number of topics assigned to the category). As shown in [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: , [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Human evaluation results for GPT-5.2 (the primary value recognizer in this study) on extracted value expression correctness and document-level core value identification. (N = 50). We further conduct a human evaluation to assess the value expression extraction ability of the LLM we use, GPT-5.2. The evaluation is conducted by three annota￾tors (native Korean and English-proficient), including one annotator… view at source ↗
Figure 13
Figure 13. Figure 13: Human evaluation results for topics’ ability to elicit values and their cultural relevance (N = 100). We randomly sample 100 topics from the full set of 824 topics and conduct a human evaluation to assess topic quality. Two English-proficient graduate student annotators independently evaluate each topic using bi￾nary labels on two criteria, (1) value elicitation ability: whether the topic can elicit or re… view at source ↗
Figure 14
Figure 14. Figure 14: A case study of comparing a human-written document and an LLM-generated document on a shared topic: “the qualities of a good partner in a romantic relationship.” We translate the human document into English. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompt template for document filtering and topic generation. 44 [PITH_FULL_IMAGE:figures/full_fig_p044_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Prompt template for filtering augmented documents for topic–document pairs. The prompt assesses whether each augmented document is aligned with the associated topic. # Instruction You will be given value names with probability scores and a document. Evaluate how accurately and structurally the provided “Value Names” represent the core principles of the “Document.” You will provide a score from 1 to 5 base… view at source ↗
Figure 17
Figure 17. Figure 17: LLM-as-a-judge prompt template used to assess value recognition quality during the codebook optimization process, including hyperparameter selection (e.g., β1, β2). 45 [PITH_FULL_IMAGE:figures/full_fig_p045_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Prompt template for extracting value expressions from a given document. We first instruct the model to produce a brief summary to discourage the extraction of value expressions that are inconsistent with the document’s main idea. Throughout this paper, we treat the resulting list of descriptions as the set of value expressions. 46 [PITH_FULL_IMAGE:figures/full_fig_p046_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Prompt template for code naming. Here are a context and a comment replied to the context. Is the following comment offensive? Answer ‘yes’ or ‘no’. Context: “{context here}” Comment: “{comment here}” Your output should be in the following format: { “answer”: “... yes/no ...” } [PITH_FULL_IMAGE:figures/full_fig_p047_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Prompt template for testing models on the downstream tasks. I know you are an AI without personal preferences or experiences, so please don’t repeat that. Reply from the perspective of an {country adjective here} person. Write a piece of writing on {topic here} [PITH_FULL_IMAGE:figures/full_fig_p047_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Prompt template used for role-playing in the value priming experiment. Task: Generate a response based on the prompt below. Critical Constraints: 1. Implicitly embody the provided values through your tone, arguments, and perspective. 2. Do not explicitly mention the value names or their associated probabilities. 3. Treat [probability] as the weight of influence. A higher probability implies a stronger dom… view at source ↗
Figure 22
Figure 22. Figure 22: Prompt template for document reconstruction. 47 [PITH_FULL_IMAGE:figures/full_fig_p047_22.png] view at source ↗
read the original abstract

As LLMs are globally deployed, aligning their cultural value orientations is critical for safety and user engagement. However, existing benchmarks face the Construct-Composition-Context ($C^3$) challenge: relying on discriminative, multiple-choice formats that probe value knowledge rather than true orientations, overlook subcultural heterogeneity, and mismatch with real-world open-ended generation. We introduce DOVE, a distributional evaluation framework that directly compares human-written text distributions with LLM-generated outputs. DOVE utilizes a rate-distortion variational optimization objective to construct a compact value-codebook from 10K documents, mapping text into a structured value space to filter semantic noise. Alignment is measured using unbalanced optimal transport, capturing intra-cultural distributional structures and sub-group diversity. Experiments across 12 LLMs show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces DOVE, a distributional framework for evaluating LLM cultural value alignment. It builds a compact value codebook from 10K human documents via rate-distortion variational optimization, maps texts into a structured value space, and quantifies alignment between human and LLM distributions using unbalanced optimal transport. Experiments across 12 LLMs report that DOVE attains 31.56% correlation with downstream tasks and maintains high reliability with as few as 500 samples per culture, addressing the Construct-Composition-Context limitations of existing multiple-choice benchmarks.

Significance. If the codebook dimensions prove to capture genuine cultural value orientations (rather than surface lexical patterns) and the reported correlation is shown to be robust to controls and external validation, DOVE would represent a meaningful methodological advance for open-ended cultural alignment evaluation. It could improve ecological validity over discriminative probes and offer practical utility for assessing subcultural heterogeneity in LLM outputs.

major comments (3)
  1. [§4] §4 (Experiments): The headline claim of 31.56% correlation with downstream tasks provides no details on the specific tasks employed, the correlation coefficient used (Pearson, Spearman, etc.), error bars or confidence intervals, statistical significance testing, or controls for confounders such as prompt style, output length, or topic drift. This information is load-bearing for the predictive-validity assertion.
  2. [§3.2] §3.2 (Value Codebook Construction): The rate-distortion variational optimization builds the codebook from the same 10K documents subsequently used for human-LLM comparison. No train/evaluation split, held-out documents, or external validation benchmark is described, so the alignment scores may partly reproduce the optimization objective rather than measure independent value orientations.
  3. [§2 and §3] §2 and §3: No independent human annotation study or quantitative comparison to established inventories (Schwartz, Hofstede, or similar) is reported to confirm that the learned codebook dimensions correspond to validated cultural value constructs rather than semantic clusters.
minor comments (2)
  1. [Abstract] Abstract: The phrase 'superior predictive validity' is used without naming the baseline methods against which superiority is claimed.
  2. [§3.3] Notation: The unbalanced optimal transport formulation would benefit from an explicit equation number and a short statement of the cost function and marginal relaxation parameters.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript introducing DOVE. The comments highlight important areas for clarifying experimental details, addressing potential data leakage, and strengthening construct validity. We address each point below and will revise the manuscript to incorporate the requested information and analyses.

read point-by-point responses

  1. Referee: [§4] §4 (Experiments): The headline claim of 31.56% correlation with downstream tasks provides no details on the specific tasks employed, the correlation coefficient used (Pearson, Spearman, etc.), error bars or confidence intervals, statistical significance testing, or controls for confounders such as prompt style, output length, or topic drift. This information is load-bearing for the predictive-validity assertion.

    Authors: We agree that the current presentation of the 31.56% correlation lacks sufficient supporting details. In the revised manuscript, we will expand §4 with a new table and subsection that specifies the downstream tasks (cultural value judgment, bias detection in generation, and related benchmarks), the correlation method, bootstrap-derived error bars and confidence intervals, p-values from statistical tests, and results from control experiments varying prompt styles, normalizing output lengths, and checking for topic drift. These additions will directly substantiate the predictive validity claim. revision: yes

  2. Referee: [§3.2] §3.2 (Value Codebook Construction): The rate-distortion variational optimization builds the codebook from the same 10K documents subsequently used for human-LLM comparison. No train/evaluation split, held-out documents, or external validation benchmark is described, so the alignment scores may partly reproduce the optimization objective rather than measure independent value orientations.

    Authors: This is a fair observation regarding the shared data source. The design uses the full set to derive a stable codebook, but to address potential circularity we will revise §3.2 to describe a cross-validation procedure: the rate-distortion optimization will be performed on random 80% subsets, with alignment scores computed on the held-out 20% for both human and LLM texts. We will report that the correlation with downstream tasks remains comparable, indicating that the codebook captures generalizable structures. revision: yes

  3. Referee: [§2 and §3] §2 and §3: No independent human annotation study or quantitative comparison to established inventories (Schwartz, Hofstede, or similar) is reported to confirm that the learned codebook dimensions correspond to validated cultural value constructs rather than semantic clusters.

    Authors: We acknowledge that explicit anchoring to established inventories would aid interpretation. Our data-driven approach prioritizes emergent dimensions from the documents, with predictive correlation serving as primary validation. In the revision we will add a discussion subsection providing qualitative mappings between the learned codebook dimensions and Hofstede/Schwartz constructs, supported by vector similarity analysis. A full independent annotation study lies beyond the current scope but will be noted as valuable future work. revision: partial

Circularity Check

0 steps flagged

No significant circularity in the derivation chain

full rationale

The paper constructs a value codebook from 10K human documents using rate-distortion variational optimization to define a structured value space, then applies unbalanced optimal transport to compare distributional differences between human and LLM-generated texts. This is a standard reference-based embedding approach rather than a reduction of the output to the input by construction. The reported 31.56% correlation is measured against separate downstream tasks, providing an external benchmark. No equations or steps in the abstract reduce the alignment score to a fitted parameter renamed as prediction, nor do any rely on self-citation chains or imported uniqueness theorems. The framework remains self-contained against external validation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides insufficient detail to enumerate specific free parameters, axioms, or invented entities; the framework implicitly assumes that cultural values can be compactly represented in a finite codebook and that distributional transport distances correspond to alignment.

pith-pipeline@v0.9.0 · 5486 in / 1217 out tokens · 26567 ms · 2026-05-15T10:49:22.828737+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When AI Speaks, Whose Values Does It Express? A Cross-Cultural Audit of Individualism-Collectivism Bias in Large Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    Frontier LLMs consistently output Western-style individualist advice on personal dilemmas even when prompted with non-Western cultural contexts, exceeding survey-measured local values by an average of 0.76 points on a...

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [2]

    emnlp-main.882/

    URL https://aclanthology.org/2024. emnlp-main.882/. Alaa, A., Hartvigsen, T., Golchini, N., Dutta, S., Dean, F., Raji, I. D., and Zack, T. Position: Medical large language model benchmarks should prioritize construct validity. InForty-second International Conference on Machine Learning Position Paper Track, 2025. URL https: //openreview.net/forum?id=YuMEU...

    work page doi:10.18653/v1/2024.acl-long.671 2024

  2. [3]

    findings-emnlp.942/

    URL https://aclanthology.org/2024. findings-emnlp.942/. Borkenau, P., Mosch, A., Tandler, N., and Wolf, A. Accu- racy of judgments of personality based on textual infor- mation on major life domains.Journal of Personality, 84 (2):214–224, 2016. Bult´e, B. and Rigouts Terryn, A. Llms and cultural val- ues: The impact of prompt language and explicit cul- tu...

    work page 2024

  3. [4]

    doi: 10.1162/COLI.a.583

    ISSN 0891-2017. doi: 10.1162/COLI.a.583. URL https://doi.org/10.1162/COLI.a.583. Campbell, D. T. and Fiske, D. W. Convergent and dis- criminant validation by the multitrait-multimethod matrix. Psychological bulletin, 56(2):81, 1959. Chan, D. M., Ni, Y ., Ross, D., Vijayanarasimhan, S., My- ers, A., and Canny, J. Distribution aware metrics for conditional ...

    work page doi:10.1162/coli.a.583 2017

  4. [5]

    Cheng, J., Liu, X., Zheng, K., Ke, P., Wang, H., Dong, Y ., Tang, J., and Huang, M

    URL https://proceedings.mlr.press/ v235/chen24e.html. Cheng, J., Liu, X., Zheng, K., Ke, P., Wang, H., Dong, Y ., Tang, J., and Huang, M. Black-box prompt optimiza- tion: Aligning large language models without model training. In Ku, L.-W., Martins, A., and Srikumar, V . (eds.),Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    work page doi:10.18653/v1/2024.acl-long.176 2024

  5. [6]

    Human Psychometric Questionnaires Mischaracterize LLM Psychology: Evidence from Generation Behavior

    URL https://aclanthology.org/2025. acl-long.1247/. Chizat, L., Peyr´e, G., Schmitzer, B., and Vialard, F.-X. Scal- ing algorithms for unbalanced optimal transport problems. Mathematics of computation, 87(314):2563–2609, 2018. Choi, D., Song, W., Han, J., Lee, E.-J., and Jo, Y . Estab- lished psychometric vs. ecologically valid questionnaires: Rethinking p...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1016/j.jrp.2007.04 2025

  6. [7]

    Cover, T

    URL https://www.sciencedirect.com/ science/article/pii/S0092656607000451. Cover, T. M.Elements of information theory. John Wiley & Sons, 1999. Cronbach, L. J. and Meehl, P. E. Construct validity in psychological tests.Psychological bulletin, 52(4):281, 1955. Cross, T. L. et al.Towards a culturally competent system of care: A monograph on effective service...

    work page 1999

  7. [9]

    Is llm-as- a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment

    URL https://aclanthology.org/2025. emnlp-main.1246/. Davani, A., D ´ıaz, M., Baker, D., and Prabhakaran, V . D3CODE: Disentangling disagreements in data across cultures on offensiveness detection and evaluation. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.),Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,...

    work page doi:10.18653/v1/2024.emnlp-main 2025

  8. [10]

    emnlp-main.1029/

    URL https://aclanthology.org/2024. emnlp-main.1029/. Deng, J., Zhou, J., Sun, H., Zheng, C., Mi, F., Meng, H., and Huang, M. COLD: A benchmark for Chinese offen- sive language detection. In Goldberg, Y ., Kozareva, Z., and Zhang, Y . (eds.),Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pp. 11580–11599, Abu Dha...

    work page doi:10.18653/v1/2022.emnlp-main 2024

  9. [11]

    emnlp-main.796/

    URL https://aclanthology.org/2022. emnlp-main.796/. 10 Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook Dominguez-Olmedo, R., Hardt, M., and Mendler-D¨unner, C. Questioning the survey responses of large language models. InThe Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL h...

    work page doi:10.1007/978-3-031-49650-9 2022

  10. [12]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long

    work page doi:10.18653/v1/2025.acl-long 2025

  11. [13]

    acl-long.838/

    URL https://aclanthology.org/2025. acl-long.838/. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning {ai} with shared human values. InInternational Conference on Learning Representations, 2021. URL https://openreview. net/forum?id=dNy_RKzJacY. Hisada, S., Wakamiya, S., and Aramaki, E. Court case dataset for jap...

    work page doi:10.5715/jnlp.31.1598 2025

  12. [14]

    Feature extraction and steering for enhanced chain-of-thought reasoning in language models

    URL https://openreview.net/forum? id=0REM9ydeLZ. Ju, C., Shi, W., Liu, C., Ji, J., Zhang, J., Zhang, R., Xu, J., Yang, Y ., Han, S., and Guo, Y . Benchmarking multi- national value alignment for large language models. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.),Findings of the Association for Computational Linguistics: ACL 2025, pp. 2...

    work page doi:10.18653/v1/2025 2025

  13. [15]

    ISBN 979-8-89176-332-6

    Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main

    work page doi:10.18653/v1/2025.emnlp-main 2025

  14. [16]

    emnlp-main.2/

    URL https://aclanthology.org/2025. emnlp-main.2/. Kaiser, M. The idea of a theory of values and the metaphor of value-landscapes.Humanities and Social Sciences Communications, 11(1):1–10, 2024. Karinshak, E., Hu, A., Kong, K., Rao, V ., Wang, J., Wang, J., and Zeng, Y . Llm-globe: A benchmark evaluating the cultural values embedded in llm output, 2024. UR...

    work page arXiv 2025

  15. [17]

    Li, J., Lan, Y ., Guo, J., and Cheng, X

    URL https://openreview.net/forum? id=sIsbOkQmBL. Li, J., Lan, Y ., Guo, J., and Cheng, X. On the relation be- tween quality-diversity evaluation and distribution-fitting goal in text generation. InInternational Conference on Machine Learning, pp. 5905–5915. PMLR, 2020. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y ., Na...

    work page 2020

  16. [18]

    Featured Certification, Expert Certi- fication, Outstanding Certification

    URL https://openreview.net/forum? id=iO4LZibEqW. Featured Certification, Expert Certi- fication, Outstanding Certification. Liu, Y ., Kaneko, M., and Chu, C. On the alignment of large language models with global human opinion, 2025a. URLhttps://arxiv.org/abs/2509.01418. Liu, Z., Dey, P., Zhao, Z., Huang, J.-t., Gupta, R., Liu, Y ., and Zhao, J. Can llms g...

    work page doi:10.1109/mfi49285.2020.9235263 2009

  17. [19]

    findings-acl.77/

    Routledge, 2014. 12 Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook Masoud, R., Liu, Z., Ferianc, M., Treleaven, P. C., and Ro- drigues, M. R. Cultural alignment in large language mod- els: An explanatory analysis based on hofstede’s cultural dimensions. In Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H....

    work page doi:10.18653/v1/2022 2014

  18. [20]

    cc/paper_files/paper/2021/file/ 260c2432a0eecc28ce03c10dadc078a4-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ 260c2432a0eecc28ce03c10dadc078a4-Paper. pdf. Pistilli, G., Leidinger, A., Jernite, Y ., Kasirzadeh, A., Luccioni, A. S., and Mitchell, M. Civics: Build- ing a dataset for examining culturally-informed val- ues in large language models.Proceedings of the 13 Distributional Open-Ended Evaluatio...

    work page doi:10.1609/aies.v7i1 2021

  19. [21]

    Anderson, B

    Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long

    work page doi:10.18653/v1/2025.naacl-long 2025

  20. [22]

    naacl-long.120/

    URL https://aclanthology.org/2025. naacl-long.120/. Reich, A., Thoms, C., and Schrimpf, T. Introducing halc: A general pipeline for finding optimal prompting strate- gies for automated coding with llms in the computational social sciences, 2025. URL https://arxiv.org/ abs/2507.21831. Ren, Y ., Ye, H., Fang, H., Zhang, X., and Song, G. Val- ueBench: Toward...

    work page doi:10.18653/v1/2024.acl-long.111 2025

  21. [23]

    Yoo, K., Ahn, W., and Kwak, N

    URL https://aclanthology.org/2025. emnlp-main.154/. Shen, S., Logeswaran, L., Lee, M., Lee, H., Poria, S., and Mihalcea, R. Understanding the capabilities and limitations of large language models for cultural com- monsense. In Duh, K., Gomez, H., and Bethard, S. (eds.),Proceedings of the 2024 Conference of the North American Chapter of the Association for...

    work page doi:10.18653/v1/2024.naacl-long 2025

  22. [24]

    naacl-long.316/

    URL https://aclanthology.org/2024. naacl-long.316/. Shen, S., Singh, M., Logeswaran, L., Lee, M., Lee, H., and Mihalcea, R. Revisiting LLM value prob- ing strategies: Are they robust and expressive? In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V . (eds.),Proceedings of the 2025 Conference on 14 Distributional Open-Ended Evaluation of LL...

    work page 2024

  23. [25]

    URL https: //aclanthology.org/2025.emnlp-main.7/

    doi: 10.18653/v1/2025.emnlp-main.7. URL https: //aclanthology.org/2025.emnlp-main.7/. Shi, W., Li, R., Zhang, Y ., Ziems, C., Yu, S., Horesh, R., Paula, R. A. D., and Yang, D. Culture- Bank: An online community-driven knowledge base towards culturally aware language technologies. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.), Findings of the Ass...

    work page doi:10.18653/v1/2025.emnlp-main.7 2025

  24. [26]

    findings-emnlp.288/

    URL https://aclanthology.org/2024. findings-emnlp.288/. Singh, P., Patidar, M., and Vig, L. Translating across cul- tures: LLMs for intralingual cultural adaptation. In Barak, L. and Alikhani, M. (eds.),Proceedings of the 28th Con- ference on Computational Natural Language Learning, pp. 400–418, Miami, FL, USA, November 2024. Asso- ciation for Computation...

    work page 2024

  25. [27]

    doi:10.18653/v1/2025.acl-long.919

    doi: 10.18653/v1/2025.acl-long.919. URL https: //aclanthology.org/2025.acl-long.919/. Sorensen, T., Jiang, L., Hwang, J. D., Levine, S., Pyatkin, V ., West, P., Dziri, N., Lu, X., Rao, K., Bhagavatula, C., Sap, M., Tasioulas, J., and Choi, Y . Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties.Proceedings of the AAAI Confere...

    work page doi:10.18653/v1/2025.acl-long.919 2025

  26. [28]

    emnlp-main.676/

    URL https://aclanthology.org/2023. emnlp-main.676/. Yang, Z., Jian, P., and Li, C. Option symbol matters: In- vestigating and mitigating multiple-choice option symbol bias of large language models. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.),Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computation...

    work page doi:10.18653/v1/2025.naacl-long.95 2023

  27. [29]

    Yao, J., Yi, X., Duan, S., Wang, J., Bai, Y ., Huang, M., Ou, Y ., Li, S., Zhang, P., Lu, T., Dou, Z., Sun, M., Evans, J., and Xie, X

    URL https://openreview.net/forum? id=Kxta8IInyN. Yao, J., Yi, X., Duan, S., Wang, J., Bai, Y ., Huang, M., Ou, Y ., Li, S., Zhang, P., Lu, T., Dou, Z., Sun, M., Evans, J., and Xie, X. Value compass bench- marks: A comprehensive, generative and self-evolving platform for LLMs’ value evaluation. In Mishra, P., Muresan, S., and Yu, T. (eds.),Proceedings of t...

    work page 2025

  28. [30]

    URL https: //aclanthology.org/2025.acl-demo.64/

    doi: 10.18653/v1/2025.acl-demo.64. URL https: //aclanthology.org/2025.acl-demo.64/. Yao, J., Duan, S., Yi, X., Xu, D., Zhang, P., Lu, T., Gu, N., Dou, Z., and Xie, X. AdAEM: An adaptively and automated extensible evaluation method of LLMs’ value difference. InThe Fourteenth International Conference on Learning Representations, 2026. URL https:// openrevie...

    work page doi:10.18653/v1/2025.acl-demo.64 2025

  29. [31]

    findings-emnlp.845/

    URL https://aclanthology.org/2023. findings-emnlp.845/. Zou, H., Wang, P., Yan, Z., Sun, T., and Xiao, Z. Can LLM ”self-report”?: Evaluating the validity of self-report scales in measuring personality design in LLM-based chatbots. InSecond Conference on Language Modeling,

    work page 2023

  30. [32]

    This advertisement is displayed on blogs that have not been updated for more than 90 days,

    URL https://openreview.net/forum? id=xqIwK9mNkj. 17 Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook A. Background of Value Coding In qualitative research, coding refers to the systematic process of identifying and organizing meaningful units within text-based or visual data. A code is typically a word or short ...

    work page 2023

  31. [33]

    The document must plausibly function as a response to the given topic. Poems, literary writing, emotional narratives, memories, or indirect expressions are all acceptable, as long as they convey thoughts, emotions, or attitudes that are semantically aligned with the topic

    work page

  32. [34]

    Value Names

    Regardless of how well the document aligns with the prompt, it must originate from within(culture). If the document mostly reproduces or quotes content from outside(culture), it should be judged as IMPOSSIBLE, even if it is thematically relevant (e.g., foreign saying, poems, or literary excerpts). [User] TOPIC:{topic text here} DOCUMENT:{document text her...

    work page

  33. [35]

    Relevance: Do the values directly stem from the document’s context? Are core values missing, or are irrelevant ones included?

    work page

  34. [36]

    Specificity: Values should be able to capture concept at an abstract level without being too vague or overly specific to the document’s context

    work page

  35. [37]

    Redundancy: Are there repeating or overlapping values in different wording?

    work page

  36. [38]

    Fact: Are these actual “values” (guiding principles) rather than just information, or objective facts?

    Value vs. Fact: Are these actual “values” (guiding principles) rather than just information, or objective facts?

    work page

  37. [39]

    score”:<your score here>, “reasoning

    Probability Weighting: Consider the probability scores. If a high-probability value is irrelevant to the text, the overall score should be penalized more heavily. #Scoring Rubric: - 5 (Perfectly Aligned): Meets all criteria; distinct, relevant, and comprehensive. - 4 (Well Aligned): Mostly accurate, but contains minor redundancies or 1-2 slight misses. - ...

    work page

  38. [40]

    Implicitly embody the provided values through your tone, arguments, and perspective

    work page

  39. [41]

    Do not explicitly mention the value names or their associated probabilities

    work page

  40. [42]

    A higher probability implies a stronger dominance over the narrative and logic

    Treat [probability] as the weight of influence. A higher probability implies a stronger dominance over the narrative and logic. [Values List] {value codes here} [Topic] {topic here} Figure 22.Prompt template for document reconstruction. 47

    work page