pith. sign in

arxiv: 2605.16193 · v1 · pith:UIWSTUQCnew · submitted 2026-05-15 · 💻 cs.CL · cs.CY

Improving Cross-Cultural Survey Simulation with Calibrated Value Personas

Axel Abels , Elias Fernandez Domingos , Apurva Shah , Tom Lenaerts This is my paper

Pith reviewed 2026-05-20 18:40 UTC · model grok-4.3

classification 💻 cs.CL cs.CY
keywords LLM survey simulationvalue-based personascross-cultural predictionpopulation aggregationcalibration procedurecultural value dimensionsresponse diversity
0
0 comments X

The pith

Value profiles sampled from target populations let LLMs simulate survey responses with lower error across cultures.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models struggle to reproduce how opinions differ across cultures in survey simulations. This paper replaces generic demographic prompts with textual personas built directly from responses on core cultural value dimensions. Sampling many such personas from observed population distributions and averaging the model outputs produces aggregate predictions that track real human data more closely. A separate calibration step increases the spread of individual answers while keeping the central tendency intact. Gains are largest for countries that appear less often in training data, shrinking the gap to better-represented populations.

Core claim

Constructing personas from short textual descriptors of cultural values drawn from target survey distributions, then aggregating LLM replies across many sampled personas, yields population-level opinion estimates that reduce prediction error relative to sociodemographic or personality-based baselines; a calibration routine further raises response diversity without altering the estimated aggregate opinions.

What carries the argument

Value-based persona construction that converts survey answers on cultural dimensions into textual descriptors, combined with distribution-aware sampling and a post-hoc calibration step for diversity.

If this is right

  • Population-level predictions become grounded in empirically observed value distributions rather than model priors.
  • Prediction error decreases across countries, with the largest drops in populations underrepresented in training data.
  • Response distributions produced by the calibrated personas more closely match the diversity seen in human surveys.
  • The performance gap narrows between countries aligned with dominant training priors and those that are less represented.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same sampling-plus-aggregation logic could be tested on non-survey tasks such as predicting voting patterns or consumer choices within the same cultural groups.
  • If the textual descriptors prove portable, the approach might reduce the need for country-specific fine-tuning when deploying LLMs in new regions.
  • Repeating the calibration on finer demographic slices inside each country would test whether the diversity gain holds at the subgroup level.

Load-bearing premise

Short textual descriptions taken from survey responses on core cultural dimensions are sufficient for an LLM to generate answer distributions that, when aggregated over many sampled personas, match the actual spread of human responses in that population.

What would settle it

Applying the method to a held-out country with fresh survey data and finding no reduction in mean absolute error versus standard persona prompting, or finding that the calibrated responses shift the estimated population opinions, would falsify the central claim.

Figures

Figures reproduced from arXiv: 2605.16193 by Apurva Shah, Axel Abels, Elias Fernandez Domingos, Tom Lenaerts.

Figure 1
Figure 1. Figure 1: A. Inglehart-Welzel Cultural Map with selected countries highlighted with black borders. Markers with blue borders show the expected position of Gemma-4-31B, estimated from responses to the corresponding WVS survey items. Responses are elicited by prompting the model to either simulate a “typical human”, or to answer as the average person from specific countries (see Appendix A.1). Country-specific markers… view at source ↗
Figure 2
Figure 2. Figure 2: Mean absolute error (MAE) between model predictions and human responses across countries, models, and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: MAE as a function of the number of sampled [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of normalized variance by method, compared to human responses. Each boxplot summarizes [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mean absolute error (MAE) between model predictions and human survey responses across countries, models, [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean absolute error (MAE) of value-based personas as a function of the generator choices. Significance [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-item Shapley values across countries. Negative values indicate that including the item improves perfor [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Per-question comparison of MAE for value prompting versus country prompting. Each point corresponds to [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: MAE and Wasserstein distance as a function of temperature for the two methods described in Section 4.4. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of normalized variance by method, compared to human responses. Each boxplot summarizes [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Mean Absolute Error (MAE) of model predictions relative to human survey responses across countries and [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
read the original abstract

Large language models (LLMs) are increasingly used to simulate human opinions and survey responses, but their ability to reproduce population responses across cultures remains limited. Existing persona-based prompting methods typically rely on sociodemographic or personality traits, which are only indirect proxies for the values that shape human responses. We propose a value-based persona construction method that derives textual descriptors from survey responses capturing core cultural dimensions. By sampling value profiles from target populations and aggregating LLM responses across personas, we obtain population-level predictions grounded in observed value distributions. We further introduce a calibration procedure that improves response diversity while preserving estimated opinions. We show that our approach reduces prediction error across countries, with the largest improvements observed in underrepresented populations. This substantially narrows the performance gap between countries aligned with dominant LLM priors and those that are less represented in training data, while also yielding response distributions that closely match human diversity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes deriving textual value descriptors from survey items on core cultural dimensions (e.g., Hofstede-style axes), sampling personas from the empirical distribution of these profiles in target countries, prompting LLMs with the resulting personas, and aggregating the outputs to obtain population-level response distributions. A calibration step is introduced to increase response diversity while preserving mean opinions. The central empirical claim is that this value-based approach reduces prediction error relative to sociodemographic or personality-based baselines, with the largest gains in countries underrepresented in LLM training data, and produces response distributions closer to human survey variance.

Significance. If the quantitative improvements and diversity matching hold under rigorous validation, the work would provide a concrete, survey-grounded alternative to indirect persona prompting and could narrow the well-documented performance gap between WEIRD and non-WEIRD populations in LLM opinion simulation. The grounding in externally observed value distributions rather than fitted parameters is a methodological strength that avoids obvious circularity.

major comments (2)
  1. [§3.2 and §4.1] §3.2 and §4.1: The claim that short textual descriptors derived from a small number of cultural-dimension items are sufficient for the marginal LLM response distribution to recover the full human answer distribution on target questions is load-bearing for the entire method, yet the manuscript provides no direct test (e.g., comparison of per-question variance or KL divergence between persona-aggregated and human distributions before/after calibration). If the descriptors collapse variance or align too strongly with training-data priors, aggregation cannot recover the target distribution regardless of sampling.
  2. [§5, Table 3] §5, Table 3: The reported reductions in prediction error across countries are presented without error bars, statistical significance tests, or explicit comparison to a non-calibrated value-persona baseline; it is therefore impossible to determine whether the calibration step itself drives the gains or whether the improvements are reliable for the underrepresented-country subset that is the paper's primary motivation.
minor comments (2)
  1. [§3.3] Notation for the calibration parameters is introduced without a clear equation or pseudocode block; a single-line definition would improve reproducibility.
  2. [§3.1] The manuscript cites the source surveys for value profiles but does not state the exact number of items or response scales used to construct each descriptor; this detail belongs in §3.1.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive comments on our manuscript. We address each of the major comments below and indicate the revisions we will make to strengthen the paper.

read point-by-point responses

  1. Referee: [§3.2 and §4.1] §3.2 and §4.1: The claim that short textual descriptors derived from a small number of cultural-dimension items are sufficient for the marginal LLM response distribution to recover the full human answer distribution on target questions is load-bearing for the entire method, yet the manuscript provides no direct test (e.g., comparison of per-question variance or KL divergence between persona-aggregated and human distributions before/after calibration). If the descriptors collapse variance or align too strongly with training-data priors, aggregation cannot recover the target distribution regardless of sampling.

    Authors: We agree that direct tests of distributional fidelity, such as KL divergence or per-question variance comparisons, would provide additional support for the claim that the value descriptors are sufficient. Our original evaluation focused on mean absolute error for population-level predictions and on matching the overall variance of responses to human surveys. To address this concern rigorously, we will include in the revised manuscript an explicit analysis comparing the full response distributions using KL divergence and variance metrics before and after calibration. This will help verify that the short descriptors do not unduly collapse the variance and that aggregation can recover the target human distributions. revision: yes

  2. Referee: [§5, Table 3] §5, Table 3: The reported reductions in prediction error across countries are presented without error bars, statistical significance tests, or explicit comparison to a non-calibrated value-persona baseline; it is therefore impossible to determine whether the calibration step itself drives the gains or whether the improvements are reliable for the underrepresented-country subset that is the paper's primary motivation.

    Authors: We acknowledge the importance of statistical rigor in reporting the improvements. The original Table 3 presented point estimates of error reductions. In the revised version, we will add error bars computed via bootstrapping over the survey items or countries, include statistical significance tests (e.g., paired t-tests) comparing our method to baselines, and explicitly report results for the value-persona method without the calibration step. This will allow readers to assess the specific contribution of calibration and the reliability of gains in underrepresented countries. revision: yes

Circularity Check

0 steps flagged

No significant circularity; predictions grounded in external survey distributions

full rationale

The paper's core method samples value profiles directly from observed survey data on cultural dimensions and aggregates LLM responses over those empirical distributions. This is not a fitted parameter renamed as prediction, nor a self-definitional loop. The calibration step adjusts diversity on top of the base sampling without re-encoding the target responses. No load-bearing self-citation or uniqueness theorem is invoked to force the result. The derivation chain remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim depends on the assumption that value profiles extracted from surveys can be turned into effective textual prompts and that averaging LLM outputs over samples from the empirical distribution yields population-level estimates. The calibration step introduces at least one adjustable procedure whose exact parameters are not specified in the abstract.

free parameters (1)
  • calibration parameters
    The calibration procedure that trades off diversity against opinion preservation is described but not detailed; it likely contains tunable parameters or thresholds.
axioms (1)
  • domain assumption Textual descriptors derived from cultural-value survey items are sufficient to condition an LLM so that its responses, when aggregated over samples from the observed value distribution, approximate the true population response distribution.
    This premise is required to obtain the claimed population-level predictions from persona aggregation.

pith-pipeline@v0.9.0 · 5681 in / 1373 out tokens · 44257 ms · 2026-05-20T18:40:20.100274+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

  1. [1]

    Out of one, many: Using language models to simulate hu- man samples.Political Analysis, 31(3):337–351,

    [Argyleet al., 2023 ] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christo- pher Rytting, and David Wingate. Out of one, many: Using language models to simulate hu- man samples.Political Analysis, 31(3):337–351,

    work page 2023

  2. [2]

    Llms and cultural values: The impact of prompt language and explicit cultural framing.Computational Linguistics, pages 1–88,

    [Bult´e and Terryn, 2026] Bram Bult ´e and Ayla Rigouts Terryn. Llms and cultural values: The impact of prompt language and explicit cultural framing.Computational Linguistics, pages 1–88,

    work page 2026

  3. [3]

    Person- ality and politics: Values, traits, and political choice.Political psychology, 27(1):1–28,

    [Capraraet al., 2006 ] Gian Vittorio Caprara, Shalom Schwartz, Cristina Capanna, Michele Vecchione, and Claudio Barbaranelli. Person- ality and politics: Values, traits, and political choice.Political psychology, 27(1):1–28,

    work page 2006

  4. [4]

    Maxmin-rlhf: Alignment with di- verse human preferences

    [Chakrabortyet al., 2024 ] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. Maxmin-rlhf: Alignment with di- verse human preferences. InInternational Con- ference on Machine Learning, pages 6116–6135. PMLR,

    work page 2024

  5. [5]

    A test of the reproducibil- ity of the clustering of cultural variables.Cross- Cultural Research, 55(1):29–57,

    [Fog, 2021] Agner Fog. A test of the reproducibil- ity of the clustering of cultural variables.Cross- Cultural Research, 55(1):29–57,

    work page 2021

  6. [6]

    World values survey trend file (1981–

    [Haerpferet al., 2022 ] Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Juan Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Pu- ranen. World values survey trend file (1981–

    work page 2022

  7. [7]

    Modernization, cultural change, and democracy.The human development sequence,

    [Inglehart and Welzel, 2005] Ronald Inglehart and Christian Welzel. Modernization, cultural change, and democracy.The human development sequence,

    work page 2005

  8. [8]

    Changing mass priorities: The link between modernization and democracy.Per- spectives on politics, 8(2):551–567,

    [Inglehart and Welzel, 2010] Ronald Inglehart and Christian Welzel. Changing mass priorities: The link between modernization and democracy.Per- spectives on politics, 8(2):551–567,

    work page 2010

  9. [9]

    Personallm: Investigating the ability of large language models to express personality traits

    [Jianget al., 2024 ] Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Personallm: Investigating the ability of large language models to express personality traits. InFindings of the association for compu- tational linguistics: NAACL 2024, pages 3605– 3627,

    work page 2024

  10. [10]

    Cultural fidelity in large-language models: An evaluation of online language resources as a driver of model perfor- mance in value representation.arXiv preprint arXiv:2410.10489,

    [Kazemiet al., 2024 ] Sharif Kazemi, Gloria Ger- hardt, Jonty Katz, Caroline Ida Kuria, Estelle Pan, and Umang Prabhakar. Cultural fidelity in large-language models: An evaluation of online language resources as a driver of model perfor- mance in value representation.arXiv preprint arXiv:2410.10489,

    work page arXiv 2024

  11. [11]

    ArXiv preprint arXiv:2406.14805

    [Kharchenkoet al., 2024 ] Julia Kharchenko, Tanya Roosta, Aman Chadha, and Chirag Shah. How well do llms represent values across cul- tures? empirical analysis of llm responses based on hofstede cultural dimensions.arXiv preprint arXiv:2406.14805,

    work page arXiv 2024

  12. [12]

    Understanding the effects of rlhf on llm generalisation and diversity

    [Kirket al., 2023 ] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. InNeurIPS 2023 Workshop on Instruction Tuning and In- struction F ollowing,

    work page 2023

  13. [13]

    Simulating public opin- ion: Comparing distributional and individual- level predictions from llms and random forests

    [Miranda and Balbi, 2025] Fernando Miranda and Pedro Paulo Balbi. Simulating public opin- ion: Comparing distributional and individual- level predictions from llms and random forests. Entropy, 27(9):923,

    work page 2025

  14. [14]

    Representation bias in political sam- ple simulations with large language models

    [Qiet al., 2025 ] Weihong Qi, Hanjia Lyu, and Jiebo Luo. Representation bias in political sam- ple simulations with large language models. In Companion Proceedings of the ACM on Web Conference 2025, pages 1264–1267,

    work page 2025

  15. [15]

    Forecasting using relative entropy.Journal of Money, Credit and Banking, pages 383–401,

    [Robertsonet al., 2005 ] John C Robertson, Ellis W Tallman, and Charles H Whiteman. Forecasting using relative entropy.Journal of Money, Credit and Banking, pages 383–401,

    work page 2005

  16. [16]

    Unintended impacts of llm align- ment on global representation

    [Ryanet al., 2024 ] Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of llm align- ment on global representation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 16121–16140,

    work page 2024

  17. [17]

    Language model fine-tuning on scaled survey data for predicting distributions of pub- lic opinions

    [Suhet al., 2025 ] Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang. Language model fine-tuning on scaled survey data for predicting distributions of pub- lic opinions. InProceedings of the 63rd An- nual Meeting of the Association for Compu- tational Linguistics (V olume 1: Long Papers), pages 21147–21170,

    work page 2025

  18. [18]

    Random silicon sampling: Simulating human sub-population opinion using a large language model based on group-level demographic information

    [Sunet al., 2024 ] Seungjong Sun, Eungu Lee, Dongyan Nan, Xiangying Zhao, Wonbyung Lee, Bernard J Jansen, and Jang Hyun Kim. Ran- dom silicon sampling: Simulating human sub- population opinion using a large language model based on group-level demographic information. arXiv preprint arXiv:2402.18144,

    work page arXiv 2024

  19. [19]

    Cultural bias and cultural alignment of large language models

    [Taoet al., 2024 ] Yan Tao, Olga Viberg, Ryan S Baker, and Ren ´e F Kizilcec. Cultural bias and cultural alignment of large language models. PNAS nexus, 3(9):346,

    work page 2024

  20. [20]

    On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization

    [Xiaoet al., 2025 ] Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, and Weijie J Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. Journal of the American Statistical Association, 120(552):2154–2164,

    work page 2025

  21. [21]

    Should llms be weird? exploring weirdness and human rights in large language models

    [Zhouet al., 2025 ] Ke Zhou, Marios Constan- tinides, and Daniele Quercia. Should llms be weird? exploring weirdness and human rights in large language models. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Soci- ety, volume 8, pages 2808–2820,

    work page 2025

  22. [22]

    Theno-hedgeprompt asks the generator to produce descriptors that encourage the target response, but does not explicitly restrict broader associations

    designed to test whether descriptor performance depends only on capturing the target response, or also on limiting unintended inferences. Theno-hedgeprompt asks the generator to produce descriptors that encourage the target response, but does not explicitly restrict broader associations. We expect this variant to perform worse, because such descriptors ma...

    work page 2005

  23. [23]

    This demonstrates that the observed improvements are systematic rather than driven by a small number of favorable cases

    We find that the majority of points lie above the diagonal, indicating that persona prompting outperforms country roleplay across a wide range of questions. This demonstrates that the observed improvements are systematic rather than driven by a small number of favorable cases. Figure 7: Per-item Shapley values across countries. Negative values indicate th...

    work page 2025