Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Llama2 best mimics German public opinion among three LLMs, with a left-leaning bias

desk verdict A transparent German open-ended opinion-generation case study whose useful findings are undercut by an unvalidated classifier that likely does not transfer from long LLM text to short human survey text. read the letter →

arxiv 2412.13169 v2 pith:BLTDRQG5 submitted 2024-12-17 cs.CL

Bolei Ma , Berk Yoztyurk , Anna-Carolina Haensch , Xinpeng Wang , Markus Herklotz , Frauke Kreuter , Barbara Plank , Matthias Assenmacher This is my paper · ORCID
classification cs.CL
keywords algorithmicfidelitypublicopinionlargelanguagemodelspersonapromptingGermanLongitudinalElectionStudyopen-endedsurveyrepresentativenesspoliticalbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether large language models can reproduce the opinions of German subpopulations when asked an open-ended survey question about the most important problem facing Germany. Using the German Longitudinal Election Study (GLES) panel, the authors prompt Llama2, Gemma, and Mixtral with persona descriptions built from six demographic variables and compare the resulting answer distributions to real survey responses. The central claim is that Llama2 outperforms the other two models at matching the survey's category distribution, especially for groups with lower opinion diversity, and that the model fits left-leaning party supporters better than right-leaning AfD supporters. The paper also shows that adding demographic variables to the prompt, most strikingly party affiliation, improves representativeness. If true, this suggests that open-weight LLMs can serve as partial substitutes for human survey responses in some German contexts, while also exposing systematic political bias that would distort any such use.

What carries the argument

The central mechanism is persona-based prompting: each survey respondent's age, gender, leaning party, region, education degree, and vocational degree are inserted into a German-language prompt that asks the model to answer, in one concise sentence, what the most important problem facing Germany is in a given month and year. The evaluation pipeline then maps both human and LLM free-text answers onto a coarse coding scheme of 16 political categories using a fine-tuned German BERT classifier trained on 1,500 manually annotated LLM outputs (weighted F1 0.93). Representativeness is measured as the Jensen-Shannon distance between the predicted category distribution and the survey distribution, complemented by entropy, conditional entropy, information gain, and Cramér's V to assess within-group diversity and correlation patterns. The key object is the category distribution itself: the authors treat the LLM's output distribution as a sample from a synthetic population and compare it directly to the survey's observed distribution.

What would settle it

Take a random sample of several hundred human GLES survey responses and code them both with the fine-tuned BERT classifier and with human annotators using the same 16-class scheme; if the classifier's F1 against human coding on this human text is well below the 0.93 reported on LLM text, then every JS distance and model ranking in this paper is potentially an artifact of classifier bias. A second check would be to re-run the full pipeline with a different classifier (for example, a larger multilingual model) and see whether Llama2 still beats Mixtral and still favors left-leaning parties.

Watch

Extended reading notes

Core claim

The paper establishes that algorithmic fidelity, defined as the ability of an LLM to replicate the socio-cultural context and nuanced opinions of human subpopulations, varies sharply across models, subpopulations, and prompt contents. In a pre-COVID wave, Llama2 achieves a Jensen-Shannon distance of 0.28 to the survey distribution, compared to 0.29 for Mixtral and 0.62 for Gemma, with Gemma producing 42 percent COVID-related answers despite being prompted with a pre-pandemic date. Across ten waves, Llama2's representativeness declines as survey answer diversity increases, and the model consistently models supporters of the Greens and the Left better than supporters of the AfD. Ablation experiments show that including all six demographic variables reduces JS distance by 0.15 compared to a no-demographics baseline, and that removing party affiliation from the prompt hurts performance the most. The authors interpret these findings as evidence that LLMs carry a left-leaning, Western-oriented bias and that prompt composition materially changes the opinion distributions they generate.

Load-bearing premise

The entire evaluation depends on the fine-tuned BERT classifier, trained only on 1,500 LLM outputs, producing category labels for LLM-generated text that are directly comparable to the human-coded GLES survey labels, even though the two text genres differ enormously in style and length.

Editorial extensions

If this is right

  • If Llama2's fidelity holds across settings, open-weight LLMs could generate provisional estimates of German public opinion for time points between survey waves, provided the prompt includes enough demographic detail.
  • Adding party affiliation to a persona prompt substantially improves representativeness, so future survey-simulation work should treat party as a first-class conditioning variable rather than an optional demographic.
  • The finding that representativeness falls as answer diversity rises implies that LLM-generated opinion samples will be least trustworthy exactly when the public is most divided or most heterogeneous in its priorities.
  • The consistent over- and under-representation of specific policy categories (for example, under-representing Migration and Integration, over-representing Values and Political Culture) provides a concrete checklist for calibrating LLM-based survey simulations before deployment.
  • A left-leaning bias in open-ended German opinion generation means any downstream use in political forecasting or polling must either reweight outputs by party or explicitly model the party-conditioned distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The validation gap in the classifier is likely the biggest hidden risk: since the BERT classifier was trained and tested only on LLM outputs, its labels for human survey text are unverified, and a systematic mislabeling of human responses could alter every reported JS distance and ranking.
  • A natural extension would be to run the same persona-prompting pipeline in another language or on a different open-ended survey question to see whether the left-leaning bias and the party-affiliation effect are universal or specific to German political discourse.
  • The observed negative correlation between survey entropy and representativeness suggests a testable hypothesis about LLM opinion generation: models may be drawn toward the mode of a distribution and under-represent minority positions, which would make the entropy-representativeness tradeoff a general property rather than a quirk of this dataset.
  • One could design a direct falsification experiment for the classifier assumption by hiring human annotators to code a fresh sample of both human and LLM responses from multiple models and waves, then recomputing all JS distances with the human-coded labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents a case study of algorithmic fidelity of three open-weight LLMs (Llama2, Gemma, Mixtral) in generating synthetic German public opinions, using the GLES open-ended 'most important problem' question. Personas are built from six demographic variables, and the LLM outputs are classified into 16 coarse topic categories with a fine-tuned BERT classifier trained on 1,500 manually annotated LLM outputs. The authors report that Llama2 yields lower JS distance to the survey distribution than the other models (0.287 vs 0.295 for Mixtral), that representativeness varies across waves and across subpopulations (with better alignment for Greens/Left than AfD), and that including more demographic variables, especially party affiliation, improves representativeness.

Significance. If the methodological concerns were addressed, the paper would be a valuable contribution to the emerging literature on LLM-based 'silicon samples', particularly for non-English and open-ended settings. The use of a longitudinal German survey with real open-ended text, the three-model comparison, the ablation design, and the public code are strengths. The finding of a left-party bias consistent with prior work (Santurkar et al., 2023; von der Heyde et al., 2025) is a useful replication in a new context. However, the load-bearing evaluation pipeline—a BERT classifier trained and validated only on LLM outputs—needs validation on human survey text before the quantitative claims can be taken at face value.

major comments (5)
  1. [§3.3 and Tables 1, 12] The BERT classifier (§3.3) is trained and evaluated exclusively on LLM-generated text; it is never applied to or validated against human-coded GLES survey responses. Given the large stylistic gap (Table 1: average word count 25.65 for Llama2 vs 2.29 for survey) and the fixed introductory phrases used by all models (Appendix E), the classifier may rely on surface cues rather than topic semantics. Because the survey labels come from GESIS coding and the LLM labels from this classifier, differential measurement error could contaminate every JS distance (Tables 1, 2, 14; Figures 4–7) and all model rankings. I recommend manually coding a random sample of survey open-ends and of LLM outputs per model/wave and reporting the classifier's accuracy on both, including per-party and per-wave breakdowns.
  2. [§4.1, Table 1] The model-ranking claim ('Llama performs better than other LLMs') rests on a single wave (wave 12) and a Llama2-vs-Mixtral JS-distance gap of 0.287 vs 0.295 (Δ=0.008). No confidence intervals, bootstrap resampling, or significance tests are reported for these distances, and the gap is an order of magnitude smaller than the classifier's 7% error rate. The claim is therefore not quantitatively supported; please provide bootstrapped CIs (e.g., over personas or repeated generations) and a paired test, and temper the conclusion if the intervals overlap.
  3. [§4.2, Table 2] The Pearson correlation r = -0.35 between survey entropy and JS distance is interpreted as 'the model's representativeness of the population decreases as the diversity in answers increases.' Because a negative r associates higher entropy (more diversity) with lower JS distance (better representativeness), the stated direction is the opposite of what the coefficient implies. With N=10 waves, this correlation is also not statistically significant. The abstract's claim 'particularly when there is lower opinion diversity within those groups' should be reconciled with this result, or the wording should be corrected (e.g., to 'increases' if that is the actual direction).
  4. [§3.3 and §3.4] The paper does not specify how the multilabel classifier outputs are converted into the probability distributions used for JS distance and entropy. Since LLM outputs have up to 1.33 labels per sample (vs 1.03 for survey, Table 1), treating each label as an independent observation versus aggregating per sample will produce different distributions and different JS distances. Please state the exact aggregation rule and, ideally, report a sensitivity analysis using only the first or primary label.
  5. [§4.1 and §G, Table 12] The label-level deviations (e.g., 'Values, Political Culture, and Social Criticism' overrepresented with mean APE=207, 'Migration and Integration' underrepresented) are consistent with the classifier being tuned to LLM-specific phrasings rather than to the survey's coding conventions. Since the classifier was trained on the same three models it is then used to evaluate, this creates a risk of style exploitation rather than true semantic classification. Validation on human-coded survey text (as in comment 1) is necessary to rule out this alternative explanation for the party-bias results.
minor comments (6)
  1. [Abstract] The phrase 'matches the least with the right-party AfD' is ungrammatical; suggest 'matches least for the right-wing AfD.'
  2. [Figure 1] The y-axis label 'Frequency' should specify whether these are absolute or relative frequencies; the text refers to distributions.
  3. [§4.3] The sentence 'excluding only the party variable leads to a smaller improvement in JS distance' is unclear; Figure 6 appears to show that removing party increases JS distance relative to other removals, which the text should state directly.
  4. [Appendix C] The annotation procedure reports that two annotators labeled the 1,500 samples but does not report inter-annotator agreement; please add Cohen's kappa for a random subset.
  5. [Table 14] In the first row, the subpopulation column reads 'Population Population'; the duplicate should be removed.
  6. [§4.1 and §4.2] The paper does not report the number of survey respondents per wave or the number of LLM generations per persona; these counts are needed to assess the stability of the JS distance estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation compares LLM-generated category distributions to external human-coded GLES survey data, and no result is defined by the classifier's training labels.

full rationale

The paper derives its main findings (model ranking, party alignment, ablation effects) from JS distances, entropies, information gain, and Cramér's V computed between LLM output category distributions and the human-coded GLES survey distribution. The survey labels are external (GESIS 2023; GESIS 2024) and are not produced by the LLMs or by the authors' classifier. The BERT classifier in §3.3 is trained on 1,500 manually annotated LLM outputs and used only to code LLM responses; it is not fitted to the survey labels against which those responses are compared, so no comparison is forced by construction. Its validation solely on held-out LLM outputs (F1=0.93) and the large style gap between LLM and human text (Table 1: 25.65 vs 2.29 average words) raise a legitimate measurement-validity concern, but that is a correctness risk, not a circular reduction. Self-citations (e.g., Ma et al. 2024; Wang et al. 2024a,b; von der Heyde et al. 2025, which shares an author) are used as methodological precedent or as corroboration for the left-leaning-bias pattern; the bias result is independently recomputed from the paper's own experiments, so those citations are not load-bearing. No equation equates a reported result to the fitted classifier's training labels, and no parameter is fitted to the target survey distribution. Therefore no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The analysis relies on standard statistical definitions (JS divergence, entropy, mutual information, Cramér's V) which are not listed as axioms. The key domain assumptions are the representativeness of the GLES data, the validity of the classifier as a measurement bridge, and the interpretation of persona prompts as demographic conditioning. There are no free parameters in the evaluation itself; the only fitted model (BERT classifier) is a measurement tool, not a source of the central results.

assumptions (3)
  • domain assumption The GLES survey data and GESIS coding scheme accurately capture the distribution of German public opinion on the most important problem.
    The survey is used as ground truth for the comparison; if the survey or coding is biased, the fidelity measurements inherit that bias.
  • domain assumption The fine-tuned BERT classifier produces labels for LLM outputs that are equivalent to human-coded labels from the GESIS scheme.
    Introduced in §3.3; the classifier is trained only on LLM outputs, and its generalization to the survey's human text is assumed but not validated.
  • domain assumption Persona prompts with the six selected demographic variables condition the LLM on the intended subpopulation in a way that is comparable to how real respondents' demographic characteristics influence their survey answers.
    This is the basis of the subpopulation fidelity analysis; if the LLM ignores or over-relies on certain variables, the measured JS distances reflect prompt effects rather than demographic effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study." pith.science (2026). https://pith.science/paper/BLTDRQG5

@misc{pith2026241213169,
  author       = {Pith},
  title        = {Pith review of: Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLTDRQG5}},
  note         = {Machine review of arXiv:2412.13169}
}
read the original abstract

In recent research, large language models (LLMs) have been increasingly used to investigate public opinions. This study investigates the algorithmic fidelity of LLMs, i.e., the ability to replicate the socio-cultural context and nuanced opinions of human participants. Using open-ended survey data from the German Longitudinal Election Studies (GLES), we prompt different LLMs to generate synthetic public opinions reflective of German subpopulations by incorporating demographic features into the persona prompts. Our results show that Llama performs better than other LLMs at representing subpopulations, particularly when there is lower opinion diversity within those groups. Our findings further reveal that the LLM performs better for supporters of left-leaning parties like The Greens and The Left compared to other parties, and matches the least with the right-party AfD. Additionally, the inclusion or exclusion of specific variables in the prompts can significantly impact the models' predictions. These findings underscore the importance of aligning LLMs to more effectively model diverse public opinions while minimizing political biases and enhancing robustness in representativeness.

Figures

Figures reproduced from arXiv: 2412.13169 by the authors.

Figure 1
Figure 1. The distribution of the top 5 answer categories [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Translated prompt in English. The original [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Information Gain for leaning party variable (Xi). Left: population entropy (H(Y )), right: subpopula￾tion entropy (H(Y | Xi)). A large gap between left and right (H(Y ) − H(Y | Xi)) means big information gain when focusing on the samples of the subpopulation group, indicating responses with this group are less diverse. three LLMs have been prompted to generate syn￾thetic answers. The generated answers are classi￾fie… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: JS Distance of six subpopulation groups in Experiment 2. An in-depth presentation of the JS Distance for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 7
Figure 7. Figure 7: Information Gain for leaning party variable, comparing survey entropy to Llama2-all (with all variables), 1VAR-party (with only party variable), and Llama2-base (with no variables). Left: population entropy (H(Y )), right: subpopulation entropy (H(Y | Xi)). one party v…
Figure 8
Figure 8. Figure 8: JS distances of answers for the last five waves [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Original prompt template in German. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 12
Figure 12. Figure 12: Label distributions of three experimented LLMs [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: The mean and ±1 standard deviation of JS Distances for social groups in Experiment 2. 12 12 12 12 12 12 13 13 13 13 13 13 14 14 14 14 14 14 15 15 15 15 15 15 16 16 16 16 16 16 17 17 17 17 17 17 18 18 18 18 18 18 19 19 19 19 19 19 20 20 20 20 20 20 21 21 21 21 21 21 12…
Figure 14
Figure 14. Figure 14: Cramér’s Values for pairwise patterns between the six prompting variables and text answers from the [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Survey’s Subpopulation Entropy and the JS Distance in Experiment 2 (mean values for waves 12-21). [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?

    cs.CL 2025-01 reject novelty 4.0 of 10

    A persona-based LLM survey generator given MiD 2017 response statistics reproduces those statistics almost exactly, but the evaluation is circular and the main claim is unsupported.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    In addition, no interpretation should benecessary to assign an answer to a category

    When coding the answers,no contextual knowledge should be used if possible. In addition, no interpretation should benecessary to assign an answer to a category

    work page

  2. [2]

    IEEE Access, 11:47304–47320

    Deep Generative Models for Synthetic Data: A Survey. IEEE Access, 11:47304–47320. GESIS. 2023. Gles panel 2016-2021, wellen 1-

    work page 2023

  3. [3]

    WEIRD bias

    Table 7 shows label-level percentage errors. We compared the best-performing models in the 1-var-* and w/o.-* experiments. In 7 out of 14 labels, using all variables produced the lowest per- centage errors. Both JS distances and percentage errors decreased with the inclusion of more vari- ables, reinforcing the representational accuracy of the Llama2 mode...

    work page 2023

  4. [4]

    In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY , USA

    Evaluating large language models in gener- ating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY , USA. Association for Computing Machinery. Thomas M. Holbrook. 2022. An Introduction to Polit- ical and Social Data Analysis Using R. Thomas M. Holbrook. Tobias ...

    arXiv 2023

  5. [5]

    European and international politics ( < 5%) - Germany has played a key role in European Union Table 15: Checking the factual knowledge of Llama2 model without giving any survey contexts 25

    work page

  6. [6]

    Using LLMs to Model the Beliefs and Preferences of Targeted Populations

    Using llms to model the beliefs and preferences of targeted populations. Preprint, arXiv:2403.20252. Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Ben- jamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein

    work page Pith review arXiv

  7. [7]

    Preprint, arXiv:2411.10109

    Generative agent simulations of 1,000 people. Preprint, arXiv:2411.10109. Philip Resnik, Bolei Ma, Alexander Hoyle, Pranav Goel, Rupak Sarkar, Maeve Gearing, Anna-Carolina Haen- sch, and Frauke Kreuter. 2024. Topic-oriented proto- col for content analysis of text – a preliminary study. Unpublished manuscript. Paul Röttger, Valentin Hofmann, Valentina Pyat...

    arXiv 2024

  8. [8]

    Elements in Psychology and Culture

    Psychology’s WEIRD Problems. Elements in Psychology and Culture. Cambridge University Press. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. Bangzhao Shu, Lechen Zhang,...

    work page 2023

Show all 17 references
  1. [11]

    introductory phrases

    If several topics are mentioned (although the question explicitly asks for one topic), several codes are assigned in the order in which they are mentioned. A maximum of 3 codes (in order of mention) are assigned.3. if general topics are supported with additional examples (enum...

    2020

  2. [13]

    Climate change and environmental protection (25-30%) - Concerns about climate change and the environment have been growing in Germany, with increased activism and calls for stronger government action to reduce carbon emissions and protect natural resources

  3. [14]

    Economy and job security (20-25%) - The German economy has been facing challenges, including a slowdown in growth and job insecurity, particularly in the automotive and manufacturing sectors

  4. [15]

    Immigration and integration (15-15%) - Germany has been a destination for many refugees and migrants, and the integration of these individuals into society has been a persistent challenge, with debates around issues such as employment, education, and social cohesion

  5. [16]

    Healthcare and social welfare (5-10%) - Germany’s healthcare system has been under strain, with debates over funding and access to certain medical services, particularly for low-income families

  6. [21]

    ZA6838 Datenfile Version 6.0.0, https://doi.org/10.4232/1.14114

    GESIS, Köln. ZA6838 Datenfile Version 6.0.0, https://doi.org/10.4232/1.14114. GESIS. 2024. Gles codierung des wichtigsten politischen problems (2018-2022). GESIS, Köln. ZA7957 Datenfile Version 2.0.0, https://doi.org/10.4232/1.14241. Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari

    2024 doi

  7. [2017]

    my answer is C

    Designing valid and reliable vignette experi- ments for survey research: A case study on the fair gender income gap. Journal of Methods and Mea- surement in the Social Sciences, 7:52–94. Seungjong Sun, Eungu Lee, Dongyan Nan, Xiangy- ing Zhao, Wonbyung Lee, Bernard J. Jansen, ...

    2024 arXiv

  8. [2023]

    Political Analysis, 31(3):337–351

    Out of one, many: Using language mod- els to simulate human samples. Political Analysis, 31(3):337–351. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augen- stein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First W...

    2023 arXiv

  9. [2024]

    Research & Politics, 11(1):20531680241231468

    Do ais know what the most important issue is? using language models to code open-text so- cial survey responses at scale. Research & Politics, 11(1):20531680241231468. Keiichi Namikoshi, Alex Filipowicz, David A. Shamma, Rumen Iliev, Candice L. Hogan, and Nikos Arechiga

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.