{"id":"e2d78802-45ff-4e2b-b501-722a50f291e1","arxiv_id":"2412.13169","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using GLES survey data, the paper shows Llama2 produces synthetic German opinions closer to real responses than Gemma or Mixtral, with a consistent left-party bias.","lead":"This study tests whether LLMs can mimic the opinions of German survey respondents when given demographic profiles as prompts, focusing on an open-ended question about Germany's most important problem. It finds that Llama2 matches real survey answers better than Gemma or Mixtral, but all models favor left-leaning parties and struggle with right-wing AfD voters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated BERT classifier—trained only on LLM text, never on human survey text—threatens every JS-distance comparison, and the Llama2-vs-Mixtral gap (0.287 vs 0.295) is smaller than plausible classifier bias.","rationale":"The reader's weakest-assumption identification is exactly the most load-bearing concern. The entire evaluation pipeline—model ranking, party bias, ablation effects—depends on label distributions produced by a classifier that has only ever seen LLM-generated text. The paper's own qualitative analysis (Appendix E) documents massive stylistic gaps between LLM and human responses, yet the classifier's validity across that gap is never tested. The F1=0.93 on held-out LLM outputs is real evidence of internal reliability, but it does not establish that the classifier's labeling of LLM text is commensurate with human coding of survey text; any systematic label bias would propagate directly into JS distances and Cramér's V, which are the basis for every conclusion. The near-tie between Llama2 and Mixtral (0.287 vs 0.295) makes the headline model ranking especially sensitive to even small classifier artifacts. The party-bias finding could also be an artifact if persona-conditioned style shifts (e.g., AfD-prompted outputs) trigger classifier confusions. The proposed test—applying the same classifier to human-coded survey text and to out-of-wave LLM outputs, plus bootstrap intervals for the key gap—would settle whether the concern lands. Because the reader already marked the verdict CONDITIONAL and this concern reinforces that conditionality rather than overturning it, the verdict remains UNCHANGED. The paper deserves credit for releasing code, using a transparent coding scheme, and clearly stating its limitations; these do not, however, remove the need for validation of the central measurement instrument.","tokens_in":27346,"tokens_out":6229,"duration_ms":60069,"concrete_test":"Randomly sample N=500 human-coded GLES open-ended responses (from waves 12–21) and N=500 held-out LLM outputs (from each model, and from at least one wave not used for classifier training, e.g., wave 21). Have the same human annotators apply the GLES coarse coding scheme to both, then run the published BERT classifier on both sets. If the classifier's macro-F1 on human texts is substantially below its 0.93 on LLM texts (e.g., <0.8), or if per-class error rates differ significantly between Llama2 and Mixtral or between party personas, then the reported JS distances conflate model performance with classifier measurement error. Also report 95% bootstrap intervals for the Llama2-vs-Mixtral JS gap; if the interval includes zero, the 'Llama performs better' claim fails regardless of classifier quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—Llama2 better represents German subpopulations, with left-party alignment exceeding AfD—is computed exclusively from a BERT classifier (§3.3) fine-tuned on 1,500 LLM outputs (500 per model) and evaluated only on held-out LLM outputs (F1=0.93). The classifier is never validated on human-coded GLES texts, nor on LLM outputs from different waves or prompts. This matters because the comparison objects differ radically in style: survey answers average 2.29 words versus 25.65 for Llama2 (Table 1), and Appendix E reports that Llama2 uses a fixed introductory phrase in 96% of outputs. A classifier trained on such stylized text can exploit surface cues rather than topic semantics; if its error pattern varies by model, party persona, or wave, then every JS distance (Tables 1, 2, 14; Figs. 4–7; Cramér's V in Table 3) is contaminated. The model-ranking result is particularly fragile: Llama2's JS distance to survey is 0.287 vs Mixtral's 0.295, a 0.008 gap that is an order of magnitude smaller than the classifier's 7% error rate and its likely distribution shift across waves. The party-bias result could also be an artifact if AfD-persona outputs use distinctive phrasing that the classifier routes to survey-rare categories. The ablation finding that 'party' is the most influential variable is similarly downstream of classifier behavior on prompt-conditioned styles. The paper is transparent about the classifier setup, but the equivalence assumption between LLM-text classification and human coding of survey text is load-bearing and unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a case study of algorithmic fidelity of three open-weight LLMs (Llama2, Gemma, Mixtral) in generating synthetic German public opinions, using the GLES open-ended 'most important problem' question. Personas are built from six demographic variables, and the LLM outputs are classified into 16 coarse topic categories with a fine-tuned BERT classifier trained on 1,500 manually annotated LLM outputs. The authors report that Llama2 yields lower JS distance to the survey distribution than the other models (0.287 vs 0.295 for Mixtral), that representativeness varies across waves and across subpopulations (with better alignment for Greens/Left than AfD), and that including more demographic variables, especially party affiliation, improves representativeness.","tokens_in":27629,"tokens_out":8037,"duration_ms":70458,"significance":"If the methodological concerns were addressed, the paper would be a valuable contribution to the emerging literature on LLM-based 'silicon samples', particularly for non-English and open-ended settings. The use of a longitudinal German survey with real open-ended text, the three-model comparison, the ablation design, and the public code are strengths. The finding of a left-party bias consistent with prior work (Santurkar et al., 2023; von der Heyde et al., 2025) is a useful replication in a new context. However, the load-bearing evaluation pipeline—a BERT classifier trained and validated only on LLM outputs—needs validation on human survey text before the quantitative claims can be taken at face value.","major_comments":[{"comment":"The BERT classifier (§3.3) is trained and evaluated exclusively on LLM-generated text; it is never applied to or validated against human-coded GLES survey responses. Given the large stylistic gap (Table 1: average word count 25.65 for Llama2 vs 2.29 for survey) and the fixed introductory phrases used by all models (Appendix E), the classifier may rely on surface cues rather than topic semantics. Because the survey labels come from GESIS coding and the LLM labels from this classifier, differential measurement error could contaminate every JS distance (Tables 1, 2, 14; Figures 4–7) and all model rankings. I recommend manually coding a random sample of survey open-ends and of LLM outputs per model/wave and reporting the classifier's accuracy on both, including per-party and per-wave breakdowns.","section":"§3.3 and Tables 1, 12"},{"comment":"The model-ranking claim ('Llama performs better than other LLMs') rests on a single wave (wave 12) and a Llama2-vs-Mixtral JS-distance gap of 0.287 vs 0.295 (Δ=0.008). No confidence intervals, bootstrap resampling, or significance tests are reported for these distances, and the gap is an order of magnitude smaller than the classifier's 7% error rate. The claim is therefore not quantitatively supported; please provide bootstrapped CIs (e.g., over personas or repeated generations) and a paired test, and temper the conclusion if the intervals overlap.","section":"§4.1, Table 1"},{"comment":"The Pearson correlation r = -0.35 between survey entropy and JS distance is interpreted as 'the model's representativeness of the population decreases as the diversity in answers increases.' Because a negative r associates higher entropy (more diversity) with lower JS distance (better representativeness), the stated direction is the opposite of what the coefficient implies. With N=10 waves, this correlation is also not statistically significant. The abstract's claim 'particularly when there is lower opinion diversity within those groups' should be reconciled with this result, or the wording should be corrected (e.g., to 'increases' if that is the actual direction).","section":"§4.2, Table 2"},{"comment":"The paper does not specify how the multilabel classifier outputs are converted into the probability distributions used for JS distance and entropy. Since LLM outputs have up to 1.33 labels per sample (vs 1.03 for survey, Table 1), treating each label as an independent observation versus aggregating per sample will produce different distributions and different JS distances. Please state the exact aggregation rule and, ideally, report a sensitivity analysis using only the first or primary label.","section":"§3.3 and §3.4"},{"comment":"The label-level deviations (e.g., 'Values, Political Culture, and Social Criticism' overrepresented with mean APE=207, 'Migration and Integration' underrepresented) are consistent with the classifier being tuned to LLM-specific phrasings rather than to the survey's coding conventions. Since the classifier was trained on the same three models it is then used to evaluate, this creates a risk of style exploitation rather than true semantic classification. Validation on human-coded survey text (as in comment 1) is necessary to rule out this alternative explanation for the party-bias results.","section":"§4.1 and §G, Table 12"}],"minor_comments":[{"comment":"The phrase 'matches the least with the right-party AfD' is ungrammatical; suggest 'matches least for the right-wing AfD.'","section":"Abstract"},{"comment":"The y-axis label 'Frequency' should specify whether these are absolute or relative frequencies; the text refers to distributions.","section":"Figure 1"},{"comment":"The sentence 'excluding only the party variable leads to a smaller improvement in JS distance' is unclear; Figure 6 appears to show that removing party increases JS distance relative to other removals, which the text should state directly.","section":"§4.3"},{"comment":"The annotation procedure reports that two annotators labeled the 1,500 samples but does not report inter-annotator agreement; please add Cohen's kappa for a random subset.","section":"Appendix C"},{"comment":"In the first row, the subpopulation column reads 'Population Population'; the duplicate should be removed.","section":"Table 14"},{"comment":"The paper does not report the number of survey respondents per wave or the number of LLM generations per persona; these counts are needed to assess the stability of the JS distance estimates.","section":"§4.1 and §4.2"}],"recommendation":"major_revision","confidential_remarks":"The central methodological issue—the classifier's lack of validation on human text—is the main reason for major revision. The paper is otherwise transparent and the code availability is a plus. The journal may also want to consider whether the single-wave model comparison and the lack of significance testing meet the journal's empirical standards, but these are addressable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a transparent case study worth reading, but the central comparison rests on a classifier that was never shown to work on the survey text it is supposed to predict. The paper does something new: it takes the algorithmic fidelity / persona prompting framework into German open-ended 'most important problem' survey responses using three open-weight models, and it includes a careful ablation of which demographic variables move representativeness. The code and appendices are thorough, and the authors are honest about several limitations, including annotation cost and style differences.\n\nThe main soft spot is the BERT classifier. It is fine-tuned on 1,500 LLM outputs (500 per model) and only evaluated on held-out LLM outputs. The survey answers average 2.29 words, LLM answers 25.65, and the paper itself notes Llama2 starts 96% of its outputs with the same phrase. A classifier trained on such stylized text is likely exploiting surface cues, and there is zero evidence it transfers to the short human responses. If the error pattern varies by model or party persona, every JS distance is contaminated. The Llama2 vs Mixtral gap (0.287 vs 0.295) is far smaller than the plausible classifier bias, so the headline model ranking is fragile. Also, the model comparison is single-wave for two of the three models, and there are no confidence intervals or significance tests for the JS distances or the entropy correlation.\n\nThat said, I would not call this invalid. The paper reports label distributions and APE tables, and the qualitative examples are plausible. The left-party bias finding matches prior work (von der Heyde et al.), so it is not an outlandish claim. The ablation results are less exposed to the classifier issue because they compare Llama2 outputs to each other, though even there the classifier could interact with prompt-conditioned style.\n\nWho is it for: people working on LLM survey simulation or synthetic populations. It is a useful benchmark, not a breakthrough. It should get a serious referee, but I would ask for classifier validation on human survey text, bootstrap confidence intervals, and ideally multi-wave runs for all models. Without that, the specific claim that Llama2 beats Mixtral should not be cited as solid.","headline":"A transparent German open-ended opinion-generation case study whose useful findings are undercut by an unvalidated classifier that likely does not transfer from long LLM text to short human survey text.","tokens_in":28224,"tokens_out":2273,"would_cite":true,"duration_ms":22774,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Llama2 best mimics German public opinion among three LLMs, with a left-leaning bias","keywords":["algorithmic fidelity","public opinion","large language models","persona prompting","German Longitudinal Election Study","open-ended survey","representativeness","political bias"],"falsifier":"Take a random sample of several hundred human GLES survey responses and code them both with the fine-tuned BERT classifier and with human annotators using the same 16-class scheme; if the classifier's F1 against human coding on this human text is well below the 0.93 reported on LLM text, then every JS distance and model ranking in this paper is potentially an artifact of classifier bias. A second check would be to re-run the full pipeline with a different classifier (for example, a larger multilingual model) and see whether Llama2 still beats Mixtral and still favors left-leaning parties.","tokens_in":27137,"feed_emoji":"🗳️","tokens_out":2710,"duration_ms":27149,"temperature":0.7,"pith_summary":"This paper tests whether large language models can reproduce the opinions of German subpopulations when asked an open-ended survey question about the most important problem facing Germany. Using the German Longitudinal Election Study (GLES) panel, the authors prompt Llama2, Gemma, and Mixtral with persona descriptions built from six demographic variables and compare the resulting answer distributions to real survey responses. The central claim is that Llama2 outperforms the other two models at matching the survey's category distribution, especially for groups with lower opinion diversity, and that the model fits left-leaning party supporters better than right-leaning AfD supporters. The paper also shows that adding demographic variables to the prompt, most strikingly party affiliation, improves representativeness. If true, this suggests that open-weight LLMs can serve as partial substitutes for human survey responses in some German contexts, while also exposing systematic political bias that would distort any such use.","feed_headline":"Llama2 best mimics German public opinion among three LLMs","feed_subtitle":"Open-ended persona prompts align best with left-leaning parties and improve when party affiliation is included.","key_machinery":"The central mechanism is persona-based prompting: each survey respondent's age, gender, leaning party, region, education degree, and vocational degree are inserted into a German-language prompt that asks the model to answer, in one concise sentence, what the most important problem facing Germany is in a given month and year. The evaluation pipeline then maps both human and LLM free-text answers onto a coarse coding scheme of 16 political categories using a fine-tuned German BERT classifier trained on 1,500 manually annotated LLM outputs (weighted F1 0.93). Representativeness is measured as the Jensen-Shannon distance between the predicted category distribution and the survey distribution, complemented by entropy, conditional entropy, information gain, and Cramér's V to assess within-group diversity and correlation patterns. The key object is the category distribution itself: the authors treat the LLM's output distribution as a sample from a synthetic population and compare it directly to the survey's observed distribution.","core_discovery":"The paper establishes that algorithmic fidelity, defined as the ability of an LLM to replicate the socio-cultural context and nuanced opinions of human subpopulations, varies sharply across models, subpopulations, and prompt contents. In a pre-COVID wave, Llama2 achieves a Jensen-Shannon distance of 0.28 to the survey distribution, compared to 0.29 for Mixtral and 0.62 for Gemma, with Gemma producing 42 percent COVID-related answers despite being prompted with a pre-pandemic date. Across ten waves, Llama2's representativeness declines as survey answer diversity increases, and the model consistently models supporters of the Greens and the Left better than supporters of the AfD. Ablation experiments show that including all six demographic variables reduces JS distance by 0.15 compared to a no-demographics baseline, and that removing party affiliation from the prompt hurts performance the most. The authors interpret these findings as evidence that LLMs carry a left-leaning, Western-oriented bias and that prompt composition materially changes the opinion distributions they generate.","pith_inferences":["The validation gap in the classifier is likely the biggest hidden risk: since the BERT classifier was trained and tested only on LLM outputs, its labels for human survey text are unverified, and a systematic mislabeling of human responses could alter every reported JS distance and ranking. ","A natural extension would be to run the same persona-prompting pipeline in another language or on a different open-ended survey question to see whether the left-leaning bias and the party-affiliation effect are universal or specific to German political discourse. ","The observed negative correlation between survey entropy and representativeness suggests a testable hypothesis about LLM opinion generation: models may be drawn toward the mode of a distribution and under-represent minority positions, which would make the entropy-representativeness tradeoff a general property rather than a quirk of this dataset. ","One could design a direct falsification experiment for the classifier assumption by hiring human annotators to code a fresh sample of both human and LLM responses from multiple models and waves, then recomputing all JS distances with the human-coded labels. "],"forward_implications":["If Llama2's fidelity holds across settings, open-weight LLMs could generate provisional estimates of German public opinion for time points between survey waves, provided the prompt includes enough demographic detail. ","Adding party affiliation to a persona prompt substantially improves representativeness, so future survey-simulation work should treat party as a first-class conditioning variable rather than an optional demographic. ","The finding that representativeness falls as answer diversity rises implies that LLM-generated opinion samples will be least trustworthy exactly when the public is most divided or most heterogeneous in its priorities. ","The consistent over- and under-representation of specific policy categories (for example, under-representing Migration and Integration, over-representing Values and Political Culture) provides a concrete checklist for calibrating LLM-based survey simulations before deployment. ","A left-leaning bias in open-ended German opinion generation means any downstream use in political forecasting or polling must either reweight outputs by party or explicitly model the party-conditioned distributions. "],"supporting_citations":[{"why":"Introduces the concept of algorithmic fidelity and the silicon-sampling approach of conditioning LLMs on demographic backstories, which this paper directly adopts and extends to open-ended responses.","marker":"Argyle et al., 2023"},{"why":"Provides the prior German-context result that GPT-3.5 predicts voting behavior with a bias toward the Greens and the Left, the baseline this paper compares against for its party-bias finding in open-ended text.","marker":"von der Heyde et al., 2025"},{"why":"Documents that language models reflect the opinions of the WEIRD (Western, Educated, Industrialized, Rich, Democratic) population rather than the general population, which frames the left-leaning bias interpretation.","marker":"Santurkar et al., 2023"},{"why":"Supplies the GLES panel dataset with its waves, demographic variables, and open-ended survey answers that form the empirical ground truth for all comparisons.","marker":"GESIS, 2023"},{"why":"Provides the original fine-grained coding scheme for the most-important-problem question, which the authors merge into their 16 coarse classes.","marker":"GESIS, 2024"},{"why":"Justifies the practice of collapsing survey answer classes into a simpler coarse classification, directly motivating the authors' 16-class scheme.","marker":"Mellon et al., 2024"},{"why":"Establishes the method of training a classifier on human-labeled LLM outputs rather than relying on first-token probabilities, which the authors use to label their generated text.","marker":"Wang et al., 2024a,b"},{"why":"Provides the BERT architecture behind the fine-tuned German classifier used to code all LLM responses.","marker":"Devlin et al., 2019"},{"why":"Introduces the use of Jensen-Shannon distance for comparing LLM opinion distributions to survey data, which is the paper's primary representativeness metric.","marker":"Durmus et al., 2024"},{"why":"Documents the Llama-2-13b-chat model that the authors find to be the best-performing and study in depth across ten waves.","marker":"Touvron et al., 2023"}],"fun_headline_variants":["Llama2 outshines rivals in German opinion synthesis","Study: LLMs replicate German views, but with left-leaning bias","Llama2 best mimics German public, AfD least represented","AI public opinion: Llama2 leads, right-wing views poorly modeled","German opinion AI study reveals left bias, Llama2 most accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation depends on the fine-tuned BERT classifier, trained only on 1,500 LLM outputs, producing category labels for LLM-generated text that are directly comparable to the human-coded GLES survey labels, even though the two text genres differ enormously in style and length.","fun_headline_variants_meta":{"raw":{"variants":["Llama2 outshines rivals in German opinion synthesis","Study: LLMs replicate German views, but with left-leaning bias","Llama2 best mimics German public, AfD least represented","AI public opinion: Llama2 leads, right-wing views poorly modeled","German opinion AI study reveals left bias, Llama2 most accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1773,"prompt_tokens":933,"completion_tokens":840,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":750}},"tokens_in":549,"tokens_out":840,"duration_ms":8527,"temperature":1.0,"reasoning_tokens":750,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:20:46.574640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of several hundred human GLES survey responses and code them both with the fine-tuned BERT classifier and with human annotators using the same 16-class scheme; if the classifier's F1 against human coding on this human text is well below the 0.93 reported on LLM text, then every JS distance and model ranking in this paper is potentially an artifact of classifier bias. A second check would be to re-run the full pipeline with a different classifier (for example, a larger multilingual model) and see whether Llama2 still beats Mixtral and still favors left-leaning parties.","supporting_citations":[{"cited_title":"ZA6838 Datenfile Version 6.0.0, https://doi.org/10.4232/1.14114","cited_arxiv_id":null,"evidence_quote":"Provides the original fine-grained coding scheme for the most-important-problem question, which the authors merge into their 16 coarse classes."}],"review_version":1}