Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Mapping free-text LLM answers onto thermometer scales with embeddings fixes the unrealistically peaked distributions that numeric silicon sampling produces.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 03:57 UTC pith:GGBWNSBH

load-bearing objection Solid applied fix for silicon-sampling mode collapse on ANES thermometers: SSR beats numeric prompting on shape/variance with one transferable T, while mean bias stays; the construct-validity gap on the embedding map is real but does not erase the empirical result. the 4 major comments →

arxiv 2607.28550 v1 pith:GGBWNSBH submitted 2026-07-30 cs.CY

Correcting Mode Collapse in Silicon Sampling with Semantic Similarity Rating

Oscar Heath , Rohan Alexander This is my paper
classification cs.CY
keywords Silicon SamplingLarge Language ModelsCalibrationTemperature ScalingSemantic Similarity RatingMode CollapseFeeling ThermometersANES
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Silicon sampling uses large language models as stand-ins for human survey respondents, but the synthetic answers usually collapse onto a narrow peak even when the real answers are spread out. The authors argue this happens because models are weak at emitting numbers, and they show that asking only for text descriptions of feelings, then converting those texts to 0–100 scores via embedding similarity to fixed anchor phrases, yields distributions far closer to real American National Election Studies data. A single global temperature parameter fitted on 2016 data also works on 2020 data. Mean accuracy stays essentially the same while variance and overall shape improve. The method needs little calibration and plays to what language models already do well.

Core claim

Semantic Similarity Rating applied to text-only LLM outputs produces synthetic feeling-thermometer distributions with substantially lower Kullback–Leibler divergence from real ANES responses than direct numeric prompting, better-calibrated variance across respondent–target cells, and no meaningful loss in accuracy of the synthetic means, using one global temperature that generalizes from 2016 to 2020.

What carries the argument

Semantic Similarity Rating (SSR): free-text responses are embedded, cosine similarities to five hand-written anchors at 0/25/50/75/100 are min-max normalized, passed through a temperature-scaled softmax, and smoothed by kernel density estimation into a full density on the thermometer; the single temperature controls spread.

Load-bearing premise

That cosine similarity of free-text answers to five fixed anchor phrases, after normalization and a temperature softmax, recovers the same 0–100 feeling construct that human respondents use on the thermometer.

What would settle it

Apply the same anchors and 2016-fitted temperature to a new ANES wave or a different thermometer battery; if SSR still shows mode collapse or higher KL than numeric prompting, or if rewriting the anchors erases the gains, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Silicon samples can supply usable variance for confidence intervals and power calculations without model fine-tuning.
  • A single temperature learned on one survey year can be reused on later years with little degradation.
  • Thermometer-style items are a natural first target for text-then-embed pipelines in political surveys.
  • Systematic extremity bias in LLM political attitudes is not corrected by SSR and remains a separate problem.
  • Downstream work that needs the full response shape, not just the mean, becomes more trustworthy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same text-then-embed pipeline could be stress-tested on non-political Likert batteries where mode collapse has also been reported.
  • If most of the gain comes from embedding geometry rather than model choice, cheaper open embeddings may preserve most of the improvement.
  • Residual mean bias implies persona prompting still needs separate debiasing before synthetic data can replace humans for point estimates.
  • Pollsters and market researchers using synthetic respondents should prefer free-text elicitation whenever they care about spread, not only means.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that mode collapse in silicon sampling—unrealistically low variance when LLMs are prompted for numeric survey answers—stems from poor numeric generation rather than ignorance of response distributions. Revisiting Bisbee et al. (2024) on ANES feeling thermometers (Democratic/Republican parties, Liberals, Conservatives), it compares direct numeric prompting to Semantic Similarity Rating (SSR): text-only responses are embedded (Gemini Embedding 2), compared by cosine similarity to five hand-written anchors (0/25/50/75/100), min-max normalized, passed through a temperature-scaled softmax, and smoothed by KDE into a 0–100 distribution. Across three frontier models and 36 respondent-group × target cells, SSR yields substantially lower KL divergence to real ANES distributions and better-matched standard deviations, with little change in absolute error of means. A single global temperature T fit on 2016 is reported to transfer to 2020 with similar KL.

Significance. Mode collapse is a documented obstacle to using silicon samples for variance-sensitive inference (CIs, power, distributional comparisons). A training-free post-processing fix with one global calibration parameter, reproducible code/data, multi-model checks, and a 2020 transfer test would be practically useful for political methodology and adjacent fields that already use persona-conditioned LLMs. The design is transparent and builds directly on a known benchmark (Bisbee prompts and ANES cells). Credit is due for releasing code/data, reporting both shape (KL, SD) and location (MAE) metrics, and testing parameter transfer rather than only in-sample fit. The contribution is incremental method engineering rather than a new theory of LLM survey behavior, but that is appropriate for the problem.

major comments (4)
  1. [§3.3, Eqs. 1–3; §4; Table 1] §3.3 and §4 (Eqs. 1–3; Table 1; Fig. 5): The headline claim equates lower KL (and better SD match) under SSR with higher silicon-sampling fidelity. T is chosen by minimizing mean KL on the same 2016 group×target cells used to report that metric (one global T, not per cell—a partial mitigation). Without an independent check that the embedding→anchor→softmax→KDE map recovers the human 0–100 thermometer construct, part of the KL gain can be by construction (geometry + T) rather than better sampling of the survey instrument. A load-bearing addition is construct validation: e.g., human numeric ratings of the same LLM texts, or correlation of SSR scores with held-out human thermometers on matched text, reported separately from the KL-tuned objective.
  2. [§3.3; §4.1; Table 1] §3.3 vs §4.1: The fitted temperature is reported inconsistently as T=0.25 (methods: grid 0–1 by 0.05 minimizing mean KL on 2016) and as T=0.2 (results text and 2020 application; also “0.2” in the variance discussion). This is load-bearing for reproducibility and for the transfer claim in Table 1. Please state a single selected value, the exact selection rule (including whether the grid minimum was unique), and re-run 2016/2020 summaries under that fixed value.
  3. [§3.3; Appendix A.2] Appendix A.2 and §3.3: Anchor wording and locations (five fixed phrases) and the KDE step that turns five softmax masses into Q_t are free design choices not subjected to sensitivity analysis in the main results. If KL and SD calibration are highly sensitive to anchor phrasing or KDE bandwidth, the “few parameters to calibrate” claim overstates robustness. Report at least a small sensitivity check (alternate anchor wordings or leave-one-anchor-out; bandwidth variation) on the same 36 cells, or justify why anchors can be treated as fixed once written.
  4. [Abstract; §5; Fig. 5] §5 and Fig. 5 (bottom): SSR does not reduce systematic mean biases (e.g., synthetic Democrats colder on Conservatives), which the paper correctly notes. The title and abstract frame the contribution as “correcting mode collapse” and “improv[ing] the fidelity” of silicon sampling. That is accurate for variance/shape but easy to over-read as overall distributional correction. Tighten claims so fidelity is explicitly variance/shape fidelity conditional on unchanged location bias, and avoid implying that text+SSR fixes the generative model’s political representation.
minor comments (5)
  1. [§3.3; Figure 1] Figure 1 is referenced as an overview of numeric vs SSR pipelines but the manuscript text does not fully specify KDE bandwidth/kernel; add the bandwidth rule used for all reported densities.
  2. [§3.2] Model names (deepseek-v4-flash, claude-sonnet-5, gpt-5.4-mini) and embedding model (Gemini Embedding 2) should be pinned to exact API/version identifiers and decoding settings (temperature, top-p) for replication; “one response per persona” needs the sampling temperature used at generation time, distinct from SSR’s T.
  3. [§4–§5] Typos/clarity: “mode collpased” (§4); “acknowedge” (§4); “difficulty” (§5). Align “T = 0.25” / “T = 0.2” everywhere once the value is fixed.
  4. [§3; Table 1] KL is reported on continuous densities after KDE; briefly state binning/support and any floor used to avoid log-zero issues so others can recompute Table 1.
  5. [§2] Related work could more clearly separate SSR’s prior use in purchase-intent/Likert settings (Maier et al.; Pichardo) from the novel application and evaluation protocol on ANES thermometers.

Circularity Check

2 steps flagged

In-sample KL gains are partly by construction: global T is chosen to minimize mean KL on 2016, then headline fidelity is reported as lower KL on those same cells; 2020 holdout and the numeric baseline partially mitigate.

specific steps
  1. fitted input called prediction [§3.3 Semantic Similarity Rating (Eq. 3); Results ¶ on T; Table 1]
    "We empirically generated distributions for all values of T between 0 and 1, at increments of 0.05, finding that T = 0.25 minimized the mean KL divergence between the real and synthetic distributions for 2016 ANES data across all groups and questions. ... We briefly acknowedge the concern that we fit a temperature parameter to minimize KL divergence, then quantify improvements using KL divergence."

    T is selected by minimizing the same mean KL that is then used as the headline measure of SSR fidelity on 2016 (Figure 5 top; Table 1 row ‘SSR 2016 with fit parameter’). For the in-sample KL comparison, the reported improvement of calibrated SSR over raw numeric is therefore partly statistically forced by the fit, not an out-of-sample prediction. The paper’s own 2020 transfer and single-global-T defenses reduce but do not remove this reduction for the 2016 KL claim.

  2. fitted input called prediction [§4.1 Results on 2020 ANES Test Set; Table 1]
    "Applying SSR with T = 0.2 to 2020 ANES data, we obtained very similar results to the 2016 ANES data (see Figure 7). Table 1 shows that the KL divergence between the real and SSR synthetic distributions is very similar in 2020 as it was in 2016 when using the same global temperature parameter."

    Mild residual issue only: 2020 reuses the T and hand-written anchors tuned/chosen in the 2016 pipeline, so it is a temporal holdout of personas/questions, not a fully independent re-calibration or a validation that the embedding–anchor map matches human numeric ratings of the same texts. It does break pure in-sample circularity for T, which is why this is secondary to the 2016 fit-then-score step rather than a full collapse.

full rationale

The paper’s load-bearing fidelity claim is that SSR yields synthetic thermometer distributions much closer to ANES than direct numeric prompting, measured primarily by KL divergence (plus related spread). In §3.3 they explicitly grid-search a single global temperature T to minimize mean KL between real and SSR distributions on 2016 ANES across all groups/questions, then in Results/Figure 5/Table 1 report large KL drops on that same 2016 design. That is classic mild fit-then-score circularity for the in-sample KL claim: part of the reported KL improvement is forced by the choice of T. Mitigations the paper itself states and that limit the score: (i) only one global T, not per cell; (ii) they argue T is low so individuals are not flattened to uniform and variance comes from text diversity; (iii) the same T and anchors are applied unchanged to 2020 ANES with similar KL; (iv) MAE of means is essentially unchanged and is not the fitted objective; (v) raw numeric has no analogous free parameter, so the qualitative mode-collapse contrast is not pure definition. No self-definitional loop, no load-bearing self-citation uniqueness theorem, and SSR/anchors are imported from external work rather than redefined as the target. Score 5 reflects partial circularity on the primary in-sample metric without the whole result collapsing to its inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The central fidelity claim rests on a small set of modeling choices rather than a derivation: one fitted temperature, hand-specified anchor texts, a chosen embedding model and asymmetric query/document framing, min-max normalization of similarities, and the domain premise that persona-conditioned LLM text plus embedding geometry recovers ANES thermometer distributions. No new physical entities; the invented pieces are methodological constructs (SSR pipeline configuration for this task).

free parameters (3)
  • SSR temperature T = 0.25 (grid search statement) / 0.2 (later text and 2020 application); manuscript inconsistent
    Single global temperature in the softmax over normalized anchor similarities; chosen by grid search to minimize mean KL on 2016 ANES across groups and questions, then reused on 2020.
  • Anchor locations and wording (0, 25, 50, 75, 100) = Fixed hand-written phrases at 0/25/50/75/100
    Five discrete thermometer anchors and their natural-language descriptions are specified by the authors in Appendix A.2; they define the discrete support before KDE smoothing and are not learned from ANES item text.
  • KDE bandwidth / kernel for smoothing five masses to Q_t
    Gaussian KDE turns five softmax weights into a continuous density on 0–100; bandwidth is part of the mapping but not reported as selected or fixed in the main text.
axioms (5)
  • domain assumption LLMs fail at sampling numeric survey scales even when they encode attitude distributions, so text generation plus post-hoc mapping is a better elicitation channel.
    Stated in abstract and §1–2; motivated by Zhao et al. on LLMs as bad random-number generators and by observed mode collapse under numeric prompts.
  • domain assumption Persona prompts built from ANES demographics/ideology/PID (Bisbee-style) are a valid conditioning mechanism for comparing synthetic vs human thermometer distributions.
    Methods §3.1–3.2 reuse Bisbee prompts; validity of silicon sampling under these personas is assumed for the fidelity comparison.
  • ad hoc to paper Cosine similarity in Gemini Embedding 2 space between response and anchor texts, after min-max normalization, is a monotone indicator of thermometer position.
    Core of §3.3 SSR pipeline; asymmetric embed and min-max follow Pichardo (2026) but are not independently validated here against human thermometer labels on the same texts.
  • domain assumption KL divergence between synthetic and ANES marginals (pooled over respondent-group × target cells) is the right primary fidelity objective for choosing T and claiming improvement.
    Used throughout §3–4; alternatives (Wasserstein, calibration of quantiles, joint distributions) are not primary.
  • standard math Standard probability operations: cosine similarity, min-max scaling, temperature softmax, Gaussian KDE.
    Equations (1)–(3) and KDE step in §3.3.
invented entities (1)
  • SSR-mapped silicon thermometer pipeline (text prompt → embed → 5 anchors → T-softmax → KDE on 0–100) independent evidence
    purpose: Convert free-text LLM feelings into a full probability distribution on the feeling-thermometer scale without asking the model for a number.
    Composition is adapted from Maier/Pichardo-style SSR but configured here (anchors, T, ANES personas, multi-model) as the paper’s operational method; not a new physical object.

pith-pipeline@v1.2.0-daily-grok45 · 15311 in / 3701 out tokens · 71672 ms · 2026-07-31T03:57:27.933030+00:00 · methodology

0 comments
read the original abstract

Silicon sampling refers to the use of Large Language Models (LLMs) to generate responses to surveys. It has shown promise, but tends to generate response distributions with unrealistically low variance. We argue that this mode collapse is due to LLMs failure to generate numeric data, and that text responses may be better suited for this task. We analyze whether Semantic Similarity Rating can improve the fidelity of silicon sampling responses when asked about political attitudes. This method solicits text-only responses from LLMs, then maps this to a numeric scale using text embeddings. We find that this method both improves the fidelity of silicon sampling response distributions, and has few parameters to calibrate.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 1 canonical work pages

  1. [5]

    Using Large Language Models to Analyze Political Texts Through Natural Language Un- derstanding

    “Using Large Language Models to Analyze Political Texts Through Natural Language Un- derstanding. ”American Journal of Political Science , ahead of print. https://doi.org/ https: //doi.org/10.1111/ajps.70050. Bisbee, James, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson

    doi:10.1111/ajps.70050

  2. [8]

    https://arxiv.org/abs/2512.22725

    Mitigating Social Desir- ability Bias in Random Silicon Sampling . https://arxiv.org/abs/2512.22725. DeepSeek-AI

    arXiv

  3. [9]

    Can AI Language Models Replace Human Participants?

    “Can AI Language Models Replace Human Participants?” Trends in Cognitive Sciences 27 (7): 597–600. https://doi.org/https://doi.org/10.1016/j.tics.2023.04.008. Horton, John J., Apostolos Filippas, and Benjamin S. Manning

    doi:10.1016/j.tics.2023.04.008 2023

  4. [10]

    org/abs/2301.07543

    Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? https://arxiv. org/abs/2301.07543. Kaiser, Carolin, Jakob Kaiser, Vladimir Manewitsch, Lea Rau, and Rene Schallner

    arXiv

  5. [11]

    Simulating Human Opinions with Large Language Models: Opportunities and Challenges for Personalized Survey Data Modeling

    “Simulating Human Opinions with Large Language Models: Opportunities and Challenges for Personalized Survey Data Modeling. ” Adjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization (New York, NY, USA), UMAP adjunct ’25, 82–86. https://doi.org/10.1145/3708319.3733685. Le Mens, Gaël, and Aina Gallego

    arXiv

  6. [12]

    Positioning Political Texts with Large Language Models by Asking and A veraging

    “Positioning Political Texts with Large Language Models by Asking and A veraging. ” Political Analysis 33 (3): 274–82. https://doi.org/10.1 017/pan.2024.29. Maier, Benjamin F., Ulf Aslak, Luca Fiaschi, et al

    2024

  7. [13]

    https://arxiv.org/abs/2210.07316

    MTEB: Massive Text Embedding Benchmark . https://arxiv.org/abs/2210.07316. National Research Council

    Pith/arXiv arXiv

  8. [15]

    https://arxiv.org/abs/ 2602.13862

    Measuring Self-Rating Bias in LLM-Generated Survey Data: A Semantic Similarity Framework for Independent Scale Mapping . https://arxiv.org/abs/ 2602.13862. Schröder, Sarah, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen

    arXiv

  9. [16]

    https://arxiv.org/ab s/2508.06950

    Large Language Models Do Not Simulate Human Psychology . https://arxiv.org/ab s/2508.06950. Shanbhogue, Madhuri, Zhe Li, Shanfeng Zhang, et al

    Pith/arXiv arXiv

  10. [17]

    https://arxiv.org/abs/2605.27295

    Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini . https://arxiv.org/abs/2605.27295. Suh, Joseph, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang

    Pith/arXiv arXiv

  11. [18]

    Who Should i Vote for?

    Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions. https://arxiv.org/abs/2502.16761. The New York Times. 2026a. This Is What Will Ruin Public Opinion Polling for Good . https://www.nytimes.com/2026/04/06/opinion/ai-polling.html. The New York Times. 2026b. “Who Should i Vote for?” Voters Turn to a.i. Before Ca...

    Pith/arXiv arXiv 2026

  12. [19]

    I have a very unfavorable opinion of [GROUP]. I strongly dislike [GROUP] and reject almost everything [GROUP] stands for

    Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions . https: //arxiv.org/abs/2601.05414. 19 A Appendix A.1 Prompt Formats (identical to Bisbee et al. (2024)): A.1.1 System Prompt: “It is [ YEAR]. You are a [ AGE] year-old, [ MARST], [ RACETH] [ GENDER] with [ EDUCATION] making [INCOME] per ye...

    Pith/arXiv arXiv 2024

  13. [2013]

    Edited by Roger Tourangeau and Thomas J

    Nonresponse in Social Science Surveys: A Research Agenda . Edited by Roger Tourangeau and Thomas J. Plewes. The National Academies Press. https: //doi.org/10.17226/18293. OpenAI

    work page doi:10.17226/18293

  14. [2016]

    ANES 2016 Time Series Study Full Release

    “ANES 2016 Time Series Study Full Release. ” www.electionstudies.org. American National Election Studies

    2016

  15. [2020]

    ANES 2020 Time Series Study Full Release

    “ANES 2020 Time Series Study Full Release. ” www.electionstudies.org. Anthropic

    2020

  16. [2023]

    Out of One, Many: Using Language Models to Simulate Human Samples

    “Out of One, Many: Using Language Models to Simulate Human Samples. ” Political Analysis 31 (3): 337–51. https://doi.org/10.1017/pan.2023.2. Bain and Company

    doi:10.1017/pan.2023.2 2023

  17. [2024]

    Synthetic Replacements for Human Survey Data? The Perils of Large Language Models

    “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. ” Political Analysis 32 (4): 401–16. https://doi.org/10.1017/pan.2024.5. Brookings

    doi:10.1017/pan.2024.5 2024

  18. [2025]

    Specializing Large Language Models to Simulate Survey Response Distri- butions for Global Populations

    “Specializing Large Language Models to Simulate Survey Response Distri- butions for Global Populations. ” In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , edited by Luis Chiruzzo, Alan Ritter, and Lu Wang. Association for Com...

    doi:10.18653/v1/2025.naacl- 2025

  19. [2026]

    American National Election Studies

    Simulating Gun Control Attitudes After the 2025 Bondi Beach Shooting Using Persona-Conditioned LLMs . American National Election Studies

    2025