{"id":"40c4888d-aa45-4042-ac18-ad1ce97f789b","arxiv_id":"2607.28550","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Semantic Similarity Rating of LLM text responses substantially reduces mode collapse in silicon sampling of political thermometer scores versus direct numeric prompting, with one global temperature that generalizes from 2016 to 2020 ANES.","lead":"Text-only LLM survey answers, mapped to 0–100 scores via embeddings (Semantic Similarity Rating), fix the unrealistically peaked distributions that plague direct numeric silicon sampling. A single temperature parameter learned on 2016 ANES largely carries to 2020, improving distribution shape without much harm to mean accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"SSR’s KL gains may largely reflect embedding/anchor geometry plus a KL-tuned temperature, not recovery of the human thermometer construct.","rationale":"The reader correctly located the load-bearing assumption: that the embedding–anchor–softmax–KDE stack recovers the human thermometer construct so that KL/SD improvements count as sampling fidelity. I agree that is the soft spot; the multi-model pattern and 2016→2020 transfer with one global T are real strengths and keep the paper above REJECT, but without a human validation of the mapping (or a strong anchor/embedding ablation) the causal story “text fixes numeric mode collapse” remains under-supported relative to “SSR plus KL-tuned T produces ANES-like histograms.” Mean bias persistence, already noted by the authors and Bisbee et al., further limits how far distributional KL can be read as corrected silicon sampling. Verdict stays CONDITIONAL; the concrete human-rating check is the single experiment that would most cleanly settle whether the concern lands. No change to ACCEPT without that (or equivalent) evidence; no move to REJECT given the clear, reproducible improvement over raw numeric prompting on the reported metrics.","tokens_in":11292,"tokens_out":652,"duration_ms":12767,"concrete_test":"Take a stratified sample of SSR text outputs (e.g. 200–400 across the 36 cells). Collect human 0–100 thermometer ratings of those texts under the standard ANES feeling-thermometer instructions, plus (i) the SSR expected value under the paper’s anchors/T and (ii) an alternate anchor set or embedding model. If correlation of SSR scores with human ratings is weak (e.g. r<0.5) or cell-level KL to ANES rises sharply under the alternate mapping while human-aligned scores do not beat numeric baselines, the fidelity interpretation of Figures 5–6 and Table 1 does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim equates lower KL (and better SD match) under SSR with higher silicon-sampling fidelity. That only holds if the pipeline in §3.3—Gemini Embedding 2 cosine similarities to five hand-written anchors (App. A.2), min-max normalization (Eq. 2), temperature-scaled softmax (Eq. 3), then KDE—maps free text onto the same 0–100 feeling-thermometer construct that ANES respondents use. Nothing in the paper validates that mapping against human numeric ratings of the same texts. T is chosen by minimizing mean KL on the same 2016 cells used to report the headline metric (T reported as both 0.25 and 0.2), so part of the KL drop is by construction; the 2020 “holdout” reuses the same anchors and embedding model. Mean absolute error is essentially unchanged and known directional biases remain, which is consistent with SSR mostly reshaping mass via geometry and T rather than correcting the generative model’s political representation. If the embedding/anchor step is misaligned with the survey instrument, improved KL is an artifact, not evidence that mode collapse in silicon sampling has been corrected.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that mode collapse in silicon sampling—unrealistically low variance when LLMs are prompted for numeric survey answers—stems from poor numeric generation rather than ignorance of response distributions. Revisiting Bisbee et al. (2024) on ANES feeling thermometers (Democratic/Republican parties, Liberals, Conservatives), it compares direct numeric prompting to Semantic Similarity Rating (SSR): text-only responses are embedded (Gemini Embedding 2), compared by cosine similarity to five hand-written anchors (0/25/50/75/100), min-max normalized, passed through a temperature-scaled softmax, and smoothed by KDE into a 0–100 distribution. Across three frontier models and 36 respondent-group × target cells, SSR yields substantially lower KL divergence to real ANES distributions and better-matched standard deviations, with little change in absolute error of means. A single global temperature T fit on 2016 is reported to transfer to 2020 with similar KL.","tokens_in":11588,"tokens_out":1532,"duration_ms":32767,"significance":"Mode collapse is a documented obstacle to using silicon samples for variance-sensitive inference (CIs, power, distributional comparisons). A training-free post-processing fix with one global calibration parameter, reproducible code/data, multi-model checks, and a 2020 transfer test would be practically useful for political methodology and adjacent fields that already use persona-conditioned LLMs. The design is transparent and builds directly on a known benchmark (Bisbee prompts and ANES cells). Credit is due for releasing code/data, reporting both shape (KL, SD) and location (MAE) metrics, and testing parameter transfer rather than only in-sample fit. The contribution is incremental method engineering rather than a new theory of LLM survey behavior, but that is appropriate for the problem.","major_comments":[{"comment":"§3.3 and §4 (Eqs. 1–3; Table 1; Fig. 5): The headline claim equates lower KL (and better SD match) under SSR with higher silicon-sampling fidelity. T is chosen by minimizing mean KL on the same 2016 group×target cells used to report that metric (one global T, not per cell—a partial mitigation). Without an independent check that the embedding→anchor→softmax→KDE map recovers the human 0–100 thermometer construct, part of the KL gain can be by construction (geometry + T) rather than better sampling of the survey instrument. A load-bearing addition is construct validation: e.g., human numeric ratings of the same LLM texts, or correlation of SSR scores with held-out human thermometers on matched text, reported separately from the KL-tuned objective.","section":"§3.3, Eqs. 1–3; §4; Table 1"},{"comment":"§3.3 vs §4.1: The fitted temperature is reported inconsistently as T=0.25 (methods: grid 0–1 by 0.05 minimizing mean KL on 2016) and as T=0.2 (results text and 2020 application; also “0.2” in the variance discussion). This is load-bearing for reproducibility and for the transfer claim in Table 1. Please state a single selected value, the exact selection rule (including whether the grid minimum was unique), and re-run 2016/2020 summaries under that fixed value.","section":"§3.3; §4.1; Table 1"},{"comment":"Appendix A.2 and §3.3: Anchor wording and locations (five fixed phrases) and the KDE step that turns five softmax masses into Q_t are free design choices not subjected to sensitivity analysis in the main results. If KL and SD calibration are highly sensitive to anchor phrasing or KDE bandwidth, the “few parameters to calibrate” claim overstates robustness. Report at least a small sensitivity check (alternate anchor wordings or leave-one-anchor-out; bandwidth variation) on the same 36 cells, or justify why anchors can be treated as fixed once written.","section":"§3.3; Appendix A.2"},{"comment":"§5 and Fig. 5 (bottom): SSR does not reduce systematic mean biases (e.g., synthetic Democrats colder on Conservatives), which the paper correctly notes. The title and abstract frame the contribution as “correcting mode collapse” and “improv[ing] the fidelity” of silicon sampling. That is accurate for variance/shape but easy to over-read as overall distributional correction. Tighten claims so fidelity is explicitly variance/shape fidelity conditional on unchanged location bias, and avoid implying that text+SSR fixes the generative model’s political representation.","section":"Abstract; §5; Fig. 5"}],"minor_comments":[{"comment":"Figure 1 is referenced as an overview of numeric vs SSR pipelines but the manuscript text does not fully specify KDE bandwidth/kernel; add the bandwidth rule used for all reported densities.","section":"§3.3; Figure 1"},{"comment":"Model names (deepseek-v4-flash, claude-sonnet-5, gpt-5.4-mini) and embedding model (Gemini Embedding 2) should be pinned to exact API/version identifiers and decoding settings (temperature, top-p) for replication; “one response per persona” needs the sampling temperature used at generation time, distinct from SSR’s T.","section":"§3.2"},{"comment":"Typos/clarity: “mode collpased” (§4); “acknowedge” (§4); “difficulty” (§5). Align “T = 0.25” / “T = 0.2” everywhere once the value is fixed.","section":"§4–§5"},{"comment":"KL is reported on continuous densities after KDE; briefly state binning/support and any floor used to avoid log-zero issues so others can recompute Table 1.","section":"§3; Table 1"},{"comment":"Related work could more clearly separate SSR’s prior use in purchase-intent/Likert settings (Maier et al.; Pichardo) from the novel application and evaluation protocol on ANES thermometers.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"Fit for a methods-oriented political science / computational social science venue is reasonable if construct validation of the SSR map is added; without it the paper risks being read as a clever recalibration of embedding geometry rather than a fix for silicon sampling. The 2020 transfer is a genuine strength but reuses the same anchors and embedder, so it does not fully break dependence on the mapping design. I would not reject on novelty grounds—the Bisbee replication frame plus multi-model KL/SD evidence is enough—but I would hold the line on independent validation of the text→thermometer map before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a clean, useful methods paper. Against the Bisbee-style numeric silicon-sampling baseline, text-only responses mapped through SSR give much better distributional fidelity (KL and SD) on ANES feeling thermometers across three models and 36 group×target cells, with mean absolute error essentially unchanged and a single global temperature that transfers from 2016 to 2020.\n\nWhat is new is not SSR itself (Maier et al.; related mapping in Pichardo) but the head-to-head on political silicon sampling: same personas as Bisbee, multi-model comparison, explicit KL/MAE/SD reporting, and a real holdout year for T. Code and data are shipped. Figures 2–7 and Table 1 make the shape/variance claim easy to check; the paper is also honest that directional mean biases (e.g., Democrats on Conservatives) survive. That honesty helps.\n\nSoft spots, in proportion. T is fit by minimizing mean KL on the same 2016 cells used for the headline metric—mild circularity they partly blunt with one global T and the 2020 transfer. They report both 0.25 and 0.2; that should be cleaned up. Free parameters remain (anchor wording/locations, embedding model, KDE). The stress-test concern is fair: nothing validates that Gemini Embedding 2 cosine-to-five-handwritten-anchors recovers the same 0–100 construct humans use when they pick a number. Lower KL could partly be geometry plus T rather than pure recovery of the survey instrument. That weakens the strongest causal language (“correcting mode collapse because LLMs can’t do numbers”), not the empirical comparison to numeric prompting. No uncertainty on the metrics, no anchor/embedding ablation.\n\nWho it is for: anyone doing or critiquing silicon sampling in political science or survey work who needs usable variance. Worth a serious referee. I would bring it to reading group and would cite the empirical pattern. Send to peer review; ask for tighter T reporting, uncertainty, and at least a light sensitivity check on anchors/embeddings.","headline":"Solid applied fix for silicon-sampling mode collapse on ANES thermometers: SSR beats numeric prompting on shape/variance with one transferable T, while mean bias stays; the construct-validity gap on the embedding map is real but does not erase the empirical result.","tokens_in":12247,"tokens_out":534,"would_cite":true,"duration_ms":9900,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Mapping free-text LLM answers onto thermometer scales with embeddings fixes the unrealistically peaked distributions that numeric silicon sampling produces.","keywords":["Silicon Sampling","Large Language Models","Calibration","Temperature Scaling","Semantic Similarity Rating","Mode Collapse","Feeling Thermometers","ANES"],"falsifier":"Apply the same anchors and 2016-fitted temperature to a new ANES wave or a different thermometer battery; if SSR still shows mode collapse or higher KL than numeric prompting, or if rewriting the anchors erases the gains, the central claim fails.","tokens_in":12110,"feed_emoji":"📊","tokens_out":799,"duration_ms":29516,"temperature":0.7,"pith_summary":"Silicon sampling uses large language models as stand-ins for human survey respondents, but the synthetic answers usually collapse onto a narrow peak even when the real answers are spread out. The authors argue this happens because models are weak at emitting numbers, and they show that asking only for text descriptions of feelings, then converting those texts to 0–100 scores via embedding similarity to fixed anchor phrases, yields distributions far closer to real American National Election Studies data. A single global temperature parameter fitted on 2016 data also works on 2020 data. Mean accuracy stays essentially the same while variance and overall shape improve. The method needs little calibration and plays to what language models already do well.","feed_headline":"Text-then-embed fixes LLM survey mode collapse","feed_subtitle":"One temperature maps free-text answers onto real thermometer spreads without hurting means","key_machinery":"Semantic Similarity Rating (SSR): free-text responses are embedded, cosine similarities to five hand-written anchors at 0/25/50/75/100 are min-max normalized, passed through a temperature-scaled softmax, and smoothed by kernel density estimation into a full density on the thermometer; the single temperature controls spread.","core_discovery":"Semantic Similarity Rating applied to text-only LLM outputs produces synthetic feeling-thermometer distributions with substantially lower Kullback–Leibler divergence from real ANES responses than direct numeric prompting, better-calibrated variance across respondent–target cells, and no meaningful loss in accuracy of the synthetic means, using one global temperature that generalizes from 2016 to 2020.","pith_inferences":["The same text-then-embed pipeline could be stress-tested on non-political Likert batteries where mode collapse has also been reported.","If most of the gain comes from embedding geometry rather than model choice, cheaper open embeddings may preserve most of the improvement.","Residual mean bias implies persona prompting still needs separate debiasing before synthetic data can replace humans for point estimates.","Pollsters and market researchers using synthetic respondents should prefer free-text elicitation whenever they care about spread, not only means."],"forward_implications":["Silicon samples can supply usable variance for confidence intervals and power calculations without model fine-tuning.","A single temperature learned on one survey year can be reused on later years with little degradation.","Thermometer-style items are a natural first target for text-then-embed pipelines in political surveys.","Systematic extremity bias in LLM political attitudes is not corrected by SSR and remains a separate problem.","Downstream work that needs the full response shape, not just the mean, becomes more trustworthy."],"fun_headline_variants":["Text embeddings fix silicon sampling mode collapse","SSR maps LLM free text to real thermometer spreads","Semantic similarity beats numeric prompts on ANES","One temperature restores survey variance from text","Text-only LLMs plus embeddings match feeling thermometers"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That cosine similarity of free-text answers to five fixed anchor phrases, after normalization and a temperature softmax, recovers the same 0–100 feeling construct that human respondents use on the thermometer.","fun_headline_variants_meta":{"raw":{"variants":["Text embeddings fix silicon sampling mode collapse","SSR maps LLM free text to real thermometer spreads","Semantic similarity beats numeric prompts on ANES","One temperature restores survey variance from text","Text-only LLMs plus embeddings match feeling thermometers"]},"model":"grok-4.5","effort":"low","cost_usd":0.003779,"raw_usage":{"total_tokens":1131,"prompt_tokens":643,"num_sources_used":0,"completion_tokens":51,"cost_in_usd_ticks":37788000,"prompt_tokens_details":{"text_tokens":643,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":437,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":643,"tokens_out":51,"duration_ms":7715,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T03:57:27.933030+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Apply the same anchors and 2016-fitted temperature to a new ANES wave or a different thermometer battery; if SSR still shows mode collapse or higher KL than numeric prompting, or if rewriting the anchors erases the gains, the central claim fails.","supporting_citations":[],"review_version":1}