{"id":"7fab6a99-638c-42da-899b-5c77113b8b90","arxiv_id":"2501.14294","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs systematically inflate Republican positions and deflate Democratic positions relative to human survey responses, consistent with representativeness heuristics.","lead":"This paper asks whether AI chatbots exaggerate where US political parties stand, compared with what party members actually report. It finds that chatbots do exaggerate, in both directions, and that simple prompt warnings only partially correct the bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The representativeness test in Eqs. 5–6 is circular: P is computed from the same model output distribution that defines the predicted mean, so positive ε can be mechanical; a null model is missing.","rationale":"I read the paper as a new application of Bordalo et al.'s formalization to LLMs, with a reasonable descriptive foundation. The raw exaggeration finding is plausible: Figure 4 and Table 12 show consistent directional deviations across models and datasets, and the authors release code, which is independent support. The weakest point is not the raw result but the attribution to representativeness. In Bordalo's theory, representativeness is computed from empirical distributions; here it is computed from the same model distribution whose mean is being explained. This violates the independence needed for a mechanism test. The reader's weakest_assumption identifies the same circularity, and I agree with it. A reanalysis using empirical representativeness, together with a null model for ε, would settle whether the 'overemphasizing representativeness' claim survives or collapses into a descriptive observation about polarized means. Since the reader already recommended conditional acceptance with exactly these requirements, my read does not change the verdict.","tokens_in":25975,"tokens_out":6824,"duration_ms":68867,"concrete_test":"Recompute the Table 2/14 heuristic analysis with P_{A(N)} defined from empirical group distributions (ANES/MFQ empirical responses, add-one smoothed) rather than from the model's output distribution, while keeping the predicted means E^B(a|X+) unchanged. If most ε values become nonpositive or statistically indistinguishable from zero, the representativeness-mechanism claim fails and the paper reduces to a descriptive polarization result. As a complementary check, construct a null distribution that matches each LLM's mean but has no tail concentration (e.g., maximum entropy on the Likert scale) and verify whether positive ε persists; if it does, the current test is mechanical.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: a descriptive one (LLM predictions are more polarized than human predictions) and a mechanism one (this reflects representativeness heuristics). The descriptive part is reasonable. The mechanism part rests on Eqs 5–6, where ε is treated as evidence of representativeness. In the implementation, P^L_{A(N)} is computed from the same 20-response LLM distribution p^B that determines E^B(a|X+), via the exemplar definition in Eq 2 and Appendix C. Consequently, Eq 5 is not a fitted relationship between an independent representativeness measure and a predicted mean; it is an algebraic restatement: ε = (E^B(a|X+) − E(a|X+))/(P^L − 1). Whenever the model's distribution is shifted toward the high end relative to the empirical Republican distribution, and the high end has above-average tail ratio in the model's own distribution, ε is positive automatically. A model with no heuristic whatsoever—one that anchors to endpoints, repeats a default extreme, or has miscalibrated variance—would produce the same sign pattern. The same criticism applies to the Human_Pred comparison and to the κ mitigation results in Table 3, because κ also uses p^B in Eq 3. The paper supplies no null model, permutation test, or significance threshold for ε; the only statistics reported are means and standard deviations across topics (Table 14). Thus the evidence for 'overemphasizing representativeness more than humans' is not yet established, even though the raw exaggeration finding may hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether LLMs' predictions of US political party positions (Republicans, Democrats) on ANES and MFQ survey items deviate systematically from empirical human self-placements and from human predictions of party positions. The authors formalize stereotypes via the representativeness heuristic: an attribute is representative if its likelihood ratio P(a|X+)/P(a|X-) is high. They fit a kernel-of-truth parameter γ (Eq. 4) and a representativeness parameter ε (Eqs. 5–6), report that LLM predictions are more polarized than both empirical means and human predictions, that ε is often positive, and that prompt-based mitigation strategies (AWARENESS, REASONING, FEEDBACK) reduce κ (Eq. 3). The paper concludes that LLMs are susceptible to representativeness heuristics, which facilitates political stereotyping.","tokens_in":26270,"tokens_out":2337,"duration_ms":23346,"significance":"If the results hold, the paper would provide a theoretically grounded, cross-model characterization of a specific form of political bias in LLMs—systematic exaggeration of partisan positions beyond human exaggeration—and would connect it to a well-known cognitive heuristic. The descriptive part is useful and relatively robust: across six LLMs and two datasets, predicted means for Republicans are consistently higher than empirical means and human predictions, and for Democrats consistently lower. The paper ships code and uses public survey data, which supports reproducibility. However, the mechanism claim (that the exaggeration is due to representativeness heuristics) rests on fitted parameters ε, γ, and κ that, as implemented, are mathematically coupled to the very distributions they are supposed to explain. The paper provides no null model, permutation test, or confidence intervals for these parameters. Thus the paper's headline contribution—'LLMs overemphasize representativeness more than humans'—is not yet established, even though the raw exaggeration pattern is credible.","major_comments":[{"comment":"The representativeness test is circular in the current implementation. The exemplar a* in Eq. (2) and the right-tail set A(N) are computed from the model's own response distribution p^B_{a,X+}/p^B_{a,X-}, and the predicted mean E^B(a|X+) is also computed from that same p^B. Eq. (5) therefore does not relate an independent representativeness measure to a predicted mean; it solves for ε as (E^B(a|X+) − E(a|X+))/(P^L_{A(N)} − 1), which is an algebraic identity given the definitions. Any model whose distribution is shifted toward high-scale values relative to the empirical distribution will produce positive ε whenever the right tail has above-average representativeness in the model's own distribution. A model with no heuristic—say, one that anchors on endpoints, repeats a default extreme, or has miscalibrated variance—would yield the same sign pattern. The same concern applies to the κ measure in Eq. (3), which also uses p^B, and to the Human_Pred comparison, since the same construction is applied to human predictions. The manuscript reports only means and standard deviations across topics (Table 14) and no significance test, null model, or permutation baseline. The conclusion that 'LLMs tend to overemphasize representativeness more than humans' (Abstract) is therefore not supported by the evidence as presented.","section":"§3, Eqs. (5)–(6), Appendix C"},{"comment":"The fitted parameters γ and ε are reported without uncertainty quantification. Table 1 reports γ values with standard deviations in parentheses, but no significance tests, confidence intervals, or model comparison against a null model (e.g., γ=0). Table 2 reports ε with standard deviations, but again no test of whether ε differs from zero, let alone whether LLM ε exceeds Human_Pred ε. Given that several entries are within one standard deviation of zero (e.g., Llama2-70b ANES R ε = −0.84±4.96, Gpt-4 ANES R ε = −0.08±2.60), the claim of systematic positive ε is not statistically supported. The paper should provide confidence intervals or bootstrap/permutation tests for ε and γ, and should test whether the LLM ε distribution is stochastically larger than the Human_Pred ε distribution.","section":"§6, Table 1 and Table 2"},{"comment":"The mitigation analysis uses κ (Eq. 3) as the outcome measure, but κ is computed from the same model distribution p^B that the prompting interventions are expected to change. If ε is mechanically positive for shifted distributions, then a prompt that flattens the distribution or moves it toward the empirical mean will mechanically reduce κ, regardless of whether the intervention actually targets a representativeness heuristic. The claim that the mitigation strategies are 'effective in reducing the influence of representativeness' (Abstract) therefore conflates a distributional shift with a reduction in heuristic use. A proper test would compare the intervention's effect on ε or κ against a null model where the intervention merely rescales or shifts the distribution without changing its tail representativeness.","section":"§6, Table 3"},{"comment":"The kernel-of-truth analysis (Eq. 4) has a sign-direction issue that is acknowledged only in a footnote. The paper states that Eq. 4 holds 'if and only if the group has a higher average position than the other group' (footnote 3), but the empirical means in ANES do not always satisfy this ordering (e.g., Women's Rights: Republican empirical mean 2.83 vs Democratic 2.56; the prediction for Republicans is higher, but for Democrats the mean difference is small). For topics where the ordering is reversed or where the two means are close, γ can become large or negative for mechanical reasons, as seen in Table 13 (e.g., Llama2-70b γ = 4.18 for Women's Rights R, but also γ = −1.14 for Legal Rights R). The paper should filter or report which topics satisfy the ordering condition, or use a more general formulation that does not require the ordering assumption. Without this, the kernel-of-truth γ is not a clean measure of 'inflation'.","section":"§6, Table 1; §10 Limitations"}],"minor_comments":[{"comment":"The phrase 'the extent and conditions under which LLMs deviate from empirical positions are insufficiently examined' is repeated almost verbatim in the Abstract and Introduction; consider varying the wording.","section":"Abstract and §1"},{"comment":"The exemplar definition and the Laplace smoothing are described in the main text and appendix, but the value of N in A(N) is set to 2 in the main text and then the appendix says 'We set N = 2 for our analysis' earlier; please make the N selection justification explicit (e.g., robustness checks for N=1,3).","section":"§3, Eq. (2) and Appendix A"},{"comment":"The ANES sample sizes in Table 4 are very large (often >10,000 per party per topic), but the LLM responses are based on 20 repetitions per prompt. The comparison of a near-deterministic LLM (some models have zero variance in Table 12) with a large human sample should be described as a comparison of point estimates, not of distributions, and the paper should avoid implying that a small number of LLM samples can be compared to human sampling variability.","section":"§5.1 and Table 4"},{"comment":"The temperature sensitivity analysis reports CV and mean differences, but does not report the standard deviation of the Diff_D and Diff_R across temperatures; the claim that 'the deviation from the empirical mean remains relatively consistent' would be stronger with a measure of dispersion around the mean difference.","section":"Appendix E, Table 6"},{"comment":"The misinformation detection experiment is labeled as preliminary, but the accuracy values for some cells (e.g., 0.739 for Llama2-70b Democrats with party) appear to be single-point estimates without error bars. Please add confidence intervals or note the small sample sizes for Democrat-labeled statements.","section":"Appendix G, Table 10"},{"comment":"The paper alternates between 'Republican' and 'Democrat' labels and 'X+'/'X−' notation inconsistently in tables and figures. For instance, Table 2 uses R and D, but Eqs. (5)–(6) use ε_{X+} and ε_{X−}; please define the correspondence clearly in each table caption.","section":"Throughout"},{"comment":"The reference list is missing several recent works that would help contextualize the contribution, such as studies on political bias in LLMs beyond the cited ones (e.g., 'Whose opinions do language models reflect?' is cited as Santurkar et al. 2023, but the citation style is inconsistent). Please conduct a final reference check.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper presents a valuable descriptive finding—LLMs systematically exaggerate partisan positions relative to humans—but the mechanism claim is undermined by a methodological circularity in the operationalization of representativeness. The ε and κ parameters are computed from the same distribution that defines the predicted mean, so positive values may be an algebraic artifact of distributional shift, not evidence of a heuristic. This is a load-bearing issue because the title, abstract, and conclusions emphasize the heuristic interpretation. A careful revision that adds null models, permutation tests, and explicit comparisons of LLM ε against Human_Pred ε with uncertainty quantification could make the mechanism claim credible. I recommend major revision rather than rejection because the descriptive contribution is sound and the theoretical framework is appropriate; the issue is fixable within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is descriptive: across ANES and MFQ, six very different LLMs consistently place Republicans further right and Democrats further left than both empirical survey means and human predictions. That pattern is plausible, consistent, and visually robust. The kernel-of-truth analysis, which compares model outputs against external human survey data, is not circular and is a genuine new application of Bordalo et al.'s formalism to LLM behavior. The prompt-based mitigation experiments (awareness, feedback, reasoning) are also a nice addition, with code released and some human evaluation of response coherence. Credit where due: this is a fresh angle in a crowded field.\n\nThe soft spot is load-bearing. The claim that LLMs \"overemphasize representativeness\" rests on epsilon in Eqs. 5–6, and epsilon is effectively computed from the same model output distribution that defines the predicted mean it is supposed to explain. When the right-tail representativeness ratio P^L is derived from p^B, and E^B is the mean of p^B, a positive epsilon can be an algebraic restatement of a shifted distribution, not evidence of a cognitive heuristic. A model with calibrated but high-variance outputs, or one that anchors to endpoints, would produce the same sign pattern. The paper gives no null model, permutation test, or confidence interval. The kappa-based mitigation results in Table 3 inherit the same issue, so the \"mitigation reduces representativeness\" claim is shaky as a mechanism claim, even if the prompt effects on raw outputs are real. This is not a manufactured flaw; it is the difference between describing polarized outputs and attributing them to a specific heuristic.\n\nMinor issues: N=20 responses per model per prompt is on the low side, and while the temperature sensitivity check helps, it does not rescue the epsilon test. The scale-reversal design is handled carefully, which I appreciate.\n\nWho is this for? Anyone working on political bias in LLMs or applying cognitive-science formalizations to model outputs. The descriptive part is a useful data point, and the framework is transferable even if the current mechanism test is not convincing. It deserves a serious referee—the question it poses is important, and the descriptive results are solid enough to warrant publication once the mechanism claim is either reframed as \"the outputs are consistent with a representativeness-style weighting\" or supported by a proper null model and significance testing. I would send it out, but with the circularity front and center in the review request.","headline":"The descriptive finding (LLMs exaggerate partisan positions) is solid and worth knowing, but the representativeness-heuristic mechanism as tested is circular and should be reframed or fixed.","tokens_in":26806,"tokens_out":1682,"would_cite":true,"duration_ms":16944,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models systematically exaggerate US party positions beyond human survey respondents, in a pattern consistent with representativeness heuristics.","keywords":["representativeness heuristics","political stereotyping","large language models","alignment","kernel of truth","likelihood ratio","prompt mitigation","survey response bias"],"falsifier":"Simulate synthetic LLM responses by resampling the empirical partisan distributions with added noise but no representativeness weighting, then fit the paper's two linear equations to the simulated means; if positive epsilon appears as often as in the real model outputs, the representativeness mechanism is not identified by this test. Alternatively, hold out half of each model's responses to compute the representativeness ratio and the other half to compute the predicted mean; if epsilon collapses, the effect is an artifact of same-distribution regression.","tokens_in":25759,"feed_emoji":"🗳️","tokens_out":3794,"duration_ms":33692,"temperature":0.7,"pith_summary":"This paper tries to show that when large language models are asked to place the Democratic and Republican parties on political attitude scales, their answers land further from the empirical center than human survey respondents' guesses do: Republican positions are inflated, Democratic positions deflated, and the gap between the two is stretched. The authors borrow the cognitive-science idea of representativeness heuristics, where people exaggerate attributes that are diagnostically distinctive of a group, and test whether LLM responses obey the same formal pattern. They find a kernel of truth: model predictions correlate with real partisan differences, then amplify them. If the claim holds, LLM outputs about US politics are not merely left- or right-leaning but systematically stereotyped, and the same measurement lens could expose equivalent distortions for other groups.","feed_headline":"LLMs exaggerate party positions more than humans do","feed_subtitle":"Across seven models, predicted Republican stances run high and Democratic stances low, stretching real partisan gaps.","key_machinery":"The load-bearing formal object is the representativeness ratio R[a] = p(a|X+)/p(a|X-), the likelihood that a scale position occurs for Republicans versus Democrats. The paper defines the most representative attribute, the exemplar, as the argmax of this ratio, defines kappa as the ratio between the model's conditional probability at the exemplar and its representativeness, and uses two linear equations to express the predicted mean as the empirical mean plus an epsilon-weighted excess of tail representativeness. Positive epsilon is taken as evidence that the model overweights representative attributes when forming predictions.","core_discovery":"On the paper's own terms, the central discovery is a quantitative profile of LLM political stereotyping. Across seven models and two survey instruments, asking a model 'where would you place the Democratic/Republican Party' yields mean responses that consistently overshoot the empirical means of self-identified partisans: Republican-associated predictions are too high, Democratic-associated predictions are too low, and predicted party differences exceed both empirical differences and human predictions. The deviations fit the kernel-of-truth equation (positive gamma), meaning predictions track real group differences while stretching them, and the representativeness equations yield positive epsilon for most model-topic combinations, which the authors read as evidence that likelihood-ratio representativeness, not just random noise, shapes the exaggeration. Prompt strategies modeled on human debiasing, such as awareness, feedback, and reasoning instructions, reduce the kappa measure of stereotyping but do not eliminate it.","pith_inferences":["An untested implication is that the same protocol would show analogous exaggeration for non-political groups such as gender, nationality, or ethnicity, since the likelihood-ratio formalism is group-agnostic and the training-data mechanisms are not politics-specific.","The authors' own comparison of an instruction-tuned model with its base version hints that alignment training may increase rather than decrease this exaggeration; a direct test would compare base and instruction-tuned checkpoints of the same model on the full battery.","A practical extension is to use the epsilon coefficient as a monitor for political stereotyping during model deployment, recalibrating prompts or filtering outputs when epsilon drifts upward.","The downstream misinformation experiment is only suggestive; a causal test would manipulate the representativeness of the provided party context and measure changes in truth judgments."],"forward_implications":["LLM-generated survey-style outputs about US parties are systematically more polarized than the underlying empirical positions, so downstream systems that consume such outputs inherit exaggerated party differences.","Because the exaggeration exceeds human predictive bias, alignment evaluations that compare models to human average guesses will understate the distortion.","The kernel-of-truth finding implies the distortion is not arbitrary: models reproduce real partisan correlations and then amplify them, so mitigation cannot simply erase the signal.","Prompt-based self-correction can lower the measured stereotyping proxy but not remove it, suggesting the bias is partly baked into the model rather than purely a surface prompt effect.","The same likelihood-ratio lens can be applied to any paired target and reference group to test whether the exaggeration pattern is general."],"supporting_citations":[{"why":"Supplies the definition of representativeness heuristics as the overweighting of diagnostic attributes of a target group.","marker":"Kahneman & Tversky, 1972"},{"why":"Supplies the kernel-of-truth equation and the formal likelihood-ratio model of stereotypes that the paper operationalizes.","marker":"Bordalo et al., 2016"},{"why":"Supplies the cognitive-science basis for exaggeration and probabilistic judgment errors from heuristics.","marker":"Benjamin, 2019"},{"why":"Supplies the kernel-of-truth assumption that stereotypes are grounded in empirical realities.","marker":"Judd & Park, 1993"},{"why":"Supplies the survey data for empirical partisan positions and human prediction baselines.","marker":"American National Election Studies, 2022"},{"why":"Supplies the Moral Foundations Questionnaire dataset with party affiliation used for the second experiment.","marker":"Talaifar & Swann Jr, 2019"},{"why":"Supplies the self-correction idea behind the awareness and feedback mitigation prompts.","marker":"Kahneman, 2013"}],"fun_headline_variants":["LLMs overstate party stances, human surveys don't","Political stereotypes surface in LLM survey answers","Why AI predictions run hotter on party positions","LLM responses stretch partisan gaps beyond reality","Representativeness heuristics skew LLM political answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The mechanism claim rests on treating deviations of the predicted mean from the empirical mean as evidence of representativeness when the representativeness ratio is computed from the same model-output distribution that defines the predicted mean, and no null model shows that positive epsilon cannot arise mechanically from that circularity.","fun_headline_variants_meta":{"raw":{"variants":["LLMs overstate party stances, human surveys don't","Political stereotypes surface in LLM survey answers","Why AI predictions run hotter on party positions","LLM responses stretch partisan gaps beyond reality","Representativeness heuristics skew LLM political answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1465,"prompt_tokens":1000,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":616,"tokens_out":465,"duration_ms":4594,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:14:29.595839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate synthetic LLM responses by resampling the empirical partisan distributions with added noise but no representativeness weighting, then fit the paper's two linear equations to the simulated means; if positive epsilon appears as often as in the real model outputs, the representativeness mechanism is not identified by this test. Alternatively, hold out half of each model's responses to compute the representativeness ratio and the other half to compute the predicted mean; if epsilon collapses, the effect is an artifact of same-distribution regression.","supporting_citations":[{"cited_title":"Subjective probability: A judgment of representativeness","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of representativeness heuristics as the overweighting of diagnostic attributes of a target group."},{"cited_title":"Stereotypes","cited_arxiv_id":null,"evidence_quote":"Supplies the kernel-of-truth equation and the formal likelihood-ratio model of stereotypes that the paper operationalizes."},{"cited_title":"Errors in probabilistic reasoning and judgment biases","cited_arxiv_id":null,"evidence_quote":"Supplies the cognitive-science basis for exaggeration and probabilistic judgment errors from heuristics."},{"cited_title":"Definition and assessment of accuracy in social stereotypes","cited_arxiv_id":null,"evidence_quote":"Supplies the kernel-of-truth assumption that stereotypes are grounded in empirical realities."},{"cited_title":"ANES Time Series Cumulative Data File [dataset and documentation]","cited_arxiv_id":null,"evidence_quote":"Supplies the survey data for empirical partisan positions and human prediction baselines."},{"cited_title":"Deep alignment with country shrinks the moral gap between conservatives and liberals","cited_arxiv_id":null,"evidence_quote":"Supplies the Moral Foundations Questionnaire dataset with party affiliation used for the second experiment."},{"cited_title":"A perspective on judgment and choice: Mapping bounded rationality","cited_arxiv_id":null,"evidence_quote":"Supplies the self-correction idea behind the awareness and feedback mitigation prompts."}],"review_version":1}