{"id":"75a17da8-8bdd-4c28-9c3c-3a13c835b344","arxiv_id":"2506.22232","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Questionnaire Modeling uses human survey responses as in-context examples to measure LLM political bias relative to a human population, improving paraphrase stability and exposing instruction-tuning-induced bias flips.","lead":"The authors propose Questionnaire Modeling, a way to test political bias in large language models by feeding them a human respondent's answers to earlier survey questions and asking them to predict the next answer. It makes bias measurements more stable under rephrasing and reveals cases where instruction tuning flips the direction of a model's bias.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central stability claim is undercut by in-context order sensitivity: Tables 2/3 use one fixed ordering, while the paper's own ablations show that permuting order flips bias polarity; this order variance is not reflected in the reported standard errors.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing issue: the stability claim is evaluated over paraphrases and respondents, but not over the ordering of in-context examples, and the paper's own ablations show that order changes can flip bias polarity. This directly threatens the abstract's claim that QM improves the stability of question-based bias evaluation and that it may be used to compare instruction-tuned models to their base versions. I considered alternative concerns, such as the narrow reference population of Swiss elected candidates and the top-10-token truncation in probability estimation, but those are secondary: the reference population is explicitly part of the method's definition, and truncation is a known limitation with a concrete diagnostic. The order-sensitivity issue is more central because it affects every reported bias score in Tables 2 and 3 and is acknowledged but not resolved in the paper. The authors deserve credit for running and reporting the order ablations, and for releasing code and data, which makes the proposed concrete test feasible. The right outcome remains conditional acceptance: the method is promising, but the stability claim needs to be either restricted to fixed-order settings or supported by order-marginalized estimates. Since the reader already assigned CONDITIONAL, no verdict adjustment is needed.","tokens_in":19464,"tokens_out":3847,"duration_ms":50044,"concrete_test":"For each model and each of the 7 target questions, compute the QM bias score under a random sample of at least 60 permutations of the 60 in-context examples, holding the target-question paraphrase fixed; also compute scores under orderings that keep the final in-context example fixed. Then compare the standard deviation of bias across orderings with the standard errors reported in Tables 2 and 3 and with the paraphrase variability in Table 4. If the order-induced SD exceeds the reported SE for a majority of model-question pairs, or if the instruction-tuning polarity flip on Q13.4 or Q13.5 reverses across orderings, the central stability claim must be revised to 'stable only for a fixed ordering', and the main tables should report order-averaged bias with order-based confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Questionnaire Modeling improves the stability of question-based bias evaluation. In the main experiments, however, the bias score is computed for a single fixed ordering of the 60 in-context examples: respondents are averaged over, but order is not. The reported stability in Table 4 and §3.4 is measured only across paraphrases of the target question. Appendix F.1 (Figures 7 and 8) shows that permuting the in-context order, and especially changing the identity of the last in-context example, shifts bias scores substantially and can flip their polarity for several model-question pairs. Because the order effects are model- and question-dependent, the point estimates in Tables 2 and 3 are conditional on one arbitrary prompt arrangement, and the standard errors over respondents do not capture order-induced uncertainty. In particular, the headline finding that instruction tuning flips bias on Q13.4 (stay-at-home parenting) and Q13.5 (digitalization) could be an artifact of the chosen ordering. The authors acknowledge the sensitivity in Appendix F.1 and the Limitations, but they do not marginalize over order or incorporate order variance into their main stability comparison. Since the central claim is that QM provides stable measurements suitable for comparing models and model versions, this unaddressed ordering dimension is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Questionnaire Modeling (QM), a probing task in which an LLM is given 60 human survey question-answer pairs as in-context examples and must predict a target respondent's answer to a seventh attitude statement. The authors define a personalization accuracy score and a bias score equal to the difference between the model's average predicted 'yes' probability and the human mean, then report these scores for Llama 3.1 8B/70B/405B, OLMo 7B, and GPT-3.5 across seven target questions. They claim that QM improves the stability of question-based bias evaluation relative to zero-shot prompting, that instruction tuning can flip the direction of bias on several statements, and that larger models exhibit higher personalization accuracy and smaller bias scores. Ablations vary the order and length of the in-context examples and compare true human responses with randomized answers.","tokens_in":19772,"tokens_out":4118,"duration_ms":50624,"significance":"If validated, QM is a useful contribution: it grounds LLM bias measurement in a human response distribution, it is transparently defined, and the authors release code and data. The paper's honesty is a strength: it reports a randomized-context baseline, full per-question variability tables, and an order-sensitivity ablation rather than hiding negative results. The empirical pattern that instruction tuning can flip the polarity of bias on specific attitude statements is intriguing, and the trend that larger models show lower bias in this setup is worth investigating. However, the central reliability claim is currently weakened by the fact that the main point estimates are computed under a single fixed ordering of the in-context examples while the paper's own ablation shows that ordering can change and even flip bias scores substantially.","major_comments":[{"comment":"The headline bias scores in Tables 2 and 3 are computed with one fixed ordering of the 60 in-context examples, and the reported standard errors reflect respondent sampling only (as defined in Section 3.3). The order-permutation ablation in Figures 7 and 8 shows that bias scores can shift by tens of percentage points and can flip polarity for several model-question pairs when the last in-context example is changed. Because the paper's central comparisons are differences between these point estimates (e.g., the instruction-tuning flips on Q13.4 and Q13.5, and the size trend in Table 3), the authors should either average bias scores over multiple random order permutations, or explicitly quantify and propagate order-induced uncertainty into the reported estimates and confidence intervals. As written, the reader cannot tell whether the headline flips are robust properties of the models or artifacts of one arbitrarily chosen prompt arrangement.","section":"Appendix F.1 / Tables 2-3"},{"comment":"The randomized in-context baseline achieves bias variability almost identical to true QM for many model-question pairs (for example, Llama 3.1 8B Instruct: 16.5 vs 16.2; OLMo Instruct: 11.6 vs 11.7; GPT-3.5: 21.5 vs 21.7). The text acknowledges this, but the abstract and introduction still attribute the stability improvement to QM's use of human survey data. The paper should restate the contribution more precisely: providing in-context examples yields the stability gain, while the human survey responses mainly shift the bias scores and anchor them to a population. Otherwise the reader may infer a causal role for the survey content that the randomized baseline does not support.","section":"Section 5.1 / Table 4"}],"minor_comments":[{"comment":"There are several typographical inconsistencies, including 'LLama' in Table 2 and 'counter-parts' in Section 5; these should be corrected.","section":"Throughout"},{"comment":"Only the original configuration has a confidence interval; the permuted estimates are shown without uncertainty. Adding error bars or intervals to the permuted conditions would help the reader judge which polarity flips are statistically meaningful.","section":"Appendix F.1, Figures 7-8"},{"comment":"The Limitations paragraph says the paper does not investigate varying 'other aspects of the prompt,' but the ablation in Appendix F.1 does vary the order and length of in-context examples; please cross-reference that section to make the scope clear.","section":"Section 4 / Limitations"},{"comment":"Some token labels in the probability distribution figures appear truncated (e.g., a bare 'n'), and the captions refer to color coding that may not be accessible in grayscale; please ensure the final PDF renders these correctly.","section":"Figure 3 and Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computational linguistics venue and the authors have been admirably transparent about limitations. The main risk is that the abstract promises more stability than the method currently delivers, since the order-sensitivity ablation is reported but not integrated into the main estimates. I would recommend asking for a robustness analysis over random orderings and a revision of the contribution framing before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The genuinely new thing is Questionnaire Modeling: feed the model a respondent's answers to prior questions as in-context examples, Monte Carlo over respondents, and measure bias as the gap between the model's expected 'yes' probability and the human mean. That is a real addition to the LLM-bias toolkit. The paper is also refreshingly transparent—data and code are public, the randomized in-context baseline is a sensible control, and the limitations section is unusually candid.\n\nThe headline paraphrase-stability result holds up: across 50 paraphrases per target question, QM lowers the standard deviation of the bias score relative to zero-shot prompting for most model–question pairs. Personalization accuracy rises with model size, and the observation that instruction tuning can flip bias direction on specific statements is interesting.\n\nThe soft spots are real, and the biggest one is in the paper's own appendix. The main tables use one fixed ordering of the 60 in-context examples, and the standard errors are over respondents only. But Appendix F.1 shows that permuting the order—especially changing the identity of the last in-context example—shifts bias scores substantially and can flip polarity for several model–question pairs. That means the point estimates in Tables 2 and 3 are conditional on an arbitrary ordering, and the flips attributed to instruction tuning (stay-at-home parenting, digitalization) could be artifacts. The authors acknowledge the sensitivity but do not marginalize over order or report order-induced uncertainty. For a paper whose central claim is stability, this gap is load-bearing.\n\nTwo smaller concerns. The reference population is 192 elected Swiss parliamentarians, not the Swiss public; calling that 'representative of a population P' is a stretch, and 'bias relative to the Swiss electorate' would need a voter sample. And the bias score is entangled with personalization accuracy: a model that ignores context and defaults to one answer looks biased. That is defensible as a definition, but it is not the same as political bias in the everyday sense. The correlation of −0.63 between absolute bias and PA confirms the entanglement.\n\nWho is this for? Anyone working on LLM opinion probing or model-version comparisons. It deserves a serious referee. But I would want the authors to either average over multiple orderings or explicitly bound order-induced variance, and to soften the abstract's stability claim to 'paraphrase stability.' Without that, the specific flip findings are not citable as firm results.","headline":"A genuinely new probing task with honest reporting, but the main empirical findings are conditional on one in-context ordering that the paper's own ablations show can flip the results.","tokens_in":20249,"tokens_out":4381,"would_cite":true,"duration_ms":44535,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that feeding LLMs prior survey answers as in-context examples—Questionnaire Modeling—stabilizes political bias measurement enough to compare instruction-tuned models with their base versions.","keywords":["political bias","in-context learning","questionnaire modeling","bias stability","instruction tuning","LLM evaluation","survey data","personalization accuracy"],"falsifier":"Recompute the reported bias scores with, say, twenty random permutations of the sixty in-context examples for each model and target question; if the polarity of the bias flips across permutations for a substantial share of model–question pairs—or the spread of permuted scores is comparable to the difference between base and instruct models—then the stability claim collapses.","tokens_in":19308,"feed_emoji":"🗳️","tokens_out":4347,"duration_ms":38483,"temperature":0.7,"pith_summary":"Political bias testing by directly asking a language model a question and reading off 'yes' or 'no' is unstable: paraphrase or reword the question and the answer flips. This paper proposes Questionnaire Modeling (QM), which instead shows the model a partially filled human questionnaire—the answers of one real survey respondent to sixty earlier questions—and asks it to predict that respondent's answer to the target question. Repeating this over many respondents and averaging estimates the model's bias relative to the human population. The authors report that QM reduces the variability of bias scores across fifty paraphrases of each target question, and that it exposes cases where instruction tuning flips the direction of bias, such as overestimating agreement to stay-at-home parenting before tuning and underestimating it after. They also find that larger instruction-tuned models predict respondents' answers more accurately and show smaller biases.","feed_headline":"Questionnaire in-context prompts make LLM bias tests more stable","feed_subtitle":"New method compares instruction-tuned models with base versions; tuning can flip which side a model leans.","key_machinery":"The central object is the Questionnaire Modeling task: a conversation in which a user asks a sequence of questions from a real voting advice questionnaire and the assistant answers, with the model asked to predict the final answer of the same human respondent. The bias score is the mean predicted probability of 'yes' across respondents minus the mean observed human probability of 'yes'. The personalization accuracy measures how often the model predicts the respondent's actual answer. Monte Carlo sampling over the 192 respondents turns a single brittle prompt into an expectation over many prompts, which is what carries the stability improvement.","core_discovery":"The central claim is that bias evaluation becomes more reliable when the probing question is embedded in a questionnaire context drawn from real human responses. Formally, the paper defines the bias score as the difference between the model's average predicted probability of the answer 'yes' to a target question and the observed mean human answer; the prediction is made per respondent, conditioned on that respondent's answers to the other sixty questions. Averaging over the 192 elected Swiss parliamentarian respondents marginalizes over the influence of any single in-context example. The paper demonstrates that this procedure yields lower standard deviations of bias across paraphrases than zero-shot prompting, and that comparing base and instruction-tuned versions of the same model becomes meaningful because the method does not require the model to follow an instruction. The empirical finding that instruction tuning can change the direction of bias—for example, on state security, stay-at-home parenting, and digitalization—is presented as evidence that the method can detect real behavioral differences between model versions.","pith_inferences":["If the stability holds across a broader set of questions and languages, QM could serve as a standardized audit for political bias per model release, using the same survey population as a fixed reference.","The ablation's sensitivity to the identity of the last in-context example suggests a recency effect; sampling the last example randomly alongside respondent sampling could turn this sensitivity into a measured uncertainty rather than an uncontrolled confound.","The bias scores are defined relative to a specific human population (elected Swiss parliamentarians), so the same model could appear differently biased against another population; using QM as a benchmark would require fixing the reference population.","Extending QM beyond binary answers to the original Likert scale or to multi-token free responses would test whether the stability benefit persists when the discrete yes/no restriction is removed."],"forward_implications":["QM reduces the standard deviation of bias scores across fifty paraphrases compared with zero-shot prompting for the tested models.","Instruction tuning can flip the polarity of bias scores—e.g., Llama 3.1 70B switches from over- to under-estimating agreement with stay-at-home parenting.","Larger instruction-tuned Llama models show higher personalization accuracy and lower absolute bias scores, with 405B near zero bias on five of seven questions.","Randomly assigned in-context answers also reduce paraphrase variability but produce different bias scores, indicating that much of the stability comes from learning the label space rather than personalized prediction.","QM separates instructability from bias, allowing base models to be compared with their instruction-tuned counterparts even when the base model cannot follow the 'answer yes or no' instruction."],"supporting_citations":[{"why":"Supplies the in-context learning mechanism that QM builds on.","marker":"(Brown et al., 2020)"},{"why":"Provides the zero-shot political bias probing baseline and the 'Your response:' prompt format used for comparison.","marker":"(Feng et al., 2023)"},{"why":"Argues questionnaire-based probing is artificial and can be unstable, motivating the contextual QM approach.","marker":"(Röttger et al., 2024)"},{"why":"Explains that in-context examples convey the label space, supporting the random-context baseline finding.","marker":"(Min et al., 2022)"},{"why":"Shows that reordering in-context examples changes model outputs, motivating the ablation studies on example order.","marker":"(Lu et al., 2022)"},{"why":"Establishes label bias and its mitigation with in-context examples, framing the stability analysis.","marker":"(Reif and Schwartz, 2024)"},{"why":"Documents how statement length and sentiment affect responses, motivating the paraphrase-variability evaluation.","marker":"(Haller et al., 2024)"}],"fun_headline_variants":["Human survey context steadies LLM bias tests","In-context human answers stabilize LLM bias probes","Questionnaire modeling pins down LLM bias scores","Instruction tuning flips bias direction in LLM probes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main load-bearing assumption is that one fixed ordering of the sixty in-context examples gives a representative bias estimate; the paper's own ablations show that permuting the order, and especially changing the final example, shifts scores and can flip their polarity.","fun_headline_variants_meta":{"raw":{"variants":["Human survey context steadies LLM bias tests","In-context human answers stabilize LLM bias probes","Questionnaire modeling pins down LLM bias scores","Instruction tuning flips bias direction in LLM probes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1346,"prompt_tokens":868,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":484,"tokens_out":478,"duration_ms":5238,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:08:12.344499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the reported bias scores with, say, twenty random permutations of the sixty in-context examples for each model and target question; if the polarity of the bias flips across permutations for a substantial share of model–question pairs—or the spread of permuted scores is comparable to the difference between base and instruct models—then the stability claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents how statement length and sentiment affect responses, motivating the paraphrase-variability evaluation."}],"review_version":1}