{"id":"a8e095ea-f01c-4477-9a04-37667a9e8ee0","arxiv_id":"2504.19445","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs consistently judge value statements and news headlines more negatively in binary response formats than in continuous rating scales when simulating human respondents.","lead":"Large language models give more negative, opposing answers when asked binary yes/no questions than when asked for 0-10 ratings, across five models and two judgment tasks. The finding matters because small changes in response format can systematically bias LLM outputs used in surveys, sentiment analysis, and automated decisions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment 1's 'negative binary bias' is a contrast with the model's own continuous responses, not with ground truth; without an external anchor, the sign of the format effect is underdetermined.","rationale":"The reader's weakest_assumption identifies exactly this issue: the model's own continuous response is used as the unbiased proxy for internal value in Experiment 1, and the raw Delta-P measure compares binary proportions to continuous responses thresholded at 0.5. My stress-test confirms that this is the most load-bearing assumption in the paper. The headline comparative finding—binary responses yield more negative judgments than continuous ones—is robust across models and tasks, and the control conditions address label-mapping artifacts. However, labeling this difference a 'negative bias' of binary formats presupposes the continuous scale is unbiased. Experiment 2 partially rescues the interpretation for sentiment, because human ratings provide an external reference and the continuous bias is not significant there, but Experiment 1 transfers this assumption without a value-domain anchor. This does not invalidate the observed format effect; it narrows what can be concluded about which format is biased. Since the reader already issued a CONDITIONAL verdict conditioned on an external anchor or defense of the continuous baseline, my analysis does not warrant a different verdict. I set verdict_should_be to UNCHANGED rather than CONDITIONAL because the reader's verdict already reflects this condition; no adjustment is needed.","tokens_in":8528,"tokens_out":4975,"duration_ms":48085,"concrete_test":"Obtain human value judgments for the same 210 statements used in Experiment 1 (from Moore et al. 2024 or a fresh survey), then re-fit the hierarchical model exactly as Experiment 2 does: use the human rating as v_{i,j} in Eq. 1 to estimate θc_i for continuous responses, and use human ratings in Eq. 3 (instead of r_{i,j}) to estimate the binary bias θbc_i. If θc_i remains near zero and θbc_i remains significantly negative, the headline claim survives; if θc_i is significantly positive, the 'negative binary bias' is actually a continuous-format positivity bias and the paper's causal framing must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that binary formats induce a negative judgment bias depends on treating the continuous response as an unbiased measurement of the internal value. In Eq. 1, r_{i,j} = βc_i v_{i,j} + θc_i + ε, and in Experiment 1 the fitted model replaces v_{i,j} with r_{i,j} (Eq. 3), i.e., it assumes θc_i = 0. Consequently the reported group-level θbc = -1.015 (95% HDI [-1.736, -0.359]) is a measure of the binary-versus-continuous difference only; it does not establish which format is biased. Experiment 2 provides partial external validation: with human sentiment ratings as v, continuous responses show no significant bias (θc = -0.111, HDI [-0.873, 0.639]) while binary responses do (θbh = -0.885, HDI [-1.735, -0.023]). But Experiment 1 has no analogous anchor. If value-judgment continuous responses carry a positivity or scale-inflation bias (e.g., 0-10 ratings shifted upward), then the observed pattern could instead be a positive bias of the continuous format, and the practical recommendation to avoid binary options would be unsupported. The paper's own Discussion acknowledges this ambiguity ('the alignment of LLMs' response biases with human judgment remains uncertain').","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether binary vs continuous response formats lead LLMs to give systematically different judgments. In two experiments (value-statement judgments and news-headline sentiment analysis), five LLMs prompted with GSS human profiles provided either binary (Yes/No, 1/0) or continuous (0-10 or 1-6) ratings. The authors report that binary responses are systematically more negative/opposing than continuous responses, and in sentiment analysis binary responses deviate significantly from human ratings in the negative direction, while continuous responses show no significant bias. Control conditions vary label wording and mapping. The primary statistical evidence comes from descriptive ΔP differences and hierarchical Bayesian regression with group-level HDI estimates.","tokens_in":8779,"tokens_out":6141,"duration_ms":58989,"significance":"The finding that response format alone can shift LLM judgments has clear practical relevance for LLM-based psychological measurement, survey simulation, and decision support. The paper's strengths are the multi-model design (three open-source, two commercial), the inclusion of label-mapping controls, and the use of human sentiment ratings as an external anchor in Experiment 2. The two-experiment structure, with the human-anchored experiment serving as partial validation, is a good design. However, the central interpretation in Experiment 1 rests on the assumption that continuous responses are an unbiased measure of the model's internal judgment, which is not externally validated. The paper is also not reproducible as submitted, lacking code, data, and full statistical specifications. These issues are correctable in a revision.","major_comments":[{"comment":"Equation (3) replaces the internal value v_{i,j} with the continuous response r_{i,j}, implicitly assuming θc_i = 0 in Eq. (1). Consequently, the group-level estimate θbc = -1.015 (95% HDI [-1.736, -0.359]) in Experiment 1 only demonstrates a discrepancy between binary and continuous response formats; it does not establish which format is biased. If continuous 0-10 ratings are inflated toward the positive/supportive end, the observed pattern could instead be a positive bias of the continuous format. Since the abstract and discussion generalize a 'negative bias' of binary formats across both tasks, the missing external anchor in Experiment 1 is a load-bearing gap. I recommend either (a) adding an external criterion for the value-judgment task (e.g., human value ratings) or (b) reframing Experiment 1's finding as a 'format discrepancy' and reserving the signed bias claim for Experiment 2.","section":"Value Judgment (Eq. 3)"},{"comment":"The Bayesian analysis is not fully specified. The paper reports using PyMC with four chains and 2500 post-burn-in samples, but does not state the prior distributions for group-level means, variances, or the regression coefficients; does not report R-hat or effective sample sizes; and does not make clear whether Eq. (1)'s continuous-response likelihood was fitted (θc is not reported for Experiment 1). Without this information, the HDI estimates are difficult to evaluate. Please provide the complete model code or a full mathematical specification including priors and convergence diagnostics.","section":"Model Fitting"},{"comment":"No code, data, prompt templates, or model outputs are included in the manuscript or referenced in an archive. Given temperature 0 and the detailed prompt descriptions, replication should be straightforward if materials were available. Their absence is a major obstacle for a quantitative empirical paper and should be addressed by releasing all artifacts.","section":"Reproducibility"},{"comment":"The descriptive ΔP measure dichotomizes continuous responses at r > 0.5. The paper does not examine the sensitivity of its descriptive conclusions to this threshold. A sensitivity analysis across reasonable thresholds (e.g., 0.4, 0.5, 0.6) would demonstrate that the ΔP results and the sign of the discrepancy are not artifacts of the chosen cutoff.","section":"Measurement of Response Bias"}],"minor_comments":[{"comment":"The sentence 'The judgment curve should be centered on 0.5 if the LLMs are unbiased' is unclear, since the plotted curves are not centered in an obvious way; please define how 'centered' is computed.","section":"Figure 1A"},{"comment":"The condition labels 'Positive: Yes', 'Positive: No', and 'Baseline' are used in the figure but not fully defined in the caption; please add a legend explaining each condition.","section":"Figure 3A/3C"},{"comment":"Footnote 3 says 'see GSS Agents for an example' but gives no URL or reference; please provide the citation or remove the dangling reference.","section":"Footnote 3"},{"comment":"The term 'response bias' is used both for the latent shift parameter θ and for the descriptive binary-vs-continuous difference ΔP; please distinguish these, e.g., use 'format discrepancy' for ΔP.","section":"Throughout"},{"comment":"The claim that 'calibrating models through a few trial runs and applying simple post hoc adjustments, such as regression-based transformations, can help align outputs with intended interpretations' is unsupported by the analyses; please remove or provide supporting evidence.","section":"Discussion"},{"comment":"Footnote 4 states that 'some models (e.g., GPT-4o) had fewer samples, with a minimum of 8 responses per model'; this is ambiguous (per item? per condition?) and should be clarified.","section":"Footnote 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the two-experiment design is thoughtful. The main concern is the interpretational gap in Experiment 1 and the lack of reproducibility artifacts. I am recommending major revision rather than rejection because the core empirical pattern is plausible and the Experiment 2 anchor provides a partial external validation. I would like the editor to ensure that the authors share data and code and clarify the model specification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical finding—across five models and two tasks, LLMs answer more negatively in binary formats than continuous ones, with control conditions ruling out simple label-mapping effects. The paper is worth a serious referee. The stress-test note is partially right: in Experiment 1 the sign of the bias is defined relative to the model's own continuous responses, so calling it a 'negative bias' assumes the continuous scale is unbiased. Experiment 2 gives some external ground: continuous sentiment ratings align with human judgment (theta_c near zero) while binary ratings do not. That partially rescues the interpretation, but value judgments still lack an anchor. The authors should either soften the language for Experiment 1 or find an external reference for value statements.\n\nWhat's genuinely new: the specific phenomenon of a consistent negative shift in binary vs continuous formats, especially the finding that it isn't explained by a preference for '1' or by Yes/No label semantics. The K/L reversal control is a nice touch. The hierarchical Bayesian model is appropriate, and the per-model variability is shown.\n\nSoft spots, in order: (1) No code or data, which for an empirical claim like this is a real problem—the results should be reproducible. (2) The continuous response threshold at 0.5 is a free parameter; it happens to be the natural cut point, but a sensitivity check would tighten the claim. (3) The role-play prompt is the only instruction style; the authors acknowledge this, but it limits generality. (4) In Experiment 1, the 'negative bias' is a relative discrepancy, and the Discussion already admits the alignment with human judgment is uncertain—that caveat should be in the abstract or results.\n\nThe citation pattern looks fine; the contrast with human acquiescence bias (humans say Yes more, LLMs say No more) is a useful observation. The paper doesn't claim more than it shows, except possibly for the 'bias' framing in value judgments.\n\nWho should read it: anyone using LLM outputs in survey design, psychological text analysis, or decision support. It won't reorganize the field, but it isolates a cheap, controllable format effect that practitioners should know about.\n\nI'd send it to review. With code and data plus a clearer separation of relative vs absolute claims, it would be a solid contribution.","headline":"A well-controlled empirical study showing LLMs judge more negatively in binary than continuous formats; the main caveat is that Experiment 1's 'negative bias' is relative to model self-reports, though Experiment 2 provides an external anchor for sentiment.","tokens_in":9289,"tokens_out":2652,"would_cite":true,"duration_ms":27272,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs reliably answer more negatively on binary scales than on continuous scales, across value judgments and sentiment analysis.","keywords":["response format bias","binary vs continuous responses","large language models","negative judgment bias","value judgment task","sentiment analysis","hierarchical Bayesian model","response style in LLMs"],"falsifier":"Take a set of value statements with human ground-truth agreement ratings. If, relative to those ratings, LLMs' continuous responses are shifted more positive than their binary responses while binary responses match humans, the central claim would be reversed. More directly: if replacing Yes/No with two neutral letters in the value task eliminates the roughly $-1.0$ logit bias, the \"negative bias\" is a label effect rather than a binary-format effect.","tokens_in":8326,"feed_emoji":"⚖️","tokens_out":4608,"duration_ms":44229,"temperature":0.7,"pith_summary":"This paper tries to establish that the response format—binary versus continuous—changes what large language models report, in a consistent direction: when asked to answer Yes/No or positive/negative, models produce more negative judgments than when asked to rate on a continuous scale. The authors show this in two tasks, value-statement agreement and news-headline sentiment, using several open and commercial models set to simulate particular human profiles. They quantify the effect with hierarchical Bayesian models and control for label wording and option order. If the claim is right, small design choices in LLM-based surveys or decision tools can systematically skew outputs, and binary formats are not neutral alternatives to continuous scales.","feed_headline":"Binary questions make LLMs answer more negatively","feed_subtitle":"Value judgments and sentiment ratings both shift toward opposition and negativity in binary formats, across five models.","key_machinery":"The measurement apparatus is a hierarchical Bayesian regression that models binary responses as a logistic function of an internal value $v$, with continuous responses used as a proxy for that value in Experiment 1 and human ratings as the proxy in Experiment 2. Equations (3) and (4) include a group-level binary-format bias $\\theta_{bc}$ or $\\theta_{bh}$, a question-type effect, and option-label preferences. The paper also uses the raw $\\Delta$-P measure—comparing binary proportions to continuous responses thresholded at 0.5—as a model-free check. This machinery lets the paper separate a genuine format bias from label mapping or an arbitrary \"1\" versus \"0\" preference.","core_discovery":"The central discovery is that LLMs' binary judgments carry a significant negative bias relative to their continuous judgments: in the value task the group-level bias is $\\theta_{bc} = -1.015$ (95% HDI $[-1.736, -0.359]$) toward opposing statements; in the sentiment task, $\\theta_{bh} = -0.885$ (95% HDI $[-1.735, -0.023]$) toward negative, plus a \"No\" preference with $\\theta_{Yes} = -1.320$ (95% HDI $[-2.160, -0.465]$). Continuous responses correlated with human sentiment ratings ($r \\geq 0.62$) and showed no significant bias, while binary responses shifted away from human judgments. The effect held when labels were 1/0 rather than Yes/No and when labels were reversed, so it is not merely a wording artifact. The paper interprets this as a format-level inconsistency rather than simple mimicry of human acquiescence, because humans tend to say Yes more often in binary formats whereas LLMs lean negative.","pith_inferences":["Editorial inference: the continuous response may itself be positively biased rather than binary being negatively biased, since Experiment 1 has no external anchor; a follow-up with human ground-truth value judgments could decide which format is skewed.","Editorial inference: the \"No\" preference in sentiment suggests a lexical association between the word No and negativity; replacing Yes/No with neutral letters in the value task would test whether the bias survives when labels carry no valence.","Editorial inference: comparing base and instruction-tuned versions of the same model would locate whether the format bias arises during pre-training or during alignment, a distinction the paper leaves open.","Editorial inference: the same binary-versus-continuous gap could affect other LLM applications such as content moderation or clinical screening, where a scale rather than a yes/no prompt could reduce false-negative negative judgments."],"forward_implications":["LLM-based psychological text analysis will under-count positive sentiment when forced into binary positive/negative labels; continuous scales or calibrated thresholds should be used instead.","Changing response format from binary to continuous can shift mean support proportions by roughly 10–15 percentage points (for example, from 74.5% to 60.7% in the value task), large enough to flip a conclusion in survey-like applications.","Designers of LLM decision-support systems should avoid binary Yes/No response formats or apply regression-based calibration, because binary answers carry a systematic negative skew.","Evaluations that mix response formats across models or conditions will confound genuine judgment with format bias; format should be held constant in comparisons."],"supporting_citations":[{"why":"Supplies the 210 value statements used in Experiment 1.","marker":"Moore et al. (2024)"},{"why":"Source of the 213 news headlines and human sentiment ratings used in Experiment 2.","marker":"Rathje et al. (2024)"},{"why":"Provides the headline set and human annotation baseline for sentiment judgments.","marker":"Robertson et al. (2023)"},{"why":"Supplies the GSS agent profiles that define simulated respondents in both experiments.","marker":"Park et al. (2024)"},{"why":"Supplies the Delta-P measurement of response bias and the field evidence that binary versus continuous formats shift human Yes answers.","marker":"Rivera-Garrido et al. (2022)"}],"fun_headline_variants":["Binary queries bias LLMs toward negative answers","Format matters: LLMs lean negative on binary scales","Binary judgments make LLMs more pessimistic","Binary responses push LLMs toward negativity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis treats the continuous response as an unbiased proxy for the model's internal judgment, so the \"negative bias of binary formats\" could instead be a positive bias of continuous formats; only Experiment 2 has human ratings to anchor the comparison.","fun_headline_variants_meta":{"raw":{"variants":["Binary queries bias LLMs toward negative answers","Format matters: LLMs lean negative on binary scales","Binary judgments make LLMs more pessimistic","Binary responses push LLMs toward negativity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000453,"raw_usage":{"total_tokens":2260,"prompt_tokens":910,"completion_tokens":1350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1295}},"tokens_in":526,"tokens_out":1350,"duration_ms":10641,"temperature":1.0,"reasoning_tokens":1295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:52:04.809660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of value statements with human ground-truth agreement ratings. If, relative to those ratings, LLMs' continuous responses are shifted more positive than their binary responses while binary responses match humans, the central claim would be reversed. More directly: if replacing Yes/No with two neutral letters in the value task eliminates the roughly $-1.0$ logit bias, the \"negative bias\" is a label effect rather than a binary-format effect.","supporting_citations":[{"cited_title":"E., & Van Bavel, J","cited_arxiv_id":null,"evidence_quote":"Source of the 213 news headlines and human sentiment ratings used in Experiment 2."},{"cited_title":"E., Pr ¨ollochs, N., Schwarzenegger, K., P¨arnamets, P., Van Bavel, J","cited_arxiv_id":null,"evidence_quote":"Provides the headline set and human annotation baseline for sentiment judgments."},{"cited_title":"P., Accerenzi, M., & Bra˜nas-Garza, P","cited_arxiv_id":null,"evidence_quote":"Supplies the Delta-P measurement of response bias and the field evidence that binary versus continuous formats shift human Yes answers."}],"review_version":1}