Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks

Chunhui Zhang; Wei Wang; Yi-Long Lu

REVIEW 4 major objections 6 minor 23 references

Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs reliably answer more negatively on binary scales than on continuous scales, across value judgments and sentiment analysis.

desk verdict A well-controlled empirical study showing LLMs judge more negatively in binary than continuous formats; the main caveat is that Experiment 1's 'negative bias' is relative to model self-reports, though Experiment 2 provides an external anchor for sentiment. read the letter →

arxiv 2504.19445 v1 pith:XWP3ZQ7P submitted 2025-04-28 cs.CL

Yi-Long Lu , Chunhui Zhang , Wei Wang This is my paper · ORCID

classification cs.CL

keywords responseformatbiasbinaryvscontinuousresponseslargelanguagemodelsnegativejudgmentvaluetasksentimentanalysishierarchicalBayesianmodelstyleinLLMs

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the response format—binary versus continuous—changes what large language models report, in a consistent direction: when asked to answer Yes/No or positive/negative, models produce more negative judgments than when asked to rate on a continuous scale. The authors show this in two tasks, value-statement agreement and news-headline sentiment, using several open and commercial models set to simulate particular human profiles. They quantify the effect with hierarchical Bayesian models and control for label wording and option order. If the claim is right, small design choices in LLM-based surveys or decision tools can systematically skew outputs, and binary formats are not neutral alternatives to continuous scales.

What carries the argument

The measurement apparatus is a hierarchical Bayesian regression that models binary responses as a logistic function of an internal value $v$, with continuous responses used as a proxy for that value in Experiment 1 and human ratings as the proxy in Experiment 2. Equations (3) and (4) include a group-level binary-format bias $\theta_{bc}$ or $\theta_{bh}$, a question-type effect, and option-label preferences. The paper also uses the raw $\Delta$-P measure—comparing binary proportions to continuous responses thresholded at 0.5—as a model-free check. This machinery lets the paper separate a genuine format bias from label mapping or an arbitrary "1" versus "0" preference.

What would settle it

Take a set of value statements with human ground-truth agreement ratings. If, relative to those ratings, LLMs' continuous responses are shifted more positive than their binary responses while binary responses match humans, the central claim would be reversed. More directly: if replacing Yes/No with two neutral letters in the value task eliminates the roughly $-1.0$ logit bias, the "negative bias" is a label effect rather than a binary-format effect.

Watch

Extended reading notes

Core claim

The central discovery is that LLMs' binary judgments carry a significant negative bias relative to their continuous judgments: in the value task the group-level bias is $\theta_{bc} = -1.015$ (95% HDI $[-1.736, -0.359]$) toward opposing statements; in the sentiment task, $\theta_{bh} = -0.885$ (95% HDI $[-1.735, -0.023]$) toward negative, plus a "No" preference with $\theta_{Yes} = -1.320$ (95% HDI $[-2.160, -0.465]$). Continuous responses correlated with human sentiment ratings ($r \geq 0.62$) and showed no significant bias, while binary responses shifted away from human judgments. The effect held when labels were 1/0 rather than Yes/No and when labels were reversed, so it is not merely a wording artifact. The paper interprets this as a format-level inconsistency rather than simple mimicry of human acquiescence, because humans tend to say Yes more often in binary formats whereas LLMs lean negative.

Load-bearing premise

The analysis treats the continuous response as an unbiased proxy for the model's internal judgment, so the "negative bias of binary formats" could instead be a positive bias of continuous formats; only Experiment 2 has human ratings to anchor the comparison.

Editorial extensions

If this is right

LLM-based psychological text analysis will under-count positive sentiment when forced into binary positive/negative labels; continuous scales or calibrated thresholds should be used instead.
Changing response format from binary to continuous can shift mean support proportions by roughly 10–15 percentage points (for example, from 74.5% to 60.7% in the value task), large enough to flip a conclusion in survey-like applications.
Designers of LLM decision-support systems should avoid binary Yes/No response formats or apply regression-based calibration, because binary answers carry a systematic negative skew.
Evaluations that mix response formats across models or conditions will confound genuine judgment with format bias; format should be held constant in comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the continuous response may itself be positively biased rather than binary being negatively biased, since Experiment 1 has no external anchor; a follow-up with human ground-truth value judgments could decide which format is skewed.
Editorial inference: the "No" preference in sentiment suggests a lexical association between the word No and negativity; replacing Yes/No with neutral letters in the value task would test whether the bias survives when labels carry no valence.
Editorial inference: comparing base and instruction-tuned versions of the same model would locate whether the format bias arises during pre-training or during alignment, a distinction the paper leaves open.
Editorial inference: the same binary-versus-continuous gap could affect other LLM applications such as content moderation or clinical screening, where a scale rather than a yes/no prompt could reduce false-negative negative judgments.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid empirical finding—across five models and two tasks, LLMs answer more negatively in binary formats than continuous ones, with control conditions ruling out simple label-mapping effects. The paper is worth a serious referee. The stress-test note is partially right: in Experiment 1 the sign of the bias is defined relative to the model's own continuous responses, so calling it a 'negative bias' assumes the continuous scale is unbiased. Experiment 2 gives some external ground: continuous sentiment ratings align with human judgment (theta_c near zero) while binary ratings do not. That partially rescues the interpretation, but value judgments still lack an anchor. The authors should either soften the language for Experiment 1 or find an external reference for value statements.

What's genuinely new: the specific phenomenon of a consistent negative shift in binary vs continuous formats, especially the finding that it isn't explained by a preference for '1' or by Yes/No label semantics. The K/L reversal control is a nice touch. The hierarchical Bayesian model is appropriate, and the per-model variability is shown.

Soft spots, in order: (1) No code or data, which for an empirical claim like this is a real problem—the results should be reproducible. (2) The continuous response threshold at 0.5 is a free parameter; it happens to be the natural cut point, but a sensitivity check would tighten the claim. (3) The role-play prompt is the only instruction style; the authors acknowledge this, but it limits generality. (4) In Experiment 1, the 'negative bias' is a relative discrepancy, and the Discussion already admits the alignment with human judgment is uncertain—that caveat should be in the abstract or results.

The citation pattern looks fine; the contrast with human acquiescence bias (humans say Yes more, LLMs say No more) is a useful observation. The paper doesn't claim more than it shows, except possibly for the 'bias' framing in value judgments.

Who should read it: anyone using LLM outputs in survey design, psychological text analysis, or decision support. It won't reorganize the field, but it isolates a cheap, controllable format effect that practitioners should know about.

I'd send it to review. With code and data plus a clearer separation of relative vs absolute claims, it would be a solid contribution.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether binary vs continuous response formats lead LLMs to give systematically different judgments. In two experiments (value-statement judgments and news-headline sentiment analysis), five LLMs prompted with GSS human profiles provided either binary (Yes/No, 1/0) or continuous (0-10 or 1-6) ratings. The authors report that binary responses are systematically more negative/opposing than continuous responses, and in sentiment analysis binary responses deviate significantly from human ratings in the negative direction, while continuous responses show no significant bias. Control conditions vary label wording and mapping. The primary statistical evidence comes from descriptive ΔP differences and hierarchical Bayesian regression with group-level HDI estimates.

Significance. The finding that response format alone can shift LLM judgments has clear practical relevance for LLM-based psychological measurement, survey simulation, and decision support. The paper's strengths are the multi-model design (three open-source, two commercial), the inclusion of label-mapping controls, and the use of human sentiment ratings as an external anchor in Experiment 2. The two-experiment structure, with the human-anchored experiment serving as partial validation, is a good design. However, the central interpretation in Experiment 1 rests on the assumption that continuous responses are an unbiased measure of the model's internal judgment, which is not externally validated. The paper is also not reproducible as submitted, lacking code, data, and full statistical specifications. These issues are correctable in a revision.

major comments (4)

[Value Judgment (Eq. 3)] Equation (3) replaces the internal value v_{i,j} with the continuous response r_{i,j}, implicitly assuming θc_i = 0 in Eq. (1). Consequently, the group-level estimate θbc = -1.015 (95% HDI [-1.736, -0.359]) in Experiment 1 only demonstrates a discrepancy between binary and continuous response formats; it does not establish which format is biased. If continuous 0-10 ratings are inflated toward the positive/supportive end, the observed pattern could instead be a positive bias of the continuous format. Since the abstract and discussion generalize a 'negative bias' of binary formats across both tasks, the missing external anchor in Experiment 1 is a load-bearing gap. I recommend either (a) adding an external criterion for the value-judgment task (e.g., human value ratings) or (b) reframing Experiment 1's finding as a 'format discrepancy' and reserving the signed bias claim for Experiment 2.
[Model Fitting] The Bayesian analysis is not fully specified. The paper reports using PyMC with four chains and 2500 post-burn-in samples, but does not state the prior distributions for group-level means, variances, or the regression coefficients; does not report R-hat or effective sample sizes; and does not make clear whether Eq. (1)'s continuous-response likelihood was fitted (θc is not reported for Experiment 1). Without this information, the HDI estimates are difficult to evaluate. Please provide the complete model code or a full mathematical specification including priors and convergence diagnostics.
[Reproducibility] No code, data, prompt templates, or model outputs are included in the manuscript or referenced in an archive. Given temperature 0 and the detailed prompt descriptions, replication should be straightforward if materials were available. Their absence is a major obstacle for a quantitative empirical paper and should be addressed by releasing all artifacts.
[Measurement of Response Bias] The descriptive ΔP measure dichotomizes continuous responses at r > 0.5. The paper does not examine the sensitivity of its descriptive conclusions to this threshold. A sensitivity analysis across reasonable thresholds (e.g., 0.4, 0.5, 0.6) would demonstrate that the ΔP results and the sign of the discrepancy are not artifacts of the chosen cutoff.

minor comments (6)

[Figure 1A] The sentence 'The judgment curve should be centered on 0.5 if the LLMs are unbiased' is unclear, since the plotted curves are not centered in an obvious way; please define how 'centered' is computed.
[Figure 3A/3C] The condition labels 'Positive: Yes', 'Positive: No', and 'Baseline' are used in the figure but not fully defined in the caption; please add a legend explaining each condition.
[Footnote 3] Footnote 3 says 'see GSS Agents for an example' but gives no URL or reference; please provide the citation or remove the dangling reference.
[Throughout] The term 'response bias' is used both for the latent shift parameter θ and for the descriptive binary-vs-continuous difference ΔP; please distinguish these, e.g., use 'format discrepancy' for ΔP.
[Discussion] The claim that 'calibrating models through a few trial runs and applying simple post hoc adjustments, such as regression-based transformations, can help align outputs with intended interpretations' is unsupported by the analyses; please remove or provide supporting evidence.
[Footnote 4] Footnote 4 states that 'some models (e.g., GPT-4o) had fewer samples, with a minimum of 8 responses per model'; this is ambiguous (per item? per condition?) and should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported bias is a measured format contrast, not a fitted input renamed as a prediction.

full rationale

The paper's central claim is an empirical comparison of binary versus continuous LLM responses. The raw Delta-P measure is computed by subtracting binary response proportions from thresholded continuous responses, and the hierarchical Bayesian intercept theta_bc is estimated from the same observed data. The Abstract's wording, 'compared to continuous ones,' accurately describes this contrast, and the paper does not claim to derive the bias from an external first-principles input. Equation 3 substitutes the model's own continuous response r_{i,j} for the latent internal value v_{i,j}; this is an explicit modeling assumption stated in Eq. 1 ('we simply assumed that') and in the text ('we used the continuous responses r_{i,j} to replace the internal value v_{i,j}'). That assumption means the sign of the format effect is anchored to the model's continuous responses rather than to ground truth, but this is a measurement-validity limitation, not circular reasoning: no parameter is fitted to a subset and then used to predict that same subset, and no load-bearing conclusion is justified by self-citation or by an author-imported uniqueness theorem. Experiment 2 adds an external human anchor and shows a similar negative binary bias, and the Discussion explicitly acknowledges residual uncertainty about alignment with human judgment. The derivation chain is therefore self-contained as a measurement study, and no circular step is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical measurement, so the ledger is light. The main analysis assumes continuous responses are a faithful proxy for the model's internal judgment (Eq. 3), assumes a logistic mapping for binary choices, and relies on the role-play prompt to define LLM judgment. The only hand-chosen constant is the 0.5 threshold used to dichotomize continuous responses for the descriptive Delta-P measure. No invented entities.

free parameters (1)

Dichotomization threshold = 0.5
Continuous normalized responses above 0.5 were classified as Support/Positive for the raw Delta-P measure; this hand-chosen threshold affects the descriptive results but not the logistic regression.

assumptions (4)

domain assumption The LLM's continuous response is a faithful proxy for its internal judgment value (Eq. 3 replaces v_{i,j} with r_{i,j}).
The entire bias measurement in Eq. 3 uses the same model's continuous output as the predictor for binary choices, so the estimated intercept theta_bc is relative to this proxy.
standard math Binary responses relate to the latent value through a logistic function (Eqs. 3 and 4).
Standard logistic link assumption for binary choice modeling, common in psychometrics.
domain assumption LLMs' responses under the simulate-a-human-profile prompt are representative of their behavior in real decision tasks.
The paper's claim is about LLM judgments generally, but all trials use an explicit role-play prompt; the authors note this limitation in the Discussion.
domain assumption Human annotations from Rathje et al. (2024) are a valid reference for sentiment ground truth.
Experiment 2 uses averaged human ratings as the benchmark for comparing LLM bias; this is inherited from prior work without re-validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks." pith.science (2026). https://pith.science/paper/XWP3ZQ7P

@misc{pith2026250419445,
  author       = {Pith},
  title        = {Pith review of: Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWP3ZQ7P}},
  note         = {Machine review of arXiv:2504.19445}
}

read the original abstract

Large Language Models (LLMs) are increasingly used in tasks such as psychological text analysis and decision-making in automated workflows. However, their reliability remains a concern due to potential biases inherited from their training process. In this study, we examine how different response format: binary versus continuous, may systematically influence LLMs' judgments. In a value statement judgments task and a text sentiment analysis task, we prompted LLMs to simulate human responses and tested both formats across several models, including both open-source and commercial models. Our findings revealed a consistent negative bias: LLMs were more likely to deliver "negative" judgments in binary formats compared to continuous ones. Control experiments further revealed that this pattern holds across both tasks. Our results highlight the importance of considering response format when applying LLMs to decision tasks, as small changes in task design can introduce systematic biases.

Figures

Figures reproduced from arXiv: 2504.19445 by the authors.

**Figure 2.** Fitted response bias for LLMs. All models show a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Comparison of LLM’s responses under different conditions and human judgments in the sentiment analysis tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Fitted response bias in sentiment analysis. (A) All [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 11 canonical work pages

[1]

Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Choi, B. C. K., & Pak, A. W. P. (2005). A catalog of biases in questionnaires. Preventing Chronic Disease, 2(1), A13

work page 2005
[3]

Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., . . . Pan, Z. (2024). DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437

arXiv 2024
[4]

M., Liu, Y ., Alessa, A., McAuley, J., & He, Z

Echterhoff, J. M., Liu, Y ., Alessa, A., McAuley, J., & He, Z. (2024). Cognitive Bias in Decision-Making with LLMs. In Y . Al-Onaizan, M. Bansal, & Y .-N. Chen (Eds.),Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 12640–12653). Association for Computational Linguistics

work page 2024
[5]

Eigner, E., & H ¨andler, T. (2024). Determinants of llm-assisted decision-making. arXiv preprint arXiv:2402.17385

arXiv 2024
[6]

Hinsvark, A., . . . Ma, Z. (2024). The Llama 3 Herd of Mod- els. arXiv preprint arXiv:2407.21783

arXiv 2024
[7]

Hinz, A., Michalski, D., Schwarz, R., & Herzberg, P. Y . (2007). The acquiescence effect in responding to a ques- tionnaire. GMS Psycho-Social Medicine, 4, Doc07

work page 2007
[8]

Hu, T., Kyrychenko, Y ., Rathje, S., Collier, N., van der Lin- den, S., & Roozenbeek, J. (2024). Generative language models exhibit social identity biases. Nature Computa- tional Science, 1–11

work page 2024

Show all 23 references

[9]

Kuru, O., & Pasek, J. (2016). Improving social media mea- surement in surveys: Avoiding acquiescence bias in face- book research. Computers in Human Behavior, 57, 82–92

2016
[10]

D., Ebner, S., Kumar, N., Reddy, V ., Koncel-Kedziorski, R., & Tanner, C

Lovering, C., Krumdick, M., Lai, V . D., Ebner, S., Kumar, N., Reddy, V ., Koncel-Kedziorski, R., & Tanner, C. (2024). Language model probabilities are not calibrated in numeric contexts. arXiv preprint arXiv:2410.16007

2024 arXiv
[11]

T., Yao, S., Friedman, D., Hardy, M

McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Grif- fiths, T. L. (2024). Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41), e2322420121

2024
[12]

Meister, N., Guestrin, C., & Hashimoto, T. (2024). Bench- marking distributional alignment of large language models. arXiv preprint arXiv:2411.05403

2024 arXiv
[13]

Moore, J., Deshpande, T., & Yang, D. (2024). Are Large Language Models Consistent over Value-laden Questions? arXiv preprint arXiv:2407.02996

2024 arXiv
[14]

R., Willer, R., Liang, P., & Bernstein, M

Morris, M. R., Willer, R., Liang, P., & Bernstein, M. S. (2024). Generative Agent Simulations of 1,000 People. arXiv preprint arXiv:2411.10109

2024 arXiv
[15]

Qiu, Z. (2025). Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115

2025 arXiv
[16]

E., & Van Bavel, J

Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robert- son, C. E., & Van Bavel, J. J. (2024). GPT is an effec- tive tool for multilingual psychological text analysis. Pro- ceedings of the National Academy of Sciences , 121(34), e2308950121

2024
[17]

P., Accerenzi, M., & Bra˜nas-Garza, P

Rivera-Garrido, N., Ramos-Sosa, M. P., Accerenzi, M., & Bra˜nas-Garza, P. (2022). Continuous and binary sets of re- sponses differ in the field. Scientific Reports, 12(1), 14376

2022
[18]

E., Pr ¨ollochs, N., Schwarzenegger, K., P¨arnamets, P., Van Bavel, J

Robertson, C. E., Pr ¨ollochs, N., Schwarzenegger, K., P¨arnamets, P., Van Bavel, J. J., & Feuerriegel, S. (2023). Negativity drives online news consumption.Nature Human Behaviour, 7(5), 812–822

2023
[19]

Manzi, G., Graziano, M. S. A., & Becchio, C. (2024). Test- ing theory of mind in large language models and humans. Nature Human Behaviour, 8(7), 1285–1295

2024
[20]

Sumers, T., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023). Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427

2023 arXiv
[21]

Liu, Q., Liu, T., & Sui, Z. (2023). Large Language Models are not Fair Evaluators. arXiv preprint arXiv:2305.17926

2023 arXiv
[22]

R., & Brown, A

Wetzel, E., B¨ohnke, J. R., & Brown, A. (2016). Response bi- ases. Oxford University Press

2016
[23]

Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2024). Large Language Models Are Not Robust Multiple Choice Selectors. arXiv preprint arXiv:2309.03882

2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Choi, B. C. K., & Pak, A. W. P. (2005). A catalog of biases in questionnaires. Preventing Chronic Disease, 2(1), A13

work page 2005

[3] [3]

Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., . . . Pan, Z. (2024). DeepSeek-V3 Technical Report. arXiv preprint arXiv:2412.19437

arXiv 2024

[4] [4]

M., Liu, Y ., Alessa, A., McAuley, J., & He, Z

Echterhoff, J. M., Liu, Y ., Alessa, A., McAuley, J., & He, Z. (2024). Cognitive Bias in Decision-Making with LLMs. In Y . Al-Onaizan, M. Bansal, & Y .-N. Chen (Eds.),Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 12640–12653). Association for Computational Linguistics

work page 2024

[5] [5]

Eigner, E., & H ¨andler, T. (2024). Determinants of llm-assisted decision-making. arXiv preprint arXiv:2402.17385

arXiv 2024

[6] [6]

Hinsvark, A., . . . Ma, Z. (2024). The Llama 3 Herd of Mod- els. arXiv preprint arXiv:2407.21783

arXiv 2024

[7] [7]

Hinz, A., Michalski, D., Schwarz, R., & Herzberg, P. Y . (2007). The acquiescence effect in responding to a ques- tionnaire. GMS Psycho-Social Medicine, 4, Doc07

work page 2007

[8] [8]

Hu, T., Kyrychenko, Y ., Rathje, S., Collier, N., van der Lin- den, S., & Roozenbeek, J. (2024). Generative language models exhibit social identity biases. Nature Computa- tional Science, 1–11

work page 2024