{"id":"79fec4e3-1904-4861-bd8c-dbdfec81ea06","arxiv_id":"2509.08480","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs in this study did not show human-like acquiescence bias; they showed a tendency to answer 'no' in English yes/no rephrasings, though the evidence for a bias independent of question meaning is weak.","lead":"A study of five AI chatbots across legal tasks in three languages found that changing a question into a yes/no format changed their answers, often increasing 'no' responses. The authors say this shows a general 'no bias,' but the evidence only supports a narrower claim about question format sensitivity.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no'-bias claim rests on the disagree-condition rows of Table 1, where 'No' (semantically option A, per the paper's own §3.2) is benchmarked against neutral B; against the correct A baseline, 'No' falls 31-85% below the neutral A rate for all five English models.","rationale":"The paper's central claim ('regardless of whether it indicates agreement or disagreement') is the part that goes beyond the data. I traced the evidence for that 'regardless' clause: it rests entirely on the disagree-condition rows of Table 1, where the paper reports increases in 'No' responses of +190%, +6%, +292%, -83%, +17% across the five English models. Those percentages compare the disagree-condition 'No' count with the neutral condition's B count. But the paper's own §3.2 says 'no' in the disagree prompt implies agreement with the question and is equivalent to option A, and Table 1 itself labels the column 'No (A)'. Comparing 'No (A)' to neutral B conflates the token 'No' with the option it negates, and the semantic role of 'No' is precisely what the 'regardless' claim is about. Using neutral A as the baseline - the only matched comparison - gives decreases of 31-85% for every model, the opposite of a uniform 'no' bias. The reader's weakest-assumption analysis identifies this exact flaw, and I agree with it. Two supporting points: even on the paper's own baseline, Llama-3.3-70B contradicts the claimed consistency (-83%), and the metric switch in §4 from positive-rate to raw 'no' counts, while disclosed, creates the uniform pattern only by anchoring the disagree condition to the wrong baseline. Credit where due: the study is otherwise carefully constructed (five models, three languages, released code and responses, disclosed limitations including the English-only pattern and legal-domain data), and the narrower finding - that converting A/B tasks to positive yes/no questions shifts English responses toward 'No' when 'No' is option B, with 'don't you agree' phrasing attenuating the effect - is consistent across models and worth reporting separately. The overgeneralization to 'regardless of whether it indicates agreement or disagreement' is unsupported, so the reader's REJECT verdict stands; the rejection is of the central claim as stated, not of the underlying experiment.","tokens_in":18877,"tokens_out":14698,"duration_ms":111868,"concrete_test":"Recompute the English disagree-condition rows of Table 1 (and the per-task tables in Appendix D) with the semantically matched baseline: compare each model's disagree-condition 'No (A)' count against its neutral-condition 'A' count instead of neutral 'B'. Run McNemar's test on the paired neutral-vs-disagree tables for each model, mirroring the paper's own significance analysis. If, as the published counts indicate, the disagree-condition 'No' rate is significantly below the neutral 'A' rate for all five models, the 'regardless of whether it indicates agreement or disagreement' claim is falsified, and the reported conclusion must be restricted to the yesno/agree conditions where 'No' equals option B.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim - that LLMs are biased toward answering 'no' regardless of whether it indicates agreement or disagreement - rests on the disagree-condition rows of Table 1. The paper's own design section (§3.2) states that in the 'Do you disagree...' prompt, 'no', due to the double negation, implies agreement with the question and therefore option A; Table 1 labels the row 'No (A)'. Yet the relative-change percentages in that row are computed against the neutral condition's B count (e.g., Llama-3.1-8B English: 1390 'No' vs 479 neutral B = +190%). Since 'No' here is semantically the same answer as neutral A, the correct baseline is A. Recomputing for English: Llama-3.1-8B 1390 vs 2915 (-52%), Mistral 734 vs 2704 (-73%), Gemma 1605 vs 2985 (-46%), Llama-3.3-70B 269 vs 1768 (-85%), GPT-4o 1476 vs 2129 (-31%). Every model gives fewer agreement-equivalent 'No' responses in the disagree condition than 'A' responses in the neutral condition; the apparent uniformity in Table 1 is an artifact of comparing against the wrong anchor. Note also that even on the paper's own B-baseline, Llama-3.3-70B shows -83%, contradicting the claimed consistent pattern. The post-hoc switch (§4) from the original positive-rate metric to raw 'no' counts is disclosed, but it only yields a coherent 'no'-bias story if the baseline matches the semantics in each condition - and it does not in the disagree condition. Without the disagree-condition evidence, the claim collapses to the narrower, better-supported finding that yes/no reformatting of A/B items shifts English responses toward 'No' when 'No' corresponds to option B; that does not support 'regardless of whether it indicates agreement or disagreement', nor the conclusion that LLMs cannot simulate human survey responses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether large language models (LLMs) reproduce human acquiescence bias, the tendency to agree with survey statements regardless of content. It evaluates five models (Llama-3.1-8B, Mistral-Small-24B, Gemma-2-27B, Llama-3.3-70B, GPT-4o) on nine legal-domain tasks in English, German, and Polish, under five prompt conditions: neutral A/B choice, yes/no, agreement, negated agreement, and disagreement. The authors report that prompt phrasing significantly changes responses, that no consistent human-like acquiescence bias is found, and that English models display a bias toward the response 'no' regardless of whether that response indicates agreement or disagreement, leading them to conclude that LLMs are not well suited to simulate human survey responses.","tokens_in":19164,"tokens_out":9303,"duration_ms":81859,"significance":"If the central claim were supported, the paper would contribute a large-scale, multilingual negative result of interest to prompt design and to the use of LLMs as survey simulators. The study has notable strengths: more than 37,975 question variations and 152,000 responses, five models in three languages, publicly available code and data, and per-task statistical tests. However, the headline claim rests on a baseline comparison that is semantically invalid in the disagreement condition; after correcting that comparison, the evidence contradicts the 'regardless' formulation. The paper's remaining contributions—prompt sensitivity, absence of acquiescence bias, and a narrower response-mode bias in certain English conditions—are defensible but require substantial reanalysis and reframing.","major_comments":[{"comment":"The disagree-condition rows in Table 1 benchmark 'No' responses against the neutral condition's B count, but Section 3.2 states that in the disagreement prompt 'no' implies agreement with the question and is therefore equivalent to option A. The correct baseline is neutral A, not B. Recomputing the English relative changes with neutral A as baseline yields -52% (Llama-3.1-8B), -73% (Mistral-24B), -46% (Gemma-2-27B), -85% (Llama-3.3-70B), and -31% (GPT-4o), instead of the +190%, +6%, +292%, -83%, and +17% reported in the table. Every model produces fewer agreement-equivalent 'No' responses in the disagree condition than 'A' responses in the neutral condition. The disagree-condition evidence therefore does not support the abstract and conclusion claim that LLMs display a bias toward 'no' regardless of whether it indicates agreement or disagreement; in fact, it points in the opposite direction.","section":"Section 4 / Table 1; Section 3.2"},{"comment":"The paper discloses a post hoc switch from the pre-specified positive-rate metric to the absolute number of 'no' responses ('Instead of focusing on positives, we decided to focus...'). Because the new metric is introduced after observing the results and is then applied without condition-specific semantic anchors, the central 'no'-bias claim inherits the baseline mismatch described above. The pre-specified positive-rate results (Appendix D) should be reported as the primary analysis, and any 'no'-count analysis should use the semantically equivalent anchor in each condition: neutral B for the yes/no, agreement, and (with appropriate caveats) negated-agreement conditions, and neutral A for the disagreement condition.","section":"Section 4, metric switch"},{"comment":"Even under the paper's own B-baseline, the English disagree row for Llama-3.3-70B is -83% (vs neutral B), and the paper acknowledges 'all but one model,' so the asserted consistency is already weak. Once the semantically correct A-baseline is used, none of the five models shows an increase in agreement-equivalent responses in the disagree condition. The 'regardless of whether it indicates agreement or disagreement' conclusion thus collapses, and the paper's central contribution would need to be narrowed to a response-mode bias in the conditions where 'no' corresponds to the B option in English.","section":"Section 4 / Table 1, claim of consistency"}],"minor_comments":[{"comment":"The corpus name is misspelled as 'Legelbench' in Table 3 and in Section 3.1; it should be 'Legalbench.'","section":"Table 3 and Section 3.1"},{"comment":"The phrase 'pă0.05' appears to be a rendering error; it should read 'p < 0.05.'","section":"Section 4"},{"comment":"There is a typo, 'Lllama-3.3-70B,' in the paragraph discussing Table 2.","section":"Section 4"},{"comment":"Several JSON keys contain spaces (e.g., 'n e g a t e d _ a g r e e _ p r o m p t' and 'd e f i n i t i o n _ c l a s s i f i c a t i o n'); if these are not formatting artifacts, they should be corrected to valid identifiers.","section":"Appendix C"},{"comment":"The key 'neutral_anwers' is a typo for 'neutral_answers'.","section":"Appendix C"},{"comment":"The example in the Neutral condition renders 'V oid' with a space; this should be 'Void.'","section":"Section 3.2"},{"comment":"The abstract says the bias is observed 'across different models, tasks, and languages,' but the 'no' bias is only claimed for English; the abstract should align with the English-only finding reported in the results and conclusion.","section":"Abstract / Introduction / Conclusion"},{"comment":"The relative-change percentages in the German and Polish disagree rows use the same mismatched baseline (neutral B) and should be recalculated or relabeled, even though the paper does not claim a consistent pattern in those languages.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The self-citation to the AGB-DE corpus is used only as a data source and does not create circularity. The decisive issue is the baseline mismatch in Table 1 and Section 4, which invalidates the 'regardless' formulation of the central claim. I believe this is fixable through reanalysis and a substantial reframing of the conclusions, so I recommend major revision rather than outright rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper has a load-bearing error in Table 1, and the stress-test note is right. The disagreement prompt \"Do you disagree...\" invites \"No\" to mean \"I agree with the statement\" — the paper's own §3.2 says this, and Table 1 even labels the row \"No (A)\". But the relative-change percentages are computed against neutral B. Recompute against neutral A, and every English model gives fewer agreement-equivalent \"no\" responses in the disagree condition than plain A responses in the neutral condition, roughly -31% to -85%. So the abstract's \"regardless of whether it indicates agreement or disagreement\" is not supported by the reported analysis.\n\nWhat's genuinely good: the study is large (37,975+ question variations, five models, three languages), the code and raw responses are on GitHub under MIT, and the authors are transparent about the post-hoc switch from the originally planned positive-rate metric to raw \"no\" counts. That transparency is to their credit, even though the switch doesn't fix the semantic mismatch. The narrower finding — reformatting A/B tasks as yes/no or agree/disagree questions shifts English responses toward \"No\" when \"No\" corresponds to option B — is consistent and interesting. It holds across all five English models in the yes/no and agree conditions, and mostly in the negated condition. That is a useful empirical result for anyone doing LLM-as-respondent work.\n\nSoft spots, in order: (1) the disagree-condition baseline error kills the \"regardless\" claim; (2) the metric switch is disclosed but under-motivated; (3) the English-only pattern, legal-domain-only data, and single prompt template per condition are all acknowledged in the limitations, so I won't pile on. One more thing: the conclusion that LLMs are not suited to simulate human survey responses is a much bigger jump than the data can bear, even with the narrow finding.\n\nWho this is for: someone working on prompt sensitivity or LLM-as-respondent simulation would get value from the narrow result and from the dataset. The central claim as written should not be published. With the disagree-condition rows re-analyzed and the claims narrowed, it could be a solid short paper or a solid section in a larger study.\n\nMy recommendation: send it to peer review, but the referee will need to require a reanalysis. Don't desk-reject; the empirical core is real and reproducible. The headline just has to be fixed.","headline":"Solid data and clean exposition undermined by a baseline mismatch in the disagree condition that kills the abstract's 'regardless' claim; the narrower English reformatting effect is real but needs to be reported as such.","tokens_in":19826,"tokens_out":2241,"would_cite":false,"duration_ms":19547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models answer 'no' even when 'no' means agreement","keywords":["acquiescence bias","large language models","prompt sensitivity","survey design","response bias","yes/no questions","negation","legal NLP"],"falsifier":"Re-analyze the English results in Table 1 using the neutral option-A rate, not the option-B rate, as the baseline for the disagreement condition's \"no\" responses. If the disagreement-condition \"no\" rate is not consistently above the neutral A rate across the five models, the claim of a no-bias independent of semantic meaning fails.","tokens_in":18553,"feed_emoji":"🤖","tokens_out":5402,"duration_ms":41712,"temperature":0.7,"pith_summary":"This paper tests whether large language models show acquiescence bias, the human tendency to agree with survey statements regardless of content. Across five models, nine binary tasks, and three languages, it finds the opposite: when A/B choices are converted into yes/no questions, models shift toward the token \"no,\" and they do so even in disagreement-phrased prompts where \"no\" logically signals agreement. The effect is consistent across models and tasks in English, while German and Polish show significant but patternless prompt influence. The paper concludes that prompt phrasing materially changes LLM outputs and that LLMs are poorly suited to simulating human survey responses.","feed_headline":"LLMs answer 'no' even when 'no' means yes","feed_subtitle":"Five models and tens of thousands of questions show the token, not the meaning, drives answers.","key_machinery":"The instrument is a paired set of five prompt conditions built from the same binary questions: a neutral A/B choice, a yes/no question on the first option, an agreement question, a negated-agreement question, and a disagreement question. By counting absolute \"no\" responses across conditions and testing differences with McNemar's test, the design isolates the effect of question phrasing from the content of the question. The disagreement condition is the logical hinge: because \"no\" there signals agreement with the statement, a persistent rise in \"no\" responses separates a token-level response bias from a semantic agreement or disagreement bias.","core_discovery":"On its own terms, the paper's discovery is that LLMs display a bias toward the surface response \"no\" that is independent of the semantic content of that answer. In the neutral condition models choose between two options; converting the first option into a yes/no question raises the number of responses equivalent to the second option by 31 to 203 percent for English. Asking \"Do you agree...?\" produces the same increase in \"no\" answers, and asking \"Do you disagree...?\" where \"no\" means agreement with the statement still produces more \"no\" answers for most models. Because the bias tracks the token rather than the logical meaning, the paper argues it is neither acquiescence nor its opposite, and it reads the result as evidence that LLM responses are not primarily driven by reasoning.","pith_inferences":["The no-bias could be a property of the response format: models may be primed to emit the second token of a two-token set, and a test that randomly permutes the order of \"Yes\" and \"No\" would separate a true no-bias from an order effect.","A natural extension is to test the same five conditions in non-legal or conversational domains, where the distribution of true yes/no answers differs and where the paper's own limitation note leaves the question open.","The German and Polish results, where prompt effects are significant but directionless, may indicate that negation and politeness morphology interact with the bias; this goes beyond what the paper claims.","One could reverse the answer options so that the first option is \"No\" and see whether the bias tracks the word \"no\" or the position of the second option."],"forward_implications":["Converting survey items from A/B choices into yes/no format will push LLM outputs toward \"no\" in English, even when the correct answer is the first option.","LLMs are unreliable stand-ins for human survey respondents, at least on legal-domain binary tasks, because they do not reproduce the human acquiescence pattern.","Prompt phrasing is a material variable in LLM evaluation: adding \"do you agree\" or \"don't you agree\" changes accuracy in both directions depending on the task's base error distribution.","The consistency of the no-bias across model sizes suggests it is not a quirk of one architecture or scale, within the tested set.","The finding undercuts the assumption that large models answer such questions by reasoning, since logically opposite phrasings converge on the same surface answer."],"supporting_citations":[{"why":"Defines acquiescence bias and supplies the human-effect baseline that the study contrasts with.","marker":"Krosnick (1999)"},{"why":"The prior study on LLM response biases that this work extends with more questions, additional languages, and different prompt constructions.","marker":"Tjuatja et al. (2024)"},{"why":"Source of the seven English binary tasks used in the experiments.","marker":"Guha et al. (2024)"},{"why":"Source of the German A/B contract-clause corpus.","marker":"Braun and Matthes (2024)"},{"why":"Source of the Polish abusive-clause corpus.","marker":"Augustyniak et al. (2022)"},{"why":"Provides the paired chi-squared test used to assess whether prompt conditions changed responses significantly.","marker":"McNemar (1947)"},{"why":"Supports the critique that negative yes/no questions are ambiguous, motivating the disagreement-condition design.","marker":"Romero and Han (2004)"},{"why":"Documents general prompt sensitivity that motivates expecting response biases in LLMs.","marker":"Zhuo et al. (2024)"}],"fun_headline_variants":["LLMs show a 'no' bias, not acquiescence","Why LLMs always answer 'no' even when it means 'yes'","LLMs: the token 'no' wins over logic and meaning","LLMs say 'no' to agreement, 'no' to disagreement: a token bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on comparing \"no\" answers in the disagreement condition with the neutral condition's second option, even though semantically that \"no\" matches the neutral condition's first option; if that baseline is wrong, the apparent no-bias may disappear.","fun_headline_variants_meta":{"raw":{"variants":["LLMs show a 'no' bias, not acquiescence","Why LLMs always answer 'no' even when it means 'yes'","LLMs: the token 'no' wins over logic and meaning","LLMs say 'no' to agreement, 'no' to disagreement: a token bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000829,"raw_usage":{"total_tokens":3546,"prompt_tokens":797,"completion_tokens":2749,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":2668}},"tokens_in":413,"tokens_out":2749,"duration_ms":18865,"temperature":1.0,"reasoning_tokens":2668,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:08:54.365199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-analyze the English results in Table 1 using the neutral option-A rate, not the option-B rate, as the baseline for the disagreement condition's \"no\" responses. If the disagreement-condition \"no\" rate is not consistently above the neutral A rate across the five models, the claim of a no-bias independent of semantic meaning fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Polish abusive-clause corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the critique that negative yes/no questions are ambiguous, motivating the disagreement-condition design."}],"review_version":2}