{"id":"62122a6c-1523-425e-86af-ea048ca0167d","arxiv_id":"2602.04306","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM fairness scores shift substantially with positive vs negative framing of the same question, and DeFrame—a three-step self-revision prompt—reduces both average bias and this framing gap.","lead":"This paper shows that large language models give more biased answers when the same fairness question is worded in a negative versus a positive frame, and proposes a multi-step prompting method, DeFrame, that asks the model to rephrase, build fairness guidelines, and revise its answer. The work matters because standard fairness benchmarks may miss hidden bias that only appears when prompt wording changes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BBQ framing disparity may be confounded: P+ and P- are unpaired question sets, not alternative framings of the same content.","rationale":"The reader identified the same core concern: BBQ's P+/P- sets are not semantically paired. This is the most load-bearing issue because the paper's headline results—'bias under negative framings is on average 2x larger' and 'DeFrame reduces framing disparity by 92%'—are computed on BBQ. If the two sets differ in content rather than framing, then the FD metric (Eq. 2) is not measuring what the paper claims. The examples in Table 5 confirm the sets are not antonymic paraphrases: e.g., 'opportunity to learn how to use the phone' is not the opposite of 'not comfortable using the phone'; they are different attributes. The matched benchmarks (DoNotAnswer-Framed, 70Decisions-Framed) use genuine polarity-flipped counterparts, so the qualitative finding of framing disparity is likely to survive. However, the specific quantitative claims from BBQ, especially the 92% reduction, would need to be recomputed on properly paired sets. The proposed test—recompute FD on matched BBQ pairs—directly addresses this. If the FD vanishes on matched pairs, then the central claim is weakened but not entirely falsified; if it persists, the concern is mitigated. Because the reader already conditioned acceptance on this issue, I recommend no change to the CONDITIONAL verdict.","tokens_in":33962,"tokens_out":5601,"duration_ms":58545,"concrete_test":"Construct matched positive/negative pairs for every BBQ ambiguous question by applying the same antonym-flipping procedure used for DoNotAnswer-Framed (or by manually/LLM-generating a polarity-reversed version of the evaluative phrase while keeping scenario and options identical). Recompute framing disparity for the 8 models on these matched pairs. If the average absolute FD decreases substantially (e.g., by >50%) compared to the original BBQ FD, then the reported BBQ framing disparity is largely a content effect rather than a framing effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that BBQ's existing negative and non-negative question sets are used as P- and P+. These sets are not paired rewrites of the same semantic content; Table 5 shows items such as 'Who was giving the other the opportunity to learn how to use the phone?' vs. 'Who was not comfortable using the phone?' and 'Who was acting calm and reasonably?' vs. 'Who was acting irrationally?'. The evaluative attributes differ, and the questions carry different presuppositions (e.g., one presupposes a teaching event, the other a discomfort). Therefore Bias(M;P+,ϕ) and Bias(M;P-,ϕ) in Eq. 2 may differ because of question difficulty, topic, or stereotype salience, not because of framing. Consequently, the reported BBQ framing disparities (e.g., 2x–4x bias under negative framing, and DeFrame's 92% FD reduction) may be artifacts of comparing non-equivalent prompt sets. The paired benchmarks (DoNotAnswer-Framed, 70Decisions-Framed) partially mitigate this, but the headline quantitative claims are based on BBQ.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies the framing effect as an underexplored source of hidden bias in LLM fairness evaluations. It defines a \"framing disparity\" (FD) metric, Sec. 3.1 Eq. (2), and evaluates eight instruction-tuned LLMs on three benchmarks: BBQ, DoNotAnswer-Framed, and 70Decisions-Framed. The evaluation shows substantial FD, with models more biased under negative framing on BBQ but more harmful under positive framing on DoNotAnswer-Framed. The paper then proposes DeFrame, a three-stage prompting method (framing integration, guideline generation, self-revision) that explicitly reconciles opposite framings, and reports that DeFrame reduces FD by 92% and bias score by 93% on average on BBQ, while outperforming existing prompting-based debiasing baselines. Ablations show that all three components contribute to stable FD reduction.","tokens_in":34287,"tokens_out":4987,"duration_ms":60428,"significance":"If the results hold, the framing disparity metric is a useful complement to existing fairness benchmarks, highlighting that a single prompt template can give a misleading picture of an LLM's bias. The DoNotAnswer-Framed and 70Decisions-Framed extensions are paired-rewrite benchmarks, and the paper includes full prompt templates (Appendix E) and broad model coverage (8 main models plus 5 in the 30B-70B range), which is a reproducibility strength. The finding that existing debiasing methods reduce average bias but not framing-induced variance is practically important. However, the two major concerns described below—the unpaired BBQ construction and the partly self-fulfilling evaluation of DeFrame—currently cap confidence in the headline quantitative claims.","major_comments":[{"comment":"yes","section":"Sec. 3.2, Eq. (2), Table 5"},{"comment":"yes","section":"Sec. 4.1, Table 7, Sec. 4.2"}],"minor_comments":[{"comment":"yes","section":"Sec. 3.3, Table 1, Appendix F"},{"comment":"yes","section":"Sec. 3.2"},{"comment":"yes","section":"Appendix C.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The BBQ P+/P− issue is the most serious problem. The paper's central concept and the paired benchmarks are valuable, but the headline 92% reduction and the 2x/4x claims in the abstract and introduction currently rest on unpaired question sets. A revision that either re-runs the BBQ experiments with properly constructed paired framings, or restricts the quantitative claims to the paired benchmarks, would put the paper on much firmer ground. The self-fulfilling evaluation concern is also worth addressing, perhaps by evaluating DeFrame on held-out paraphrase variants."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper deserves a referee. The framing-disparity concept is simple and useful, and the paired polarity-augmented benchmarks are a real contribution. But the headline BBQ numbers are likely inflated by a measurement confound: the positive and negative BBQ question sets are not paired framings, and the effect of DeFrame is at least partly a response to its own instruction.\n\nThe new stuff: FD is just the difference in bias scores between a positive and negative prompt set, but systematically applying it to fairness benchmarks is a legitimate new lens. DoNotAnswer-Framed and 70Decisions-Framed are genuinely paired: they flip a stereotype to its opposite polarity, with paraphrases (DoNotAnswer) and flipped decision questions (70Decisions). The finding that DoNotAnswer is more harmful under positive framing is counterintuitive and a good sanity check that the phenomenon is real. The model coverage (13 models) is decent; the larger-model appendix is a plus. The ablations are clean and incrementally monotone, and the limitations section is honest.\n\nThe soft spots: first, the BBQ analysis uses BBQ's existing negative vs non-negative questions as P- and P+. From the examples in Table 5 these differ in content, not just polarity: 'acting calm and reasonably' vs 'acting irrationally' are different predicates. So the 2x-4x FD on BBQ and the 92% reduction after DeFrame may be measuring question difficulty or stereotype salience, not framing. The paired benchmarks support the concept but not those specific headline numbers. Second, DeFrame's evaluation partly measures the method's own instruction: the prompt tells the model to answer consistently across the original and a flipped framing, so a drop in FD on that same dimension isn't surprising. A held-out framing test would be stronger. Third, there are no error bars or significance tests, and 70Decisions-Framed is a single run. Code and data should be released.\n\nNone of this kills the paper. The conceptual core — fairness scores are unstable across genuinely paired polarity framings, and existing debiasing doesn't fix it — is supported by the two new benchmarks. I'd send it to peer review and tell the authors to re-pair BBQ or drop the BBQ headline, report variance, and release artifacts. It's a useful paper for anyone working on bias evaluation and prompt sensitivity.","headline":"A real hidden-bias axis and genuinely paired benchmarks — but the BBQ headline numbers rest on unpaired prompt sets and the DeFrame evaluation is partly self-fulfilling.","tokens_in":34673,"tokens_out":4525,"would_cite":true,"duration_ms":51040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM fairness evaluations are distorted by framing: with the same stereotype, positive versus negative wording produces substantially different measured bias, and the paper's DeFrame method reduces that framing disparity by 92% and bias scor","keywords":["framing effect","LLM fairness","framing disparity","debiasing","prompt sensitivity","hidden bias","BBQ","self-revision"],"falsifier":"Build a paired-frame version of BBQ where each item is rewritten by flipping only the comparative relation (e.g., 'X is more likely to be calm than Y' / 'Y is more likely to be irrational than X') while keeping the same attribute pair and answer options, then recompute framing disparity across the eight models. If FD on paired rewrites is near zero, the headline framing-disparity result is largely an artifact of unmatched content; if it stays large, the framing claim is confirmed.","tokens_in":33909,"feed_emoji":"⚖️","tokens_out":5319,"duration_ms":54236,"temperature":0.7,"pith_summary":"This paper tries to establish that a large part of so-called hidden bias in LLMs is a framing effect: the same stereotype, posed in positive wording versus negative wording, draws measurably different bias levels from the same model. To make this measurable, the paper defines framing disparity, the difference between a model's bias score under positive and negative framings of a benchmark. Across eight instruction-tuned models and three fairness benchmarks, the gap is common and sometimes large: in BBQ, negative framings produce about twice the bias of positive ones, up to four times in some categories, and existing debiasing prompts mostly lower the frame-averaged bias without closing the frame gap. The paper's DeFrame method rephrases each prompt into its opposite framing, writes a fairness guideline from both versions, and then has the model revise its first answer; on BBQ this reduces framing disparity by 92% and bias score by 93% on average. A sympathetic reader would take away that fairness numbers are wording-dependent and that a cheap inference-time remedy can make them more stable.","feed_headline":"DeFrame cuts LLM framing disparity by 92% in BBQ","feed_subtitle":"Bias scores under positive vs negative wording can differ 2x; most debiasing methods leave the gap untouched.","key_machinery":"The central object is framing disparity, FD = Bias(P+) − Bias(P−), a signed difference (reported as |FD|) built on each benchmark's own bias metric. The remedy is DeFrame, a three-stage inference-time prompting pipeline: framing integration, where the model rewrites the question's evaluative phrase into its antonym, producing the opposite framing; guideline generation, where the model writes a rule requiring both phrasings to be answered consistently and without identity-based assumptions; and self-revision, where the initial answer is checked and revised against that guideline. The paper argues that explicitly reasoning about the opposite framing is what stabilizes responses across phrasing","core_discovery":"On the paper's own terms, fairness in LLMs is framing-dependent. Framing disparity equals the difference in a benchmark's bias score between positive and negative phrasings of the same stereotype. Across eight LLMs and three benchmarks the gap is substantial: on BBQ, negative framings score about twice the bias of positive ones (up to four times for disability status); on DoNotAnswer-Framed, positive framings draw more harmful responses; on 70Decisions-Framed, reframing can flip which demographic group is favored. Existing prompting debiasing lowers frame-averaged bias but not framing disparity, sometimes making it worse. DeFrame—antonym rephrasing, guideline writing, self-revision—reduces b","pith_inferences":["The BBQ framing sets are not paired rewrites: the paper reuses BBQ's pre-existing negative and non-negative question sets, which differ in the trait words themselves (e.g., 'calm and reasonably' vs 'irrationally'). Part of the measured disparity may therefore be stereotype-content difference rather than pure framing.","A direct testable extension would run DeFrame on strictly paired rewrites ('A is better than B' vs 'B is worse than A') to see whether the 92% reduction survives when the semantic content is provably identical.","DeFrame's guideline nudges ambiguous cases toward 'unknown'; part of its bias-score reduction may come from a general abstention effect rather than frame-awareness, which could be isolated by comparing against a guideline that only demands consistency, not neutrality.","The same framing-disparity lens could be applied to other LLM properties such as toxicity or sycophancy, where positive/negative wording may hide systematic variation."],"forward_implications":["Fairness reports that quote a single bias number are incomplete; the same model can be fair under one wording and biased under another.","Benchmark builders should augment with paired opposite-framing prompts and report both frame-averaged bias and framing disparity.","In high-stakes decisions, outcomes can flip solely from wording; frame-aware revision is a lightweight guard against this.","Scaling model size lowers absolute bias in the paper's large-model experiments, but framing disparity persists, so capacity alone is not the fix.","Existing debiasing methods that ignore framing can even increase disparity, so debiasing evaluation should check per-frame behavior, not just the mean."],"fun_headline_variants":["Framing flips LLM bias scores; DeFrame steadies them","LLM fairness varies by wording — DeFrame closes the gap","Debiasing that works: DeFrame cuts framing-induced bias","DeFrame: Making LLM fairness consistent across framings","Framing disparity exposed; DeFrame reduces it"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the positive and negative prompt sets used to measure framing disparity really are alternative phrasings of the same semantic content; in BBQ, the paper uses pre-existing negative and non-negative question sets that are not matched rewrites, so the framing gap could partly be a content gap.","fun_headline_variants_meta":{"raw":{"variants":["Framing flips LLM bias scores; DeFrame steadies them","LLM fairness varies by wording — DeFrame closes the gap","Debiasing that works: DeFrame cuts framing-induced bias","DeFrame: Making LLM fairness consistent across framings","Framing disparity exposed; DeFrame reduces it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3567,"prompt_tokens":744,"completion_tokens":2823,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":2737}},"tokens_in":488,"tokens_out":2823,"duration_ms":21555,"temperature":1.0,"reasoning_tokens":2737,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:38:49.902684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a paired-frame version of BBQ where each item is rewritten by flipping only the comparative relation (e.g., 'X is more likely to be calm than Y' / 'Y is more likely to be irrational than X') while keeping the same attribute pair and answer options, then recompute framing disparity across the eight models. If FD on paired rewrites is near zero, the headline framing-disparity result is largely an artifact of unmatched content; if it stays large, the framing claim is confirmed.","supporting_citations":[],"review_version":1}