{"id":"61e397e9-6382-4ca4-a209-0a671f9ee597","arxiv_id":"2412.11679","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Subtracting the weight difference between a model trained on stereotyped text and a pre-trained model reduces measured bias on SEAT by about 0.18 effect-size points without hurting GLUE scores at scale factor one.","lead":"This paper trains language models on biased text, computes the difference between the trained and original weights, and subtracts that difference from the original model to reduce bias. The method is a cheap, data-driven alternative to manually building debiasing datasets, though its gains on the SEAT benchmark are modest and some strong baselines still outperform it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No matched control for generic fine-tuning: a control vector trained on non-stereotyped text could produce the same SEAT drop, so the claim that the Bias Vector carries bias-specific information is unverified.","rationale":"I focused on the empirical core of the paper rather than the linear-separability framing because the reported results themselves can settle the question. The method's novelty is that a vector computed from biased data carries bias-specific information. The paper demonstrates that subtracting this vector lowers SEAT at λ=1 and that GLUE is roughly flat, but it never rules out the trivial alternative: any task vector from continual MLM on a corpus of similar size and register, even one containing no stereotypes, would lower SEAT by a comparable amount. The paper's own §5.6 supports the plausibility of this alternative by showing that SEAT approaches zero as representations collapse at large λ; the control experiment is needed to show that at λ=1 the drop is not the beginning of the same collapse. The appendix comparison to existing debiasing methods uses only gender subtests and does not include a neutral fine-tuning control. I therefore agree with the reader's CONDITIONAL verdict, but I locate the load-bearing assumption somewhat differently: not primarily 'bias is linearly separable,' but 'the bias vector is specifically bias-aligned rather than a generic adaptation direction.' A single matched control experiment would either validate the method or falsify its central claim.","tokens_in":16418,"tokens_out":3334,"duration_ms":33493,"concrete_test":"Run a control condition: continually train each of BERT, ALBERT, and RoBERTa on an equal-sized corpus of StereoSet intrasentence sentences with the blanks filled by anti-stereotype or by a random/neutral distractor, using identical hyperparameters (lr=1e-4, batch=128, 30 epochs, same 10 seeds). Compute the control vector V_ctrl = θ_ctrl − θ_org, subtract at λ=1, and report SEAT effect sizes exactly as in Table 3. If the control achieves an average reduction statistically indistinguishable from BV(all,1) (e.g., within one standard error across seeds), then the 0.177-point improvement is not attributable to bias-specific information, and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that subtracting V_bias = θ_bias − θ_org removes bias because the vector encodes stereotypical associations learned from StereoSet. The paper compares only against the pre-trained baseline; it never compares against a control Bias Vector built from an equal-sized corpus of non-stereotyped (or anti-stereotyped) text trained with identical hyperparameters and seeds. Without this control, the observed SEAT improvement at λ=1 could be an artifact of any continual-MLM fine-tuning vector: subtracting a generic adaptation direction may shrink embedding separability generally, lowering effect sizes without specifically removing stereotype associations. This is not a hypothetical: the paper itself shows (Sec. 5.6) that large λ drives SEAT toward zero precisely because representations collapse, so the metric is sensitive to representation degradation. At λ=1 GLUE is roughly preserved after fine-tuning, but fine-tuning can mask mild representational damage, and SEAT is measured on the unfine-tuned debiased encoder. The load-bearing assumption is therefore not merely linear separability, but that the vector is specifically bias-aligned; a matched control experiment is the minimal check. The appendix comparison to existing debiasing methods (Table 5) does not remedy this, as those methods are trained on gender-specific data and evaluated only on gender SEAT subtests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'Bias Vector' method for debiasing pre-trained language models. The method first continually trains a model on biased text (StereoSet intrasentence sentences with stereotype fill-ins), computes the parameter difference between this biased model and the original pre-trained model (the bias vector), and then subtracts a scaled version of this vector from the original weights to obtain a debiased model. Experiments on BERT, ALBERT, and RoBERTa report an average improvement of 0.177 in absolute SEAT effect sizes at λ=1, while GLUE performance is reported as approximately unchanged. The paper also analyzes the effect of the scaling factor λ and shows that large λ values drive SEAT effect sizes toward zero, but attributes this to a collapse of representations rather than genuine debiasing.","tokens_in":16604,"tokens_out":9087,"duration_ms":82437,"significance":"If the central claim holds, the method would offer a simple, data-efficient debiasing approach that avoids constructing dedicated debiasing datasets, consistent with recent task-arithmetic ideas. The paper has notable strengths: it evaluates three model architectures, excludes StereoSet from the evaluation suite to reduce leakage concerns, constructs bias vectors over ten seeds, and honestly analyzes in Section 5.6 that large λ values can make SEAT scores decrease because of representation collapse rather than true bias removal. However, the main claim is not yet fully established. The paper lacks a matched control vector trained on non-stereotyped text, the per-bias-type vectors are inconsistent across models, the headline results rely on average absolute effect sizes, and no variance or significance information is provided for the GLUE comparison. These issues leave open the possibility that the observed improvements arise from generic fine-tuning effects or incipient representation degradation rather than from the removal of bias-specific information.","major_comments":[{"comment":"The central claim that V_bias encodes bias-specific information is not verified by the experiments as presented. The only comparison is against the pre-trained baseline; there is no control vector constructed from an equal-sized corpus of non-stereotyped (or anti-stereotyped) text trained with identical hyperparameters and seeds. Without such a control, the observed SEAT reduction at λ=1 could be produced by any continual-MLM fine-tuning direction that slightly degrades embedding separability, especially given the paper's own finding (Section 5.6) that SEAT approaches zero when representations collapse. Please add a matched control experiment—for example, a vector trained on neutral text from the same domain—and show that the bias vector reduces SEAT significantly more than the control.","section":"Section 3.2, Section 5.1"},{"comment":"The headline 0.177 improvement is reported for the aggregated 'all' bias vector only, and the per-bias-type vectors are inconsistent. For ALBERT, BV(gender,1) and BV(religion,1) yield average absolute effect sizes of 0.736 and 0.735, which are above the pre-trained baseline of 0.675; for BERT, the individual vectors give much smaller improvements than BV(all,1). The paper should report signed effect sizes and standard deviations across the ten seeds for each vector, and explain why the aggregate vector behaves differently from its components; without this, the claim of consistent bias mitigation is not supported.","section":"Section 5.1, Table 3"},{"comment":"The main tables report averages of absolute effect sizes. This can conflate genuine debiasing with bias reversal: if a debiasing step changes a positive effect size to a negative one of similar magnitude, the absolute value changes while the bias flips direction. Section 5.5 explicitly reports such reversal for intermediate λ, and Table 5 reports signed effect sizes. Please report signed effect sizes for all conditions and test whether the signed values move toward zero, not merely whether absolute values decrease.","section":"Section 5.1, Section 5.5, Table 5"},{"comment":"The GLUE experiments are run with a single fixed seed, so the claim that performance is 'not degraded' at λ=1 is not statistically supported. The observed changes (BERT +0.3%, ALBERT +0.6%, RoBERTa −0.2%) are within the range of typical seed-to-seed variation for GLUE fine-tuning. Please report multiple seeds with means and standard deviations, or at least provide a paired bootstrap or significance test over several seeds for the λ=1 condition.","section":"Section 4.4, Table 4"},{"comment":"The comparison with existing debiasing methods in Table 5 is indirect: the baseline and baseline-method rows are cited from Meade et al. (2022), while the Bias Vector rows are from the present paper's own evaluation pipeline. Differences in evaluation code, sentence templates, or data splits could account for part of the reported differences. The claim that the proposed method is 'sometimes competitive' with embedding-space methods would be more convincing if the existing methods were re-run under the same codebase (e.g., the bias-bench code the authors cite) with matched seeds and evaluation settings.","section":"Appendix D.2, Table 5"}],"minor_comments":[{"comment":"There are several typos and grammatical issues, including 'ahve' (Section 2.1), 'neasurements' (Section 2.1), 'emperical' (Appendix D.2), 'Arrguments' (Appendix C), and the title line 'Bias V ector' in the first line of the manuscript.","section":"Throughout"},{"comment":"The table caption says the results are for λ=1, but the table also includes rows for λ=10 and λ=100; the caption should be revised to describe the full table content.","section":"Table 4, caption"},{"comment":"The exclusion of LayerNorm layers from the vector subtraction is stated without motivation or ablation. A sentence explaining why this choice was made, and ideally an ablation showing its effect, would be helpful.","section":"Section 3.2"},{"comment":"The paper acknowledges that SEAT does not directly evaluate profession bias, but the main table reports an average over gender, race, and religion subtests. It would be useful to also report the results separately by SEAT subtest to clarify which bias types drive the observed improvements.","section":"Section 5.3"},{"comment":"The reference list for CDA includes multiple works that use different formulations (e.g., counterfactual data augmentation, dialogue debiasing); a sentence clarifying which variant was used in the cited results would improve reproducibility.","section":"Appendix D.2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of the journal but the novelty is incremental relative to task-arithmetic literature. The main weakness is the missing control vector experiment, which is needed to substantiate the claim that the bias vector carries bias-specific information. If the authors add a matched control, report signed effect sizes with variance, and provide multi-seed GLUE results, the paper could become acceptable. The indirect comparison in Table 5 is a further concern that should be addressed. There is no indication of misconduct or inappropriate citation practice. The honest discussion of representation collapse is a positive feature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is worth a serious referee, but the headline is a half-step ahead of the evidence. The new thing is applying task arithmetic to debiasing: train on stereotyped text, form the weight difference, subtract it. That's a clean, practical idea, and the evaluation isn't circular—StereoSet is used for training and excluded from SEAT. The scaling analysis is the most valuable part; the authors show that large λ drives SEAT to zero because representations collapse, not because bias is removed. That honesty is a real strength.\n\nThe soft spots are real but not disqualifying. The biggest is the missing matched control: no vector trained on an equal-sized non-stereotyped corpus with the same hyperparameters and seeds. Without that, the SEAT improvement at λ=1 could be any generic fine-tuning direction shrinking separability. The per-bias vectors actually argue against a pure generic effect—BV(gender,1) worsens ALBERT while BV(all,1) improves it—so the control is needed to establish the mechanism, not to rescue a hopeless claim. Second, GLUE scores have no variance; 'preserves performance' rests on a single seed and the RoBERTa number is negative. Third, the appendix comparison to existing methods covers only gender bias and the method underperforms in some cells; that's acceptable, but it should be presented as context, not as superiority. The abstract says 'confirmed an average improvement of 0.177 points'—accurate, but it hides that per-bias vectors are inconsistent.\n\nWho gets value: anyone working on debiasing or weight arithmetic. The method doesn't need manual debiasing data, which is a genuine practical advantage, and the cost is a few hours of MLM training. Replication is feasible even without code, though code would help. I'd send this to peer review with a request for the control experiment and multi-seed GLUE. The core idea is worth testing, and the paper is honest about its limits.","headline":"Worth a serious referee, but the central claim needs a matched control before it carries the weight the abstract puts on it.","tokens_in":17210,"tokens_out":3037,"would_cite":true,"duration_ms":31152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that subtracting a Bias Vector—the weight difference from over-training on stereotyped text—from a pre-trained language model cuts measured bias by 0.177 points on average and preserves downstream task performance.","keywords":["bias mitigation","task arithmetic","Bias Vector","weight editing","SEAT","GLUE","StereoSet","masked language modeling"],"falsifier":"Repeat the continual training with several random seeds and check whether the Bias Vectors agree in direction (high pairwise cosine similarity) and whether the per-seed SEAT improvements bracket zero; if the vectors are seed-dependent or the SEAT spread exceeds the 0.177-point average gain, the claimed bias direction is not a stable property of the weights.","tokens_in":16148,"feed_emoji":"⚖️","tokens_out":8614,"duration_ms":75543,"temperature":0.7,"pith_summary":"The paper tries to show that social bias in a language model can be treated like a task vector in weight space: continually train the same model on stereotype-saturated text, take the difference between the trained and original weights as a 'Bias Vector', and subtract that vector back off the original model. The claim is that this removes a measurable amount of bias without any manually curated debiasing dataset. Evaluated on the SEAT benchmark, BERT, ALBERT, and RoBERTa improved by an average of 0.177 effect-size points at scaling factor λ=1, while GLUE scores stayed essentially unchanged. The authors also argue that pushing λ high drives SEAT effect sizes toward zero not because bias is gone but because the model's representations collapse, so large-λ scores should not be read as successful debiasing.","feed_headline":"Subtract a bias vector to cut LM bias by 0.177","feed_subtitle":"One weight-space subtraction shrinks SEAT effect sizes across BERT, ALBERT, and RoBERTa without hurting GLUE scores.","key_machinery":"The central object is the Bias Vector, defined as the weight-space difference between a model continually trained on biased text and the original model, $V_{\\mathrm{bias}} = \\theta_{\\mathrm{bias}} - \\theta_{\\mathrm{org}}$. It is applied through the simple rule $\\theta_{\\mathrm{debias}} = \\theta_{\\mathrm{org}} - \\lambda V_{\\mathrm{bias}}$, so the mechanism is vector addition and subtraction of parameter tensors under the assumption that models share an architecture; the scalar $\\lambda$ controls how much bias is subtracted. The paper also excludes Layer Normalization parameters from the vector, arguing those layers only normalize activations and carry no bias information. The mechanism carries the argument because it turns debiasing into a one-line arithmetic operation: no debiasing corpus, no additional training at inference, and the same vector can be scaled up or down to trade bias reduction against preservation of language ability.","core_discovery":"The central discovery is that the arithmetic of task vectors transfers to bias: with $\\theta_{\\mathrm{org}}$ the pre-trained weights and $\\theta_{\\mathrm{bias}}$ the weights after masked-language-model continual training on stereotyped StereoSet sentences, the vector $V_{\\mathrm{bias}} = \\theta_{\\mathrm{bias}} - \\theta_{\\mathrm{org}}$ encodes bias, and $\\theta_{\\mathrm{debias}} = \\theta_{\\mathrm{org}} - \\lambda V_{\\mathrm{bias}}$ yields a debiased model. At $\\lambda=1$ this lowers average absolute SEAT effect sizes from 0.672/0.675/0.733 to 0.447/0.534/0.570 for BERT/ALBERT/RoBERTa, an average improvement of 0.177 points, while average GLUE scores move from about 0.776/0.779/0.794 to 0.779/0.785/0.792. The same experiment shows that subtracting too much (large λ) makes effect sizes collapse toward zero and GLUE scores fall sharply, which the paper reads as evidence that over-debiasing destroys representations rather than truly removing bias. Bias-type-specific vectors (race, profession, gender, religion, and all combined) all reduce SEAT scores at λ=1, and the Layer Normalization layers are excluded from the subtraction because they are treated as normalization-only.","pith_inferences":["As the paper's own future-work section anticipates, the recipe should transfer to decoder-only LLMs; the open question is whether a Bias Vector built on 8,498 sentences remains a clean direction when subtracted from a much larger model.","Because the vector is built from StereoSet, whose stereotypes were annotated from a U.S. perspective, the method can only remove the bias directions present in that corpus; in another cultural context it could leave local stereotypes intact or distort them, so corpus coverage is a hidden boundary condition.","The bias-direction reversal seen at intermediate λ before collapse suggests that subtracting the vector moves the model along a continuum between the biased and original checkpoints; per-bias or per-layer λ tuning could exploit this to get finer control than the single global scalar used here.","A sharper evaluation than GLUE would be to compare layer-wise contributions of the Bias Vector; if only a few layers carry the bias direction, subtracting from those layers alone could cut SEAT with less collateral damage than the uniform subtraction the paper applies."],"forward_implications":["With λ=1, all four bias-type vectors (race, profession, gender, religion) and the combined vector lower average absolute SEAT effect sizes for BERT, ALBERT, and RoBERTa, so a single subtraction step is a usable debiasing recipe across BERT-family models.","GLUE averages after debiasing at λ=1 are 0.779 (BERT), 0.785 (ALBERT), and 0.792 (RoBERTa), within about 0.2–0.6 points of the pre-trained baselines, so the paper's debiasing does not trade away downstream language understanding.","Because large λ drives both SEAT and GLUE scores down simultaneously, near-zero effect sizes on SEAT are not, by themselves, evidence of successful debiasing; task performance must be checked in the same experiment.","Debiasing with the profession vector alone also improved SEAT scores on other categories, which the paper attributes to overlapping bias directions; this implies that composing multiple bias vectors requires care to avoid double-subtracting shared components."],"supporting_citations":[{"why":"Supplies the task arithmetic idea that the Bias Vector directly adapts: model behavior can be edited by adding or subtracting weight-space vectors.","marker":"Ilharco et al., 2023"},{"why":"Supplies the StereoSet intrasentence stereotyped sentences used to continually train the biased LMs.","marker":"Nadeem et al., 2021"},{"why":"Supplies the SEAT benchmark whose effect sizes are the paper's primary bias measure.","marker":"May et al., 2019"},{"why":"Provides the empirical survey, evaluation code, and baseline method results (CDA, Dropout, INLP, SentDebias) that frame the experimental setup.","marker":"Meade et al., 2022"},{"why":"Provides the BERT model, one of the three pre-trained LMs the method is tested on.","marker":"Devlin et al., 2019"},{"why":"Provides the ALBERT model, the second testbed architecture.","marker":"Lan et al., 2020"},{"why":"Provides the RoBERTa model, the third testbed architecture.","marker":"Liu et al., 2019"},{"why":"Supplies the GLUE benchmark used to verify that debiased models keep downstream task performance.","marker":"Wang et al., 2018"}],"fun_headline_variants":["Subtract bias vector: LM bias down 0.177 on SEAT","Weight subtraction debiases LMs: 0.177 improvement","Bias vector method: no data, just subtract weights","Cut LM bias by 0.177 with vector arithmetic","Task arithmetic removes bias: 0.177 SEAT drop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model's bias is stored as a direction in parameter space that can be cleanly separated from general language ability, so that subtracting a fixed multiple of the Bias Vector removes bias without erasing other capabilities.","fun_headline_variants_meta":{"raw":{"variants":["Subtract bias vector: LM bias down 0.177 on SEAT","Weight subtraction debiases LMs: 0.177 improvement","Bias vector method: no data, just subtract weights","Cut LM bias by 0.177 with vector arithmetic","Task arithmetic removes bias: 0.177 SEAT drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":2119,"prompt_tokens":1066,"completion_tokens":1053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":965}},"tokens_in":682,"tokens_out":1053,"duration_ms":9337,"temperature":1.0,"reasoning_tokens":965,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:41:50.565763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the continual training with several random seeds and check whether the Bias Vectors agree in direction (high pairwise cosine similarity) and whether the per-seed SEAT improvements bracket zero; if the vectors are seed-dependent or the SEAT spread exceeds the 0.177-point average gain, the claimed bias direction is not a stable property of the weights.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the task arithmetic idea that the Bias Vector directly adapts: model behavior can be edited by adding or subtracting weight-space vectors."}],"review_version":1}