{"id":"ef6fc3d3-4417-4cb1-b7a2-22c4952b085b","arxiv_id":"2501.10150","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"2DAMA adapts model weights with a modified concept-erasure projection that erases stereotypical gender associations while preserving factual gender signals, and reports bias reductions in English and in English-to-German/Czech/Russian translation.","lead":"2DAMA is a new method that edits large language models to remove gender stereotypes while retaining factual gender cues, and the authors test it on English language modeling and on English-to-German, Czech, and Russian translation. It matters because it tries to solve a known trade-off in debiasing: keeping useful gender information while reducing harmful bias, which could make translation and text generation fairer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Translation evidence for factual-gender preservation rests on a signed-difference metric; the reported negative ΔG values are bias reversals, not improvements, so the central 'dual' claim is unsupported for MT.","rationale":"The reader's weakest_assumption focuses on the linear feed-forward approximation underlying DAMA-LEACE and Dual Debiasing. That is a genuine theoretical limitation, but the paper's own empirical LM results are at least consistent with the method working despite the violation, so it is not the single most load-bearing point. The translation half of the central claim is more fragile. The abstract and Section 5.1 claim that 2DAMA reduces stereotypical bias in translation while preserving factual gender. The only metric used to support the factual-gender preservation in translation is ΔG, and the reported numbers are either larger in magnitude (German t=1.00: -10.1 vs. 3.7) or worse (Czech t=1.00: 7.2 vs. 2.1) than baseline when interpreted as a signed difference. The paper treats the negative German value as an improvement, which is only coherent if one mistakes sign reversal for bias elimination. If ΔG is meant as an absolute difference, then German t=1.00 gives 10.1, which is not an improvement either. The same issue appears in Figure 5b, where negative ΔG values near t=1.00 are marked as part of the best setting. This matters because the novelty over DAMA and DAMA+LEACE is precisely the 'dual' component: erase stereotypes while retaining factual gender. If the multilingual evidence for that component is a misread difference score, the central claim is much weaker. A concrete recomputation of |ΔG| and per-gender F1 would settle the point. Because the LM experiments and the ΔS reductions in translation still support a conditional acceptance with corrections, I do not move the verdict away from CONDITIONAL; it stays UNCHANGED in the sense that the reader's verdict already requires such corrections.","tokens_in":20944,"tokens_out":8858,"duration_ms":85813,"concrete_test":"Recompute Table 4 and Figure 5b by reporting ΔG as signed 'male F1 minus female F1' and also computing |ΔG| and per-gender F1 for German and Czech at t=1.00. Check whether |ΔG(2DAMA t=1.00)| is reduced relative to |ΔG(ALMA-R baseline)| in both languages; if it is not reduced, or if the negative value corresponds to female F1 exceeding male F1 by a wide margin, then the Section 5.1 claim that 2DAMA preserves equitable factual gender in translation is contradicted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the interpretation of the representational-bias metric used to support the translation half of the central claim. In Table 4, the paper reports ΔG (the difference in translation F1 between masculine and feminine entities) and marks it with a downward arrow as if lower is better. Under the standard signed definition, German 2DAMA at t=1.00 moves ΔG from 3.7 to -10.1: this is a large reversal of bias, not a reduction. Under an absolute-value reading it is 10.1, worse than baseline. Czech ΔG worsens from 2.1 to 7.2 even at t=1.00. The text in Section 5.1 interprets these values as evidence that higher thresholds 'preserve more factual signal' and produce 'equitable representation of factual gender', but ΔG does not measure factual-signal preservation; it measures asymmetric translation accuracy between genders. Without a valid measure of factual-gender preservation in translation, the paper's distinctive claim—dual debiasing that keeps factual gender while removing stereotypes in machine translation—is unsupported for the multilingual setting. The LM experiments may still support a weaker claim, but the translation results, as reported, do not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 2DAMA (Dual Debiasing Algorithm through Model Adaptation), a post-hoc model-editing method that aims to reduce stereotypical gender bias in large language models and in machine translation while preserving factual gender information. The method combines DAMA (Limisiewicz et al., 2024), LEACE (Belrose et al., 2023), and a new covariance-based decomposition that, given a hand-set threshold t, nullifies dimensions associated with stereotypical bias while retaining dimensions correlated with factual gender. Experiments on Llama 2 (7B, 13B), Llama 3 (8B), and ALMA-R (13B) report reductions on bias metrics (linear stereotype coefficients, WinoBias, WinoMT, BUG) with modest perplexity and ARC degradation. The paper also studies hyperparameter choices and cross-lingual transfer of debiasing signals.","tokens_in":21101,"tokens_out":4920,"duration_ms":49562,"significance":"If the central claim were fully supported, the paper would make a useful contribution: it addresses an understudied trade-off between debiasing and preserving legitimate gender cues, and it is among the first attempts to debias LLM-based machine translation post hoc. The paper presents a theorem relating concept erasure and feature preservation, a layer-wise analysis of where stereotypical and factual gender signals reside, and a cross-lingual generalization study. These analyses are valuable. However, the significance is currently tempered by two issues: the practical 2DAMA algorithm explicitly abandons the theorem's conditions, and the translation experiments use a signed ΔG metric whose negative values are interpreted as preservation rather than as bias reversal. The English language-modeling results are more solid, but they alone do not support the paper's distinctive dual-debiasing claim for machine translation.","major_comments":[{"comment":"The interpretation of ΔG for 2DAMA (t = 1.00) is not supported by the numbers. The paper marks ΔG with a downward arrow (lower is better) but then presents a negative value for German (3.7 → −10.1) as evidence that higher t 'tends to preserve more factual signal' and yields 'equitable representation of factual gender.' Under the standard signed definition, a negative ΔG is a bias reversal, not a reduction; under an absolute-value reading, the bias magnitude increases from 3.7 to 10.1. For Czech, ΔG worsens from 2.1 to 7.2 even at t = 1.00. Moreover, ΔG measures the difference in translation accuracy between masculine and feminine entities; it does not measure factual-gender preservation. The paper needs either a corrected interpretation of these results or a direct metric of factual-gender retention (e.g., accuracy on held-out factually gendered sentences) to support the MT half of the dual-debiasing claim.","section":"Section 5.1, Table 4"},{"comment":"The practical Dual Debiasing algorithm is not covered by the theorem that precedes it. The authors state that the conditional-independence assumption (Zb ⊥ Zf | X) is 'strong and unlikely to hold' and that they 'abandon theoretical guarantees' in the actual algorithm, replacing the exact projection with a thresholded version that uses an empirically chosen t. Consequently, the introduction's promise of 'strong theoretical backing' for 2DAMA is not fulfilled for the method as implemented. The paper should either provide an approximation bound or a robustness analysis showing that the thresholded projection still approximately preserves Zf under near-conditional-independence, or explicitly reposition Theorem 3 as motivational rather than as a guarantee for the evaluated method. As it stands, the central dual-debiasing claim for the practical algorithm rests entirely on the empirical evaluation, which has the ΔG problems noted above.","section":"Section 2.4, Theorem 3"},{"comment":"The sign convention for the factual-gender coefficient af is inconsistent and obscures the factual-preservation claim. Table 1 places a downward arrow on af, implying that lower values are better, yet Section 4.1 interprets a higher af (e.g., 2DAMA's 0.042 vs. DAMA+LEACE's 0.028 for Llama 2 7B) as preserving factual gender information. The same ambiguity applies to the arrow on the intercept b. The authors should clarify the direction of all metrics in the table and, for af, report a measure that explicitly reflects preservation of factual gender rather than a generic regression coefficient that could be lower for other reasons.","section":"Section 3.3, Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the comparisons. Several key differences are small (e.g., Table 2: perplexity 21.51 vs. 21.28; COMET/chrf changes under 1 point in Table 4), so the repeated conclusion that general performance is 'not significantly affected' is not statistically supported. At minimum, the authors should report variance across multiple evaluation runs or seeds, or use a resampling-based test for the headline bias reductions, so that readers can assess whether the observed improvements are distinguishable from noise.","section":"Tables 1, 2, and 4"}],"minor_comments":[{"comment":"The abstract and the conclusion contain the misspelling 'Dabiasing'; change to 'Debiasing' throughout. Additional typos include 'Moonrose-Penrose' (Appendix A.1), 'Erarsuer' (Section 2.2 heading), 'Seterotypical' (Section 4.2 heading), 'psuedoinverse' (Appendix A.1), and 'Germna' (Figure 5 caption).","section":"Abstract and Section 7"},{"comment":"The condition (B) in Theorem 3 is malformed: it is written as 'ΣX,ZbΣT X,Zf' without an equality or a stated property. The proof relies on the two covariance spaces being orthogonal, so the statement should be corrected, e.g., 'ΣX,Zb ΣT_{X,Zf} = 0'.","section":"Section 2.4, Theorem 3 statement"},{"comment":"The statement of Theorem 3 also contains a grammatical repetition: 'Zb and Zf Zb ⊥ Zf |X'. Please rephrase to 'Zb is conditionally independent of Zf given X.'","section":"Section 2.4, Theorem 3 statement"},{"comment":"The claim of being 'the first to propose a method for debiasing LLM in machine translation tasks' should be qualified in light of prior work on debiasing neural MT (e.g., Saunders and Byrne, 2020; Zmigrod et al., 2019), even if those works are not LLM-based. The phrasing should clarify the novelty relative to those lines of work.","section":"Section 6.2"},{"comment":"The description of the linear model (Eq. 2) says 'The slope coefficient tells how shifts in annotated scores across professions impact the difference in prediction probabilities.' This is correct, but the table header should specify whether the reported coefficients are on the original scale or multiplied (as in Figure 4, where they are 100·as and 100·af).","section":"Section 3.3"},{"comment":"The table reports both WinoMT and BUG columns, but the text mentions only ΔS and ΔG; it would be helpful to point out that the BUG results are also part of the evaluation and to note that BUG and WinoMT use different sentence distributions.","section":"Section 5.1 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's strongest contribution is the LM-side analysis: the layer-wise covariance visualization and the threshold-based preservation of factual gender are interesting, and the English experiments are reasonably convincing. However, the MT half of the central claim needs to be either repaired with a valid factual-preservation metric or substantially softened. The theoretical section is also overpromising: Theorem 3 does not cover the practical algorithm, and the authors admit the required condition is unrealistic. If the authors can add a direct factual-gender evaluation in translation and reframe the theory appropriately, the paper could be a solid contribution to the debiasing literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the English LM experiments mostly hold up as an incremental but useful extension of DAMA/LEACE. The new projection with threshold t is a genuine modification, the layer-wise covariance analysis is informative, and applying model-editing to MT debiasing is new. But the translation half of the central claim is not supported by the reported numbers. The paper treats ΔG as if lower is always better. Under the standard signed definition, German 2DAMA at t=1.00 moves ΔG from 3.7 to -10.1: that is a reversal of bias, not a reduction. Under an absolute-value reading it is worse than baseline. Czech goes from 2.1 to 7.2 even at t=1.00. Section 5.1 reads these as preserving factual gender and equitable representation, but ΔG measures accuracy difference between masculine and feminine entity translations, not factual-signal retention. So the distinctive 'dual' claim for machine translation is unsupported as reported. This is the load-bearing flaw, and it is fixable by reporting directional components or using a proper factual-preservation metric.\n\nThe theoretical backing also has a known soft spot: the Dual Debiasing theorem assumes conditional independence and orthogonal subspaces, which the authors admit are unrealistic; the practical algorithm is a heuristic with an empirically chosen threshold. That is acceptable if framed as such, but the paper should not lean on the theorem for the practical method.\n\nWhat the paper does well: LM bias reductions are consistent across models; the perplexity/ARC trade-off is modest; the analysis of which dimensions carry stereotype vs. factual signal is useful; the cross-lingual transfer experiments (Table 5) are a nice addition. No fabricated results on the LM side.\n\nFor peer review: I would send it out. The method is useful enough and the flaws are correctable. The authors need to fix the ΔG interpretation and either add error bars or hedge the claims. A referee could turn this into a solid paper. I would not cite the MT claim until corrected; the LM part is citable.","headline":"The LM side of 2DAMA is a credible incremental contribution, but the translation results are misread: the negative ΔG values are bias reversals, not factual-gender preservation.","tokens_in":21702,"tokens_out":2271,"would_cite":true,"duration_ms":22655,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"2DAMA is a post-hoc tuning method that removes stereotypical gender bias from language models and machine translation while preserving factual gender cues and most general performance.","keywords":["gender bias","debiasing","large language models","machine translation","model editing","concept erasure","factual gender","fairness"],"falsifier":"Evaluate a debiased model on a held-out set of professions and on naturalistic sentences not used to build the projection; if the stereotypical coefficient stays close to the original level while the factual coefficient drops, the dual-debiasing claim would be contradicted.","tokens_in":20664,"feed_emoji":"⚖️","tokens_out":7789,"duration_ms":70951,"temperature":0.7,"pith_summary":"The paper introduces 2DAMA, a post-hoc adaptation method for pre-trained language models that erases stereotypical gender associations while deliberately keeping factual gender information. The authors argue that debiasing should not remove all gender signals, because pronouns and grammatical gender in morphologically rich languages are legitimate factual cues. They combine the DAMA model-editing approach with the closed-form LEACE concept erasure, then add a covariance-based threshold to nullify only directions that carry stereotype information. Experiments on Llama 2, Llama 3, and the ALMA-R translation model show reduced gender bias in English language modeling and in translation to German, Czech, and Russian, with modest impact on general quality. The paper claims this is among the first methods to debias machine translation while preserving factual gender.","feed_headline":"2DAMA removes gender stereotypes, keeps factual gender","feed_subtitle":"A covariance-based edit targets biased directions in transformer layers, preserving factual gender across languages.","key_machinery":"The central machinery is a linear projection applied to the weights of transformer feed-forward layers. The paper assumes the layer acts as a least-squares map $V = SU - \\epsilon$, so the optimal debiasing matrix is $P^* = (I - W^\\dagger P_{W\\Sigma} W) S$, where $W$ whitens the key-value covariance and $P_{W\\Sigma}$ projects onto the covariance direction of the bias signal. Dual Debiasing modifies the projection to keep, rather than nullify, directions whose covariance with factual gender exceeds a bias-to-feature threshold $t$. The authors apply this projection to roughly one-third of the model's mid-upper feed-forward layers.","core_discovery":"The paper claims that a modified LEACE projection applied to transformer feed-forward layer weights, guided by a bias-to-feature threshold, can suppress stereotypical gender associations while retaining factual gender information. Theoretically, it extends LEACE to linear transformations and proves that under conditional independence of bias and feature signals given the latent representation, the projection leaves factual gender covariance intact; in practice, when signals are correlated, it uses a threshold $t$ to keep dimensions whose covariance with factual gender is $t$ times larger than with bias. Empirically, with $t=0.05$, the method erases about 99% of stereotypical covariance while retaining over 30% of factual-gender covariance in each edited layer. The authors also find that factual gender is concentrated in a few high-covariance dimensions, whereas stereotype signal is spread across more dimensions with lower covariance.","pith_inferences":["Editorial inference: If stereotype and factual gender occupy largely orthogonal linear subspaces in feed-forward layers, the same covariance-threshold trick could be applied to other protected attributes where a legitimate factual component exists, such as age or nationality.","Editorial inference: The cross-lingual results suggest that stereotype direction learned in one language transfers to related target languages; a testable next step is to measure whether prompt-language similarity predicts transfer strength.","Editorial inference: The paper evaluates on profession prompts and coreference benchmarks; a natural stress test would be long-form generation or naturalistic text, where factual gender preservation might interact with narrative context."],"forward_implications":["2DAMA reduces stereotypical gender bias in English language modeling on Llama 2 and Llama 3 while keeping perplexity and ARC performance close to the original.","It reduces stereotypical bias in English-to-German, English-to-Czech, and English-to-Russian translation with only small drops in COMET and chrF quality.","With a low bias-to-feature threshold, 2DAMA erases most of the stereotype covariance while keeping a substantial share of factual-gender covariance in edited layers.","Cross-lingual tests show that prompts in one language can reduce stereotypical bias in another (e.g., German prompts help Czech), while representational bias is better controlled with in-language prompts.","2DAMA does not require a preset dimensionality for bias erasure; the LEACE-based projection learns the extent of bias signals implicitly."],"supporting_citations":[{"why":"Supplies the DAMA model-adaptation framework, the prompt-based key/value extraction, and the baseline that 2DAMA streamlines.","marker":"(Limisiewicz et al., 2024)"},{"why":"Provides the closed-form LEACE concept-erasure projection used both in DAMA-LEACE and in the Dual Debiasing projection.","marker":"(Belrose et al., 2023)"},{"why":"Provides the 219 professions with stereotype and factual gender scores used to build debiasing prompts and the linear evaluation coefficients.","marker":"(Bolukbasi et al., 2016)"},{"why":"Defines the WinoBias coreference benchmark and the delta-G/delta-S bias metrics used for English evaluation.","marker":"(Zhao et al., 2018)"},{"why":"Defines the WinoMT translation bias benchmark and the delta-G/delta-S metrics used for German, Czech, and Russian evaluation.","marker":"(Stanovsky et al., 2019)"},{"why":"Provides the Llama 2 base models on which the English debiasing experiments are run.","marker":"(Touvron et al., 2023)"},{"why":"Provides the Llama 3 model used in English debiasing experiments.","marker":"(Dubey et al., 2024)"},{"why":"Provides the ALMA-R translation model used in multilingual debiasing experiments.","marker":"(Xu et al., 2024)"}],"fun_headline_variants":["2DAMA: erase stereotypes, keep gender facts","2DAMA cuts stereotypes, preserves factual gender","Debias with 2DAMA: drop stereotypes, keep factual gender","Erase bias, keep gender facts: 2DAMA method","2DAMA: target bias, spare factual gender"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a transformer feed-forward layer behaves like a linear least-squares map from inputs to outputs, so a linear projection computed from prompt-derived vectors can erase bias in that layer and transfer to unseen text.","fun_headline_variants_meta":{"raw":{"variants":["2DAMA: erase stereotypes, keep gender facts","2DAMA cuts stereotypes, preserves factual gender","Debias with 2DAMA: drop stereotypes, keep factual gender","Erase bias, keep gender facts: 2DAMA method","2DAMA: target bias, spare factual gender"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2813,"prompt_tokens":851,"completion_tokens":1962,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":1880}},"tokens_in":467,"tokens_out":1962,"duration_ms":13714,"temperature":1.0,"reasoning_tokens":1880,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:23:21.123191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate a debiased model on a held-out set of professions and on naturalistic sentences not used to build the projection; if the stereotypical coefficient stays close to the original level while the factual coefficient drops, the dual-debiasing claim would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the closed-form LEACE concept-erasure projection used both in DAMA-LEACE and in the Dual Debiasing projection."},{"cited_title":"Zou, Venkatesh Saligrama, and Adam Tauman Kalai","cited_arxiv_id":null,"evidence_quote":"Provides the 219 professions with stereotype and factual gender scores used to build debiasing prompts and the linear evaluation coefficients."}],"review_version":1}