{"id":"5bb0cf95-a78c-4076-b78c-0d00556231eb","arxiv_id":"2506.11880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A resume-scoring system built on BERT or RoBERTa reproduces artificially injected gender bias in training scores, and two gender-information removal methods reduce that bias on a synthetic recruitment dataset.","lead":"This paper tests whether language models used to score resumes learn gender bias from biased training scores, and whether removing gender cues from the text or from the model's internal representations prevents that bias. It is a concrete demonstration of bias mitigation in an automated hiring pipeline, relevant to anyone building or regulating AI recruitment tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Approach 1's reported bias reduction may be an artifact of evaluation leakage: the same validation split is used both to select gender-proxy tokens via Integrated Gradients and to compute the Table 2 fairness metrics, so the result is not an unbiased estimate on unseen resumes.","rationale":"The manuscript makes two claims it wants the reader to believe: (1) Transformer-based scoring tools learn synthetic gender bias from biased labels, and (2) both tested mitigation approaches remove that bias. The first is well supported by Table 1: both BERT and RoBERTa show large D_KL values and male-skewed shortlists when trained on y_G. The second is not equally secure. The reader's conditional verdict is appropriate, but the most load-bearing weakness is not the synthetic nature of FairCVdb per se, since a synthetic testbed is a legitimate proof-of-concept for a controlled bias-injection study. The decisive issue is the evaluation protocol for Approach 1: the same validation split is used both to choose which tokens are masked via Integrated Gradients and to compute the fairness metrics in Table 2. Fitting the token mask to the evaluation split is analogous to tuning hyperparameters on the test set; it gives an optimistic, potentially invalid estimate of the method's effect. This concern is concrete and testable, unlike a general uncertainty about real-world transfer. If the proposed test fails, the 'both methods' claim is unsupported and the paper should be revised to either report a held-out evaluation or restrict the claim to Approach 2. Approach 2's residual bias should also be acknowledged more carefully in the abstract, but it does not by itself undermine the weaker claim that bias was reduced. No machine-checked proofs or released code are present, so independent verification is the only route to confidence. I would keep the reader's CONDITIONAL verdict, making the condition explicit: Approach 1 must be re-evaluated with token selection and outcome measurement on disjoint splits.","tokens_in":14655,"tokens_out":7037,"duration_ms":148245,"concrete_test":"Re-run Approach 1 with a strictly disjoint protocol: compute Integrated Gradients attributions and select the gender-proxy token mask using only the training split (or an auxiliary development split), freeze that mask, retrain the scoring tool, and evaluate the same Table 2 metrics (D_KL, gender ratio, recall) on a test split never used for token selection or hyperparameter choices. If the D_KL/ratio improvements on the test split are much smaller than the roughly 0.02 values in Table 2, the reported Approach 1 result is leakage-driven. Also report results over at least 3 random seeds to quantify variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that both proposed methods 'successfully reduced gender information... preventing biased behaviors' rests on the evaluation protocol for Approach 1. In 'Approach 1: Via Model Explainability', the token mask is derived from the validation set: Integrated Gradients attributions are computed on validation resumes, the 20 most relevant tokens per resume are aggregated by gender and sector, and the top-30 tokens per group are replaced with [MASK] before retraining. All reported metrics in Table 2 (D_KL, gender proportions, ratio, recall) are then computed on the same validation set. This is evaluation leakage: the mask is fitted to the very split used to measure the outcome, so the reduction from D_KL=0.31/0.34 to roughly 0.02 may reflect overfitting to the evaluation split rather than a generalizable debiasing method. No separate test set is reported anywhere in the paper. Approach 2 (LNTL) does not share this leak, but it leaves measurable residual bias (ratio about 0.84, recall gap about 6.8 points), so the 'both methods' conclusion depends specifically on the validity of Approach 1's Table 2 numbers. If the leak is the cause of the improvement, the strongest version of the central claim collapses to a weaker statement about Approach 2 only.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses gender bias in Transformer-based resume scoring systems. It proposes and evaluates two bias mitigation methods on the FairCVdb synthetic dataset: Approach 1, which uses Integrated Gradients to identify gender-proxy tokens in biographies and masks them before retraining; and Approach 2, an adaptation of Learning-Not-to-Learn (LNTL) that adversarially removes gender information from a latent representation. Experiments with BERT and RoBERTa show that biased training labels lead to biased predictions, and that both proposed methods reduce gender imbalance in a top-500 shortlist according to demographic ratio, D_KL, and recall metrics. The paper concludes that both methods successfully reduced gender information and prevented biased behaviors despite biased training data.","tokens_in":14937,"tokens_out":2243,"duration_ms":23541,"significance":"If the results are valid, the paper offers a practical comparison of two debiasing strategies for a high-stakes NLP application, with the explicit goal of reducing allocational harm in AI-based recruitment. The work is positioned within an active and important research area, and the use of a public synthetic testbed (FairCVdb) makes the experiments reproducible in principle. Strengths include the evaluation across two transformer architectures, multiple fairness criteria (statistical parity, equality of opportunity, 4/5 rule), and a qualitative t-SNE analysis that connects latent geometry to the bias-removal mechanism. However, the validity of the central claim hinges on the integrity of the evaluation protocol, and the synthetic nature of the dataset limits external generalization. The paper also provides no statistical significance testing or error bars, so the quantitative differences between methods are not yet firmly established.","major_comments":[{"comment":"The evaluation protocol for Approach 1 suffers from a form of evaluation leakage: the same validation split is used both to select the gender-proxy tokens via Integrated Gradients and to compute the fairness metrics in Table 2. The text states that attributions are computed on the validation set, the top tokens aggregated by gender and sector, and the resulting mask is applied before retraining; all reported proportions, ratios, D_KL, and recall values in Table 2 are then measured on that same validation split. This can inflate the apparent debiasing because the mask is fitted to the very split used for evaluation. A held-out test set, or a nested/cross-validated protocol, is required to support the claim that Approach 1 'successfully reduced gender information' in a generalizable way. This issue is load-bearing because the strongest version of the central conclusion ('both methods successfully reduced gender information') depends on the Approach 1 numbers in Table 2.","section":"Approach 1: Via Model Explainability"},{"comment":"Approach 2 leaves measurable residual bias: the demographic ratio is 0.845 (BERT) and 0.831 (RoBERTa), D_KL is 0.0422 and 0.0487, and the recall gap between men and women is about 6.8 percentage points for BERT (86.4% vs 79.6%). The authors acknowledge in the text that 'gender information was not completely removed,' yet the Conclusions state that both methods 'successfully reduced gender information in the pipeline, thus preventing biased behaviors.' This overstates the evidence. The paper should either soften the conclusion to 'substantially reduced but not eliminated' or provide a formal statistical test showing that the residual gap is not significant. Without such a test, the claims about 'preventing biased behaviors' are not supported by the reported effect sizes.","section":"Table 2 / Approach 2: Via Adversarial Learning"},{"comment":"The entire evaluation rests on FairCVdb, a synthetic dataset of 24,000 profiles whose ground-truth scores are generated by a weighted linear combination of competencies and whose biased scores are obtained by artificially injecting gender-based bias. The paper provides no validation on real resumes or real annotation processes. If real hiring pipelines exhibit different bias mechanisms (e.g., bias in free-text descriptions, contextual cues, or label noise beyond a simple additive gender shift), the reported mitigation results may not transfer. The authors should discuss this limitation explicitly and, if possible, include at least a small real-world case study or a more realistic bias injection protocol to strengthen external validity.","section":"Materials and Methods: Dataset: FairCVdb"},{"comment":"All results in Tables 1 and 2 are point estimates with no variability measures (e.g., standard deviations or confidence intervals across random seeds). Since the comparison between Approach 1 and Approach 2 relies on small differences in D_KL (e.g., 0.0267 vs 0.0422 for BERT), and since training runs may vary with initialization, the paper should report repeated-run statistics or at least error bars. Without these, it is not possible to determine whether the observed differences are meaningful or within noise.","section":"Experimental Setup and Results / Table 2"}],"minor_comments":[{"comment":"There is a typo: 'they are are susceptible' should be 'they are susceptible.'","section":"Abstract"},{"comment":"The phrase 'prima facia evidence' should be 'prima facie evidence.'","section":"Fairness Criteria"},{"comment":"Several cross-references are left as empty 'Sect.' and 'Section' placeholders (e.g., in 'Experimental Setup and Results' and 'Approach 1'). These should be resolved to actual section numbers before publication.","section":"Throughout"},{"comment":"The sentence 'the tool clearly has some sense of it' is informal for a journal and would benefit from a more precise statement, e.g., 'the latent representations still contain some gender-discriminative information, as evidenced by the within-cluster separation in the t-SNE projection.'","section":"Approach 2: Via Adversarial Learning"},{"comment":"The conclusion that 'both methods successfully reduced gender information' should explicitly acknowledge the residual bias in Approach 2 and the evaluation-leakage risk in Approach 1, since these are directly relevant to the strength of the claims.","section":"Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a fairness/AI conference or journal, but the central quantitative claim is undermined by the evaluation protocol of Approach 1. The authors should be asked to rerun the experiments with a clean split for token-selection versus evaluation, and to temper the conclusions accordingly. The heavy reliance on the authors' own FairCVdb and related self-citations is acceptable given the line of work, but the synthetic dataset limitation should be openly discussed. I would not recommend rejection because the core idea is sound and the issues are fixable with a revised protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: this paper cleanly shows that BERT and RoBERTa resume scorers learn gender bias from biased training labels, and it tests two mitigation methods. The empirical setup is thoughtful, and the paper is honest about the weaker performance of the adversarial approach. The problem is that the stronger-looking result from the explainability approach is compromised by evaluation leakage, so the central claim needs qualifying.\n\nWhat's new and good: The paper adapts two established techniques—Integrated Gradients-based token masking and LNTL adversarial training—to a Transformer-based recruitment scoring pipeline on the FairCVdb testbed. The finding that both Transformers amplify biased scores into skewed shortlists is clearly supported. The token inspection is useful: words like 'children' and 'family' appear in the female sets but not male sets, consistent with known stereotype proxies. The t-SNE analysis of latent representations is a nice visual confirmation that bias affects the feature space and that LNTL partially restores sector structure. The paper also correctly notes that Approach 2 leaves residual bias, with ratios around 0.84 and D_KL around 0.04, and it doesn't oversell that part.\n\nThe soft spots: Approach 1 selects tokens via Integrated Gradients on the validation set and then reports the post-mitigation metrics on that same validation set. The paper never mentions a separate test set. So the sharp drop in D_KL from 0.31 to 0.02 may partly reflect the model being retrained with tokens that are informative about that specific validation split. That's a genuine flaw. Additionally, no error bars or multiple seeds are reported, so we can't tell if the differences between methods are stable. The synthetic nature of FairCVdb limits transferability, though the authors acknowledge this and use it as a controlled testbed.\n\nIs the central argument still standing? The basic claim that biased training scores lead to biased predictions is solid. But the strong version 'both methods successfully reduced gender information' depends on Approach 1's numbers. With the leak, the evidence for that claim weakens to Approach 2 alone, which shows partial mitigation. The paper should be revised to either use a separate validation set for token selection and a held-out test set for evaluation, or report both.\n\nWho is this for? Researchers working on fairness in automated hiring and NLP bias mitigation. It's a useful case study, but not a definitive method comparison. Given the clean problem setup and the clear documentation of the leak, I'd send it to peer review with a request for revision, not desk-reject it.","headline":"Useful case study with a real evaluation leak: Approach 1's token selection and evaluation share the same validation split, so the paper's strongest result is not trustworthy as reported.","tokens_in":15494,"tokens_out":2005,"would_cite":true,"duration_ms":24609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A privacy-enhancing framework that removes gender information from either the input text or the hidden representations can stop Transformer-based resume scorers from reproducing gender bias in their training data.","keywords":["gender bias","large language models","bias mitigation","automated recruitment","algorithmic fairness","Integrated Gradients","adversarial learning","FairCVdb"],"falsifier":"A direct falsifier would be to run both mitigation methods on a real resume corpus with naturally occurring gender-annotated hiring outcomes: if the demographic ratio of the shortlist stays below 0.8 or the score-distribution divergence stays above roughly 0.05 after either intervention, the claim that these methods prevent biased behaviour does not transfer beyond the synthetic testbed.","tokens_in":14454,"feed_emoji":"⚖️","tokens_out":8184,"duration_ms":83272,"temperature":0.7,"pith_summary":"This paper argues that Transformer-based resume scoring systems will learn gender bias whenever the training scores are biased by gender, and that the bias can be stopped by keeping gender information out of the learning pipeline. On the FairCVdb testbed of 24,000 synthetic profiles, BERT- and RoBERTa-based scorers trained on gender-biased scores produced shortlists that were about two-thirds male and fell below the U.S. 4/5 fairness rule. Two interventions — masking gender-linked words found by Integrated Gradients, and an adversarial learning-not-to-learn step that strips gender from hidden representations — both restored near-equal demographic ratios while improving overall recall. The result matters because biased shortlisting is an allocational harm: it withholds job opportunities from one demographic group, and the proposed framework prevents that even when the training labels themselves are biased.","feed_headline":"Two bias-removal methods make AI resume scoring fair despite biased data","feed_subtitle":"On 24,000 synthetic resumes, masking gender words or stripping gender from hidden layers restores balanced shortlists","key_machinery":"The load-bearing machinery is a two-stage pipeline — a frozen Transformer text encoder (BERT or RoBERTa) followed by a small MLP fusion head — plus two interventions that cut gender information from different stages. Approach 1 applies Integrated Gradients to attribute each token's contribution to the predicted score, collects the most relevant tokens per gender and labor sector, and masks gender-linked and stereotype-linked words before retraining. Approach 2 adapts learning-not-to-learn (LNTL): an auxiliary classifier tries to predict gender from a 300-dimensional hidden layer of the fusion head, and a min-max adversarial objective minimises the mutual information between that layer and gender while preserving score prediction. Both interventions target the channel that lets the model connect text to the biased target: the latent representation of the biography.","core_discovery":"The central discovery is that a recruitment model can be prevented from reproducing data bias even when the training labels are biased. When the scoring target is gender-biased, both BERT- and RoBERTa-based scorers encode gender in their latent text representations and use it to inflate male candidates' scores, dropping the shortlist demographic ratio to about 0.44-0.49. Removing gender information in the input space (replacing gender-linked tokens and names with [MASK]) brings the ratio above 0.96; removing it in latent space via adversarial training brings the ratio above 0.83 and yields the highest overall recall of all configurations. The adversarial method still leaves a residual gender signal visible in t-SNE projections, but the system satisfies the 4/5 fairness rule and no longer reproduces the biased labels.","pith_inferences":["A natural testable extension is to combine both methods: mask gender proxies first and apply LNTL afterward, which might remove the residual within-cluster gender separation seen in t-SNE and push the latent-space configuration closer to the near-perfect parity of Approach 1.","The same input-space and latent-space interventions should transfer to other sensitive attributes such as ethnicity (FairCVdb already includes ethnicity-biased scores) and to other high-stakes text tasks like loan or admissions screening, though each new context will need its own proxy-token discovery.","Because FairCVdb's biographies come from a public dataset of occupation descriptions, the list of proxy tokens (e.g., 'children', 'family', 'husband') is likely domain-specific; in real resumes the proxies may shift, so an automated token-discovery step is essential rather than a fixed word list.","The paper treats gender as binary because the dataset does; the method itself is attribute-agnostic, but applying it to non-binary or intersectional categories would require data and evaluation criteria the current testbed does not provide."],"forward_implications":["When trained on biased scores, both BERT- and RoBERTa-based scorers show demographic ratios around 0.44-0.49, below the 4/5 rule threshold, confirming that Transformer models exploit gender proxies in text.","Masking gender-linked tokens detected by Integrated Gradients brings the demographic ratio above 0.96 and the score-distribution divergence below 0.03, while improving overall recall over the biased baseline.","Adversarial removal of gender from the hidden layer satisfies the 4/5 rule with ratios above 0.83 and yields the highest recall values (83.0% for BERT, 80.0% for RoBERTa) of any configuration.","Neither method requires changing the pretrained language model; both operate on the input text or on the small fusion head, so they can be added to existing resume-scoring pipelines.","The biased baseline's latent embeddings cluster by gender rather than by labor sector; after adversarial training, labor-sector structure returns but a within-cluster gender distinction remains."],"supporting_citations":[{"why":"Introduces FairCVdb, the 24,000-profile multimodal dataset whose blind and gender-biased scores define the training and evaluation setup.","marker":"Peña et al. 2020a"},{"why":"Supplies the biography corpus used in FairCVdb and the earlier finding that gendered pronouns, names, and other tokens act as proxies for gender in occupation classification.","marker":"De-Arteaga et al. 2019"},{"why":"Provides Integrated Gradients, the attribution technique used in Approach 1 to detect gender-relevant tokens.","marker":"Sundararajan, Taly, and Yan 2017"},{"why":"Defines learning-not-to-learn, the adversarial method adapted in Approach 2 to remove gender information from latent representations.","marker":"Kim et al. 2019"},{"why":"Formalizes statistical parity and equality of opportunity, the fairness criteria used to evaluate shortlist behaviour.","marker":"Hardt, Price, and Srebro 2016"},{"why":"Supplies BERT, one of the two Transformer encoders used in the text understanding module.","marker":"Devlin et al. 2019"},{"why":"Supplies RoBERTa, the other Transformer encoder used in the text understanding module.","marker":"Liu et al. 2019"},{"why":"Defines the FairCVtest splits and prior fairness testbeds on FairCVdb used in the experiments.","marker":"Peña et al. 2023"}],"fun_headline_variants":["Masking gender words or adversarial training restores fair AI shortlists","Two bias-fighting tricks: mask gender words or retrain hidden layers","AI resume bias fixed by masking gender tokens or adversarial debiasing","LLM bias in hiring: two debiasing methods work despite biased labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic FairCVdb data, whose biased scores are created by artificially injecting gender into a weighted formula, behaves like real resumes with naturally occurring annotation bias.","fun_headline_variants_meta":{"raw":{"variants":["Masking gender words or adversarial training restores fair AI shortlists","Two bias-fighting tricks: mask gender words or retrain hidden layers","AI resume bias fixed by masking gender tokens or adversarial debiasing","LLM bias in hiring: two debiasing methods work despite biased labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000513,"raw_usage":{"total_tokens":2439,"prompt_tokens":834,"completion_tokens":1605,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":1539}},"tokens_in":450,"tokens_out":1605,"duration_ms":15264,"temperature":1.0,"reasoning_tokens":1539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T01:01:16.317145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier would be to run both mitigation methods on a real resume corpus with naturally occurring gender-annotated hiring outcomes: if the demographic ratio of the shortlist stays below 0.8 or the score-distribution divergence stays above roughly 0.05 after either intervention, the claim that these methods prevent biased behaviour does not transfer beyond the synthetic testbed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the biography corpus used in FairCVdb and the earlier finding that gendered pronouns, names, and other tokens act as proxies for gender in occupation classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Integrated Gradients, the attribution technique used in Approach 1 to detect gender-relevant tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines learning-not-to-learn, the adversarial method adapted in Approach 2 to remove gender information from latent representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formalizes statistical parity and equality of opportunity, the fairness criteria used to evaluate shortlist behaviour."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies BERT, one of the two Transformer encoders used in the text understanding module."}],"review_version":1}