{"id":"5a0dc662-9341-44ed-96b4-8a3500808fd9","arxiv_id":"2504.21400","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Six open-source LLMs show large, model-dependent gender biases in interview callback recommendations on 332,044 real job ads, with women often recommended for lower-paid, stereotypically female occupations.","lead":"Researchers audited six open-source AI models using 332,044 real job postings from India to see if the models recommend a man or a woman for an interview when the candidates are equally qualified. They found large gender imbalances in the models' callback recommendations, with women steered toward lower-wage and stereotypically female jobs, and showed that changing the AI's simulated personality changes the bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt-order confound: reversing the Mr./Ms. order flips female callback rates by ~98pp for Ministral and shifts most models by >10pp, so Table 1's 1.4–87.3% range may be measuring positional bias, not gender bias.","rationale":"The reader's conditional verdict centers on SOC mapping, wording of the abstract, uncertainty quantification, and the RLHF mechanism. Those are legitimate, but the more fundamental threat is in Section 4.1: the paper itself shows the primary outcome measure is not stable to prompt order for most models. Ministral's female callback rate moves from 1.39% to 99.86% when the order of 'Mr. X' and 'Ms. X' is reversed, which indicates near-complete positional preference rather than a gender preference. Since the headline claim of 'large and model-dependent gender bias' is built on single-order female callback rates, the central evidence does not currently identify gender bias. A counterbalanced re-analysis could restore the claim for at least Llama-3.1, whose rate is stable, and possibly for other models after averaging, but that re-analysis is not present. The findings for Llama-3.1's linguistic and personality analyses may survive, but the cross-model headline and the model-comparison tables should not be accepted as they stand.","tokens_in":30158,"tokens_out":7545,"duration_ms":81951,"concrete_test":"Re-run the audit with prompt order counterbalanced at the observation level: for each model and job posting, query both 'Mr. X or Ms. X' and 'Ms. X or Mr. X' (same decoding settings), and report per-order female callback rates, the order-effect size delta, and the order-averaged FCR per model. If any model has |delta| > 0.20 or if the two order conditions fall on opposite sides of 0.50, that model's FCR does not identify gender bias. Recompute Table 1, dissimilarity indices, and wage gaps using order-averaged recommendations; if the 1.4–87.3% range and 'most models favor men' disappear or materially change, the current headline is an artifact.","verdict_should_be":"REJECT","load_bearing_attack":"Section 4.1 reports the paper's own order-reversal check: switching Prompt A.1 from 'Mr. X or Ms. X' to 'Ms. X or Mr. X' moves the female callback rate from 1.39% to 99.86% for Ministral, from 73.24% to 99.46% for Llama-3, from 87.33% to 99.17% for Gemma, and from 61.33% to 79.64% for Granite. These swings are not small perturbations; they show that the outcome variable is, for most models, dominated by the position of the gendered label in the prompt rather than by a stable preference between equally qualified candidates. The headline range (1.4% to 87.3%) and the abstract's 'most models tend to favor men' are therefore not robust estimates of gender bias. The stability of Llama-3.1 (41.02% vs. 41.13%) is good for the in-depth analyses, but it does not validate the cross-model comparison in Table 1. Because female callback rates are the input to the dissimilarity index and wage-gap estimates, the order confound propagates into the occupational-segregation and posted-wage findings as well.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper audits six mid-sized open-source LLMs for gender bias in hiring recommendations. For each of 332,044 real job postings from India's National Career Services portal, the authors prompt each model to choose between two equally qualified candidates, Mr. X and Ms. X, and measure the female callback rate (FCR). They report large cross-model variation (FCR from 1.4% to 87.3%), compute a dissimilarity index of occupational segregation, estimate posted-wage gaps between jobs recommended to women versus men, analyze linguistic correlates of recommendations, and probe how Big Five personality infusions and historical-figure personas change model behavior. The central claim is that most models favor men, especially for higher-wage roles, and that recommendations reproduce occupational gender segregation.","tokens_in":30308,"tokens_out":6175,"duration_ms":57992,"significance":"The scale of the audit (over 40 million LLM queries on real job advertisements) and the direct elicitation of gendered recommendations are notable strengths, moving beyond small benchmark studies. If the headline results were robust, this would be an important contribution to the algorithmic-bias and labor-economics literatures. However, the lack of validation of the response parser and the SOC mapping, and especially the extreme sensitivity of the headline numbers to prompt order, mean that the cross-model claims are not currently supported.","major_comments":[{"comment":"The order-reversal robustness check reported in the text shows that reversing the order of 'Mr. X' and 'Ms. X' in Prompt A.1 changes the female callback rate by 18 to 98 percentage points for four of the six models (Ministral: 1.39% to 99.86%; Llama-3: 73.24% to 99.46%; Gemma: 87.33% to 99.17%; Granite: 61.33% to 79.64%). This demonstrates that for most models the outcome variable is dominated by the position of the gendered label rather than by a stable preference between equally qualified candidates, so the headline range of 1.4% to 87.3% in Table 1 and the abstract's claim that 'most models tend to favor men' are not supported. The abstract is also internally inconsistent with Table 1, where three of the six models recommend women in 61% to 87% of cases. The cross-model comparisons of the dissimilarity index and wage gaps in Table 1 inherit this non-robustness because they are computed from the same order-sensitive callback labels.","section":"Section 4.1 and Table 1"},{"comment":"The mapping of job postings to 2018 SOC occupations is performed by taking the nearest O*NET occupation summary under cosine similarity of all-mpnet-base-v2 sentence embeddings, but no validation of mapping accuracy is reported. Because the dissimilarity index (Section 3.2) and the wage-gap regressions with occupation fixed effects (Section 3.3) depend entirely on this assignment, a misassignment rate of even a few percent could materially distort the occupational-segregation results and occupation-level wage comparisons. Please provide a human-annotated validation sample, report precision/recall of the mapping, and show robustness to alternative mapping methods (e.g., keyword matching or LLM-based classification).","section":"Section 3.2"},{"comment":"The parsing rule that classifies a response as male or female based on the presence of 'Mr.' or 'Ms.' is not validated. Models may answer with phrases such as 'the male candidate', 'candidate X', or 'Mr. Smith', which would be misclassified or counted as abstentions, and the abstention rate under the reversed-order prompt is not reported. Please validate the parser on a random sample of outputs (e.g., 200 responses per model) and report the classification accuracy and abstention disagreement rates.","section":"Section 3.1"},{"comment":"The paper itself cautions in Section 5 that 'simply looking at the output tokens across models without considering the token probabilities might be misleading,' yet Table 1 and the abstract present raw token-based female callback rates as the headline result. The thresholding exercise shows that the rank ordering of models by dissimilarity index changes materially when token probabilities are used (e.g., Granite and Qwen become the least segregated), so the raw-output presentation in Table 1 is not a reliable summary of model behavior. Please either present the probability-thresholded results as the main analysis or clearly state that Table 1 reflects a biased prompt-and-parsing protocol.","section":"Section 5 and Table 1"}],"minor_comments":[{"comment":"The sentence 'most models tend to favor men' is contradicted by Table 1, where Gemma, Llama-3, and Granite favor women; please revise the abstract and introduction to be consistent with the results.","section":"Abstract"},{"comment":"The dependent variable is written as ln(wage_ijst) and the treatment as Fcallback_ijst, but notation elsewhere uses Fcallback_{p,ijst}; please standardize the subscripts.","section":"Equation 3.1"},{"comment":"Figure 2 reports a correlation of 83.95% between female callback rate and explicit requests for women at the 2-digit level, but the figure itself is not shown in the text; please add a pointer to the figure and describe the direction of the correlation.","section":"Section 4.3"},{"comment":"The manuscript cites several working papers and unpublished manuscripts (e.g., Bafna et al. 2025, Chaturvedi et al. 2024b); please include DOIs or stable URLs where available, and clarify which parts of the analysis rely on data or code from the authors' prior work.","section":"References"},{"comment":"The text states that Margaret Sanger appears both as a women's rights advocate with high female callback rates and as a controversial figure with high refusal rates (47.68%); please reconcile these two descriptions to avoid confusion.","section":"Section 7.2"}],"recommendation":"major_revision","confidential_remarks":"The order-reversal results in Section 4.1 are the most serious issue and should be the focus of the revision. The authors are advised to rerun the audit with randomized label order, or to reframe the paper around the stable Llama-3.1 results only. If the cross-model results are retained, the abstract and Table 1 must be revised to reflect the order sensitivity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the scale is real: 332k real job postings, six open-source LLMs, 40M queries, a direct head-to-head elicitation of male vs. female callback recommendations. That is a substantial empirical contribution, well beyond the small resume audits in this literature. Second, the paper's own order-reversal check (Section 4.1) shows that reversing \"Mr. X or Ms. X\" to \"Ms. X or Mr. X\" swings the female callback rate from 1.4% to 99.9% for Ministral, and by double digits for Llama-3, Gemma, and Granite. Only Llama-3.1 is stable (41.02% vs. 41.13%). The abstract's \"most models tend to favor men\" and Table 1's 1.4%–87.3% range therefore mostly measure positional bias, not gender preference. The authors see this and wisely restrict the in-depth language, occupation, and persona analyses to Llama-3.1, but they never correct the headline.\n\nWhat is genuinely good: the direct preference elicitation avoids the name-based confounds of prior audits; the occupation-level segregation and wage-gap analysis, done on a stable model, is a useful template; and the lexical and Big Five/persona results are internally consistent and suggestive. The correlation between Llama-3.1's callback rates and employers' explicit gender requests (84% at the 2-digit level) is a nice convergent check. The paper ships no code or data, but the design is reproducible in principle.\n\nSoft spots, in order of severity. The prompt-order confound is load-bearing for the headline; it should be front and center, not a robustness footnote. The SOC mapping via embedding cosine similarity is unvalidated, and the occupation-level results rest entirely on it. The parsing rule (presence of \"Mr.\" or \"Ms.\") is crude and unvalidated, though the probability thresholding in Section 5 partly mitigates this. The agreeableness-RLHF mechanism in the conclusion is asserted, not tested. Also, none of the headline rates come with uncertainty intervals or sampling details.\n\nWho benefits: labor economists studying algorithmic hiring, AI fairness researchers, and anyone deploying open-source LLMs for shortlisting. The paper deserves a serious referee: the core phenomenon is plausible and corroborated by prior work, the data effort is real, and the Llama-3.1 analyses are worth engaging. But I would send it back for major revision on the framing, with a clear request to validate the occupation mapping and to report order-robust results for every claim about cross-model differences.\n\nMy recommendation: accept for peer review; do not desk reject. The faults are reparable and the contribution, once properly scoped to the stable model, is meaningful.","headline":"A large and serious audit whose headline cross-model comparison is not robust: prompt-order effects flip female callback rates by huge margins for most models, and only Llama-3.1 is stable enough for the paper's own deeper analyses.","tokens_in":30958,"tokens_out":2196,"would_cite":true,"duration_ms":22234,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source LLMs show large, model-dependent gender bias in hiring callbacks.","keywords":["large language models","generative AI","algorithmic bias","gender discrimination","occupational segregation","resume screening","Big Five personality traits","job postings"],"falsifier":"Take a random sample of roughly 1,000 postings, have trained coders assign 2018 SOC codes by hand, and compare those labels to the embedding-based mapping; if the agreement rate is low or the mapping errors correlate with the words that drive female callbacks, the segregation and wage-gap estimates would not survive.","tokens_in":29850,"feed_emoji":"⚖️","tokens_out":9431,"duration_ms":93753,"temperature":0.7,"pith_summary":"This paper tries to establish that open-source large language models, when used to shortlist candidates, reproduce and amplify gender bias even when the two candidates are explicitly equally qualified. Across 332,044 real job postings, six models were asked to choose between \"Mr. X\" and \"Ms. X\" for an interview callback; the female callback rate runs from 1.4% (Ministral) to 87.3% (Gemma), showing that the bias is model-dependent rather than a fixed property of the task. Most models steer women toward lower-wage, stereotypically female occupations and men toward higher-wage ones, producing occupational segregation and a female wage penalty in recommended callbacks. The bias is tied to wording: job ads mentioning empathy, writing, or flexibility raise the chance of a female recommendation, while coding, hardware, finance, supervision, and night shifts lower it. If correct, the result matters because the same models are already entering real screening pipelines, and a fairness fix that works for one model or prompt does not transfer to others.","feed_headline":"Female callback rates span 1.4% to 87.3% across LLMs","feed_subtitle":"Six models split on who gets the interview, and most steer equally qualified women toward lower-wage jobs.","key_machinery":"The machinery is a head-to-head audit design. Each job posting is fed through a fixed prompt that asks the model to choose one of two equally qualified applicants, \"Mr. X\" or \"Ms. X\"; responses are parsed by string matching, and the female callback rate is $\\text{FCR}=N_{\\text{Ms.}}/(N_{\\text{Ms.}}+N_{\\text{Mr.}})$. To measure segregation, postings are embedded with a sentence transformer and matched to the nearest 2018 Standard Occupational Classification occupation by cosine similarity, then aggregated into the dissimilarity index $D = \\frac{1}{2}\\sum_o \\left|N_o^f/N^f - N_o^m/N^m\\right|$. To compare models on equal footing, the female-callback probability from the model's tokens is thresholded at values $\\rho\\in[0,1]$ so that callback parity (50% female) is imposed before segregation and wage gaps are recomputed. The same thresholding is used to relate Big Five trait infusion—ten persona prompts, high and low on each of openness, conscientiousness, extraversion, agreeableness, and emotional stability—and 99 historical-figure personas to segregation and wage disparity.","core_discovery":"The central discovery is that the gender of an equally qualified applicant changes whether an LLM grants a callback, and the direction and size of that effect depend on which model is asked. On the same set of job postings, female callback rates are 1.39% for Ministral, 17.30% for Qwen, 41.02% for Llama-3.1, 61.33% for Granite, 73.24% for Llama-3, and 87.33% for Gemma; the most balanced model also abstains most often, refusing a gendered choice on 5.88% of postings. Models sort applicants by occupation even after the aggregate imbalance is removed: the dissimilarity index across six-digit SOC occupations is 8.25% for Llama-3.1 but reaches 49.58% for Ministral, and at callback parity the index ranges from roughly 21% to 38%. Most models post a female wage penalty, with women recommended for jobs paying 4.1 to 22.7 log points less; two models show a female wage premium, but only because their female callback rates are extremely low or skewed. The paper also claims that the behavior is partly an agreeableness and compliance effect: models agree with explicit gender requests in 55–92% of cases (Cohen's $\\kappa$), and a deliberately less agreeable persona refuses on ethical grounds and reduces occupational segregation, while high openness and high conscientiousness increase it.","pith_inferences":["An implication the authors leave implicit: the same lexical attribution scores could be used pre-deployment to flag job advertisements whose wording predicts biased model callbacks, turning the audit into a text-screening tool.","Because the occupation mapping has no human validation, a natural extension is to benchmark embedding-based SOC assignment against manual coding; the paper's segregation and wage conclusions would be strengthened or weakened by that check.","If the agreeableness-bias explanation holds, model updates that change RLHF reward tuning could move callback rates without changing the prompt; a testable prediction is that \"helpful and harmless\" fine-tuned variants will show different segregation indices than base models.","The design compares one male and one female name only; extending the audit to multiple names, caste or religion cues, and intersectional identities would tell whether the gender gap is additive or interacts with other protected attributes."],"forward_implications":["If LLMs are used to shortlist real applicants, equally qualified men and women will receive systematically different callback rates, with the direction determined by the model choice.","Deployment without intervention would reproduce occupational segregation: women would be steered toward care, writing, and administrative roles while men are steered toward technical, financial, and supervisory roles.","Models comply with explicit gender preferences in job ads; since such requests are rare (about 2% of postings) yet followed at high rates, even small amounts of gendered ad text can bias the pipeline.","Calibrating a decision threshold to equalize aggregate callback rates is not enough: segregation remains at 21–38% at callback parity, so fairness requires occupation-level monitoring.","Persona steering changes callback rates by tens of percentage points; any audit result is therefore tied to the prompt, and \"debiased\" prompting must be validated on the target corpus."],"supporting_citations":[{"why":"Supplies the job-posting dataset and the 37 data-driven skill categories used in the skill analysis.","marker":"Chaturvedi et al. (2024a)"},{"why":"Provides the embedding-based occupation mapping approach used to assign postings to 2018 SOC codes.","marker":"Bafna et al. (2025)"},{"why":"Supplies the sentence-transformer embeddings used for the posting-to-occupation cosine similarity mapping.","marker":"Reimers and Gurevych (2019)"},{"why":"Supplies the Personality Prompting trait descriptions used to infuse Big Five traits into Llama-3.1.","marker":"Jiang et al. (2023)"},{"why":"Supplies the TIPI-style prompt format used to elicit the model's perceived Big Five scores for historical figures.","marker":"Cao and Kosinski (2024)"},{"why":"Supplies the LIWC-22 dictionary used to measure psycholinguistic features of job ads.","marker":"Pennebaker et al. (2022)"},{"why":"Documents the agreeableness and social-desirability bias in LLMs that the paper invokes to explain compliance behavior.","marker":"Salecha et al. (2024)"},{"why":"The classic callback-correspondence benchmark the paper positions its direct-elicitation audit against.","marker":"Bertrand and Mullainathan (2004)"},{"why":"Prior evidence of gender bias in job recommender systems that motivates the audit of modern LLMs.","marker":"Zhang and Kuhn (2024)"}],"fun_headline_variants":["LLM gender bias in callbacks swings with model choice","Women's callback rates differ 60x across six LLMs","Most AI hiring models prefer men for higher pay","Agreeableness traits steer LLM hiring bias","Callback gap: model choice, not merit, decides"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that embedding-based matching of each job posting to a U.S. occupation code is accurate enough that the measured occupation-level callback patterns actually reflect model behavior rather than misclassification.","fun_headline_variants_meta":{"raw":{"variants":["LLM gender bias in callbacks swings with model choice","Women's callback rates differ 60x across six LLMs","Most AI hiring models prefer men for higher pay","Agreeableness traits steer LLM hiring bias","Callback gap: model choice, not merit, decides"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000755,"raw_usage":{"total_tokens":3404,"prompt_tokens":1040,"completion_tokens":2364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":2297}},"tokens_in":656,"tokens_out":2364,"duration_ms":17256,"temperature":1.0,"reasoning_tokens":2297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:04:38.996230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of roughly 1,000 postings, have trained coders assign 2018 SOC codes by hand, and compare those labels to the embedding-based mapping; if the agreement rate is low or the mapping errors correlate with the words that drive female callbacks, the segregation and wage-gap estimates would not survive.","supporting_citations":[{"cited_title":"Chaturvedi, K","cited_arxiv_id":null,"evidence_quote":"Provides the embedding-based occupation mapping approach used to assign postings to 2018 SOC codes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sentence-transformer embeddings used for the posting-to-occupation cosine similarity mapping."},{"cited_title":"Xu, S.-C","cited_arxiv_id":null,"evidence_quote":"Supplies the Personality Prompting trait descriptions used to infuse Big Five traits into Llama-3.1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TIPI-style prompt format used to elicit the model's perceived Big Five scores for historical figures."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LIWC-22 dictionary used to measure psycholinguistic features of job ads."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the agreeableness and social-desirability bias in LLMs that the paper invokes to explain compliance behavior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The classic callback-correspondence benchmark the paper positions its direct-elicitation audit against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior evidence of gender bias in job recommender systems that motivates the audit of modern LLMs."}],"review_version":1}