{"id":"2a688ea6-a804-4b8f-ad35-638e2a68aa34","arxiv_id":"2412.02528","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A case study showing that AI admissions models can have persistent specificity and sensitivity bias by race and first-generation status, even when overall accuracy is high and common fairness metrics look satisfied.","lead":"This paper trains a linear SVM on six years of undergraduate admissions data from one university's School of Science and measures how prediction errors differ by gender, race, and first-generation status. It finds persistent differences in false admit and false reject rates for race and first-generation students that are invisible in overall accuracy, and it shows that switching to a test-optional policy would admit more women, non-white, and first-generation students.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The persistent-bias claim hinges on an arbitrary 5% cutoff and no uncertainty quantification; formal confidence intervals or split-exceedance rates are needed before the claim is fully supported.","rationale":"I read the paper as a transparent, policy-relevant case study. Its strengths are the three cohort analyses, the repeated random splits, and the explicit admission that the 5% threshold is subjective. My concern is not the use of a threshold per se; it is that the paper's main conclusion is expressed as a categorical claim of persistent bias, yet no statistical uncertainty is attached to the quantities that define it. The aggregate tables suggest the race effects are large and likely robust, but some gender and first-generation gaps sit near the 5% boundary. The reader's weakest assumption identified the same issue, and I agree. I therefore do not move the verdict: the paper is acceptable conditionally, on the authors adding uncertainty quantification or a formal threshold test, and on reconciling the Section 4.2 versus Section 4.3 gender statements.","tokens_in":12769,"tokens_out":7906,"duration_ms":84792,"concrete_test":"For each starred claim in Tables 7-9, save the per-split subgroup specificity/sensitivity differences across the 500 Monte Carlo runs. Compute the 2.5th and 97.5th percentiles (or a bootstrap CI) for each difference and the percentage of runs in which the absolute difference meets the 5% criterion. Also run a permutation test with subgroup labels shuffled within each split to obtain p-values for the null of equal subgroup errors. If any claimed bias has a 95% CI whose lower bound is below 5% (or includes zero) or fewer than, say, 95% of runs cross the 5% threshold, the persistence claim is not established and should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the SVM creates specificity/sensitivity bias that persists—is operationalized only as a >=5% difference in subgroup metrics (Section 3). The threshold is acknowledged as subjective, and Tables 7-9 report only mean values over 10/100/500 splits, not the distribution of those splits or any confidence interval or hypothesis test. Consequently, differences such as the 0.056 gender specificity gap for the larger test-optional model, or first-generation sensitivity gaps near 0.06, may be within sampling noise or may not reliably exceed the threshold even if the point estimate exceeds 5%. The issue is load-bearing because all three headline findings and the persistence conclusion are stated in terms of this threshold; if the threshold is moved or if uncertainty is accounted for, some starred findings may not survive. The paper deserves credit for repeated random splits and for disclosing the threshold, but the statistical support for 'bias' as defined is incomplete.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes bias in linear support vector machine models trained on six years of undergraduate admissions data from a School of Science at a large urban research university. The authors compare three cohorts (test-required, test-optional, and a larger simulated test-optional dataset) and evaluate model predictions for differences in accuracy, specificity, and sensitivity across gender, race/ethnicity, and first-generation status, using a 5% difference threshold to declare bias. They also report Brier scores and balance-based fairness metrics, and repeat random train/test splits to assess the persistence of observed biases. The main claims are that the models show specificity bias against white students in all three cohorts, sensitivity bias against non-white and first-generation students in the test-required and larger test-optional cohorts, and a gender specificity bias in the larger test-optional cohort that emerges only in the aggregate analysis.","tokens_in":12900,"tokens_out":7521,"duration_ms":66812,"significance":"The paper tackles a socially important and timely question, and it has several strengths: it uses a real institutional dataset, the authors are transparent about the subjective nature of the 5% threshold, and they make a genuine effort to assess stability via repeated random splits (up to 500 trials). The observed disparities in error patterns are policy-relevant and merit further investigation. However, the statistical support for the persistence claims is currently incomplete, and the internal inconsistency between Sections 4.2 and 4.3 weakens confidence in the presented results. If the authors add proper uncertainty quantification, resolve the contradiction, and clarify the evaluation protocol, the paper could be a useful contribution to the literature on bias in AI-based admissions.","major_comments":[{"comment":"The definition of bias as a difference of 5% or more in subgroup accuracy, specificity, or sensitivity is explicitly acknowledged as subjective, yet all headline findings are operationalized through this threshold. The aggregate results in Tables 7-9 report only mean values over 10, 100, and 500 random splits, with no standard deviations, confidence intervals, exceedance rates, or hypothesis tests. As a result, the claim that biases persist over up to 500 splits is not statistically supported; for example, the gender specificity difference in the larger test-optional model is 0.056-0.057, only slightly above the threshold, and it is possible that many individual splits fall below it. I recommend reporting the distribution of differences across splits, including the proportion of trials exceeding the threshold, and a sensitivity analysis showing how findings change for alternative thresholds.","section":"Section 3, Tables 7-9"},{"comment":"There is a direct contradiction between the single-split and aggregate analyses for gender. Section 4.2 and Table 4 state that \"None of the predictive models where Gender is the sensitive variable show bias,\" but Section 4.3 and Table 7 report a Specificity bias in the larger test-optional model, with differences of 0.057, 0.057, and 0.056 across 10, 100, and 500 trials. The paper must reconcile these statements; as written, the aggregate result invalidates the earlier claim, and the discrepancy is not discussed.","section":"Section 4.2 vs. Section 4.3, Tables 4 and 7"},{"comment":"The evaluation protocol is ambiguous. The text says that for each trial two-thirds of the data is randomly selected for training and that each model is validated using five-fold cross-validation, but it does not state whether the reported Specificity, Sensitivity, and fairness metrics are computed on the held-out one-third test set, on cross-validation folds, or on the training set. This is essential for interpreting Tables 4-9 and for replicating the analysis. Please clarify the exact computation of the reported metrics.","section":"Section 3 and Section 4.3"},{"comment":"The Methods section promises, for each of the three groups, three analyses (GPA included/test scores excluded, test scores included/GPA excluded, both included), each repeated for three sensitive variables. Section 4.1, however, reports only three scenarios: test-required with both GPA and test scores, test-optional with GPA only, and the full dataset with GPA only. The remaining six planned analyses are neither presented nor explicitly omitted with justification. This mismatch between the stated design and the reported results should be resolved.","section":"Section 3 and Section 4.1"},{"comment":"The claim that specificity/sensitivity bias not reflected in the fairness metrics \"underscores an important limitation of bias and fairness metrics\" is overstated. All metrics are evaluated using the same arbitrary 5% threshold, and the fairness metrics (Brier score, balance for positive/negative class) measure calibration rather than error-rate disparities, so differences in their behavior are expected. The discussion should be reframed as a comparison of threshold-based error rates with calibration-based metrics, rather than a general limitation of fairness metrics.","section":"Section 4.3, fairness metrics"}],"minor_comments":[{"comment":"Table 2 is difficult to read because the three variable-importance lists are interleaved in a single block; consider separating them into three panels.","section":"Table 2"},{"comment":"Section 4.2 uses the phrase \"significant differences\" in overall accuracies without any statistical testing; please either perform significance tests or use neutral wording such as \"observed differences.\"","section":"Section 4.2"},{"comment":"The paper does not report base rates or class proportions for the subgroups within each cohort, which would help contextualize the specificity/sensitivity differences in Tables 4-9.","section":"Section 3, Tables 4-6"},{"comment":"The definitions of Balance for the Negative Class and Balance for the Positive Class are informal; please provide formal mathematical definitions for reproducibility.","section":"Section 3"},{"comment":"The abstract's mention of \"various scenarios\" is broader than the three scenarios actually reported in Section 4.1; align the abstract and results.","section":"Abstract and Section 4.1"},{"comment":"Several references in the Related Work section are incomplete (the citation is shown as \"0\"); these should be fixed.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The contradiction between Sections 4.2 and 4.3 regarding gender bias is significant enough that I recommend the authors carefully audit their analysis pipeline, as it may indicate a bug or an inconsistent definition of bias across sections. The paper would also benefit from a statistical review to address the multiple comparisons and uncertainty quantification issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is the persistence check: running 500 stratified splits and showing that specificity/sensitivity gaps by race and first-gen status keep showing up is a reasonable way to argue the finding is not a single-split accident. The simulated larger test-optional dataset is a nice idea, and the paper is honest about its limits. The point that high overall accuracy can hide subgroup error differences, and that standard fairness metrics miss them, is worth making. Credit also for disclosing that data are not public.\n\nBut the paper's central claim is only as strong as the 5% threshold, which is arbitrary and never justified. You acknowledge it is subjective, but then you treat it as a hard line throughout. Worse, the aggregate tables report only means over trials. No standard deviations, no confidence intervals, no significance tests. So a difference of 0.056 in the gender specificity gap, or 0.099 in first-gen sensitivity, could be within sampling noise. The persistence claim needs either a distribution of split differences or a split-level exceedance rate, not just a mean.\n\nThere is also a real internal contradiction. Section 4.2 says no gender bias exists in any model. Section 4.3 says the larger test-optional model predicts women are incorrectly admitted more than men. The single split in Table 4 shows a 0.032 specificity gap, below threshold; the aggregate analysis shows 0.057, above. You need to explain that the aggregate supersedes the single split, or that the single split was atypical. As written, it looks like you changed your conclusion without saying so.\n\nThe larger test-optional dataset is a simulation because deleting test scores does not reproduce the applicant pool under a real test-optional policy. You note this correctly, but then you headline the simulated dataset's bias findings alongside the real cohorts. That is defensible if clearly separated, but it is a soft spot.\n\nOverall this is a solid case study that deserves referee time. It is not a methodological breakthrough, but the educational fairness angle is relevant, and the persistence analysis is a step up from the prior paper. The statistical gaps are fixable: add uncertainty quantification, run a sensitivity analysis on the threshold, and reconcile the gender contradiction. With those revisions it could be a useful contribution. I would engage with it in a reading group because it sparks good discussion about how to operationalize bias, and I would accept it for peer review despite the flaws. I would not cite it as a definitive statistical result until the persistence claim is backed by proper inference.","headline":"A transparent, credible case study that undermines itself with an arbitrary bias threshold, missing uncertainty quantification, and a direct contradiction between the single-split and aggregate gender results.","tokens_in":668,"tokens_out":802,"would_cite":false,"duration_ms":35176,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Models trained on six years of admissions data create persistent bias against non-white and first-generation students, incorrectly rejecting them more often despite high accuracy and passing fairness metrics.","keywords":["machine learning","bias detection","fairness metrics","predictive modeling","test-optional admissions","college admissions","algorithmic bias","higher education"],"falsifier":"Re-analyze the same data with statistical significance testing on the subgroup specificity and sensitivity differences across the 500 repeated splits, for example by computing confidence intervals or p-values for each gap. If the intervals for the reported race, gender, and first-generation gaps include zero, or the effect sizes are trivially small, then the claimed biases would not be established; conversely, tight intervals far from zero would confirm them.","tokens_in":12530,"feed_emoji":"🎓","tokens_out":9396,"duration_ms":84227,"temperature":0.7,"pith_summary":"This paper tries to show that machine-learning models trained on real undergraduate admissions data carry a hidden form of bias: they misclassify students in different sensitive groups at different rates, even when overall accuracy is high and standard fairness metrics look fine. Using six years of admissions records from a large urban university, the authors built linear support-vector-machine models predicting direct admission to the School of Science, then compared accuracy, specificity, and sensitivity across gender, race/ethnicity, and first-generation status. They report that all three models incorrectly admit white students more often than non-white students, and that models for the test-required cohort and the simulated larger test-optional dataset incorrectly reject non-white and first-generation students more often than their counterparts. These gaps exceed a 5 percent threshold and persist across up to 500 random train/test splits, while overall accuracy stays near 80 to 89 percent. The matter is worth caring about because test-optional admissions policies and AI-assisted admission decisions are spreading, and bias that is invisible to common fairness metrics could still harm students if such models are deployed.","feed_headline":"AI admission models mis-admit white students and mis-reject minorities","feed_subtitle":"Even high-accuracy models show persistent bias invisible to standard fairness metrics.","key_machinery":"The argument is carried by a linear support vector machine that classifies each applicant as a direct admit or not, using features such as GPA, standardized test scores, gender, race/ethnicity, first-generation status, residency, and campus-application count. Bias is declared when the difference in accuracy, specificity, or sensitivity between subgroups of a sensitive variable exceeds 5 percent, a threshold the authors state is chosen somewhat subjectively. To test persistence, each model is retrained over 10, 100, and 500 random two-thirds/one-thirds splits with proportionate stratified sampling on the sensitive variable, and mean subgroup metrics are compared. Three fairness metrics, Brier Score, Balance for the Negative Class, and Balance for the Positive Class, are computed alongside, and the paper uses the known incompatibility of fairness criteria to explain why these metrics mostly pass while specificity and sensitivity gaps fail.","core_discovery":"The central discovery is that the bias in the models is not a fluke of one train/test split and is not visible from overall accuracy or from the Brier Score and balance metrics. For race, every cohort model predicts white students will be incorrectly admitted more than non-white students, a specificity gap above 5 percent, and the test-required and larger test-optional models also predict non-white students will be incorrectly rejected more than white students, a sensitivity gap above 5 percent. For first-generation status, the same two models predict first-generation students will be incorrectly rejected more often and non-first-generation students incorrectly admitted more often than their counterparts. For gender, the larger test-optional model predicts women will be incorrectly admitted more than men. The paper treats these uneven error rates as evidence that the predictive models create new bias relative to sensitive variables, and it argues that fairness metrics can be simultaneously satisfied while such group-level error bias exists because the models are highly accurate.","pith_inferences":["Because the paper uses historical admission decisions as ground truth, the model's uneven error rates may partly reflect bias in the original human and policy decisions, not just algorithmic bias; the method cannot cleanly separate the two, so the word 'new bias' is stronger than what the design can prove.","A statistically grounded threshold, such as a 95 percent confidence interval that excludes zero, might change which of the reported gaps count as bias; the 5 percent rule is exploratory, and the headline list of biases should be read with that caveat.","The same subgroup-specific specificity and sensitivity audit could be applied to other high-stakes classifiers where accuracy is high and common fairness metrics look acceptable, such as hiring screens, loan approval, or clinical triage tools.","A testable extension would be to retrain the models with reweighted or resampled training data designed to equalize specificity and sensitivity across subgroups and then measure how much overall accuracy must be sacrificed to remove the reported gaps."],"forward_implications":["If the central claim is right, a model can look fair by standard aggregate metrics yet still systematically mis-admit or mis-reject protected groups, so admissions AI should be audited for subgroup specificity and sensitivity, not just overall accuracy.","The finding that standardized test scores dominate the test-required model implies that making tests optional can change admission outcomes for a large number of students, including more women, non-white students, and first-generation students.","The persistence of the gaps over hundreds of random splits means the bias is not an artifact of a particular training set, so mitigation efforts must address the data or the model rather than the split.","The simultaneous satisfaction of all three fairness metrics in most conditions supports the theoretical point that one fairness definition can conceal another, and suggests fairness reports should include error-rate breakdowns.","The larger simulated test-optional dataset, while useful for evaluating models, should not be read as a prediction of what a true test-optional applicant pool would look like; the authors explicitly caution against that inference."],"supporting_citations":[{"why":"The prior case study this paper extends; it supplies the initial models, dataset pipeline, and the earlier finding of bias that motivated the more complete analysis.","marker":"[1]"},{"why":"Survey evidence that Black and Hispanic students are more likely to apply to test-optional schools, setting up the demographic-impact question.","marker":"[6]"},{"why":"Study of how test-optional policies change application behavior and enrolled demographics, providing a comparison point for the paper's own demographic results.","marker":"[7]"},{"why":"Study of test-optional policies at selective liberal arts colleges, offering a contrasting result on diversity that the paper discusses.","marker":"[8]"},{"why":"Theoretical result that fairness definitions cannot all be satisfied simultaneously, which the paper uses to explain why the fairness metrics mostly pass while specificity and sensitivity gaps appear.","marker":"[25]"}],"fun_headline_variants":["AI admissions bias evades standard fairness metrics","Study: admission AI shows bias despite high accuracy","Group bias in AI admissions invisible to fairness metrics","Admission models carry bias standard fairness checks miss","AI admission bias persists even when fairness metrics pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire bias verdict rests on a 5 percent difference in error rates between subgroups, a cutoff the authors admit is chosen subjectively, and no statistical significance test is applied to determine whether the observed gaps could arise by chance.","fun_headline_variants_meta":{"raw":{"variants":["AI admissions bias evades standard fairness metrics","Study: admission AI shows bias despite high accuracy","Group bias in AI admissions invisible to fairness metrics","Admission models carry bias standard fairness checks miss","AI admission bias persists even when fairness metrics pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1390,"prompt_tokens":918,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":402}},"tokens_in":534,"tokens_out":472,"duration_ms":5689,"temperature":1.0,"reasoning_tokens":402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:20:24.759239+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-analyze the same data with statistical significance testing on the subgroup specificity and sensitivity differences across the 500 repeated splits, for example by computing confidence intervals or p-values for each gap. If the intervals for the reported race, gender, and first-generation gaps include zero, or the effect sizes are trivially small, then the claimed biases would not be established; conversely, tight intervals far from zero would confirm them.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Survey evidence that Black and Hispanic students are more likely to apply to test-optional schools, setting up the demographic-impact question."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Study of how test-optional policies change application behavior and enrolled demographics, providing a comparison point for the paper's own demographic results."},{"cited_title":"Belasco, Kelly O","cited_arxiv_id":null,"evidence_quote":"Study of test-optional policies at selective liberal arts colleges, offering a contrasting result on diversity that the paper discusses."}],"review_version":1}