Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

mFARM, a geometric mean of five fairness scores, catches clinical-LLM harms that parity and equalized odds overlook.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new multi-metric fairness framework for clinical LLMs, applied to two large MIMIC-IV-based benchmarks, shows that context scarcity hurts fairness more than quantization does.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Genuinely useful MIMIC-IV fairness benchmarks plus an open implementation, wrapped in a significance-gated composite score whose cross-task numbers shouldn't be trusted as comparable. the 4 major comments →

arxiv 2509.02007 v1 pith:NFLHX6XV submitted 2025-09-02 cs.AI

mFARM: Towards Multi-Faceted Fairness Assessment based on HARMs in Clinical Decision Support

Shreyash Adappanavar , Krithi Shailya , Gokul S Krishnan , Sriraam Natarajan , Balaraman Ravindran This is my paper
classification cs.AI
keywords fairness assessmentclinical decision supportlarge language modelsallocational harmstability harmlatent harmMIMIC-IVmodel alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that fairness in high-stakes medical AI cannot be read off a single number such as statistical parity or equalized odds, because a model can look fair on average while behaving badly for one demographic group. To make the case, it builds two large, controlled benchmarks from real hospital records—ED Triage and Opioid Analgesic Recommendation—in which the same clinical case is re-rendered with twelve race × gender descriptors across three context tiers, producing over 50,000 prompts. It then proposes mFARM, the geometric mean of five fairness scores targeting three distinct harms: allocational (mean favoritism), stability (variance and deviation from a neutral baseline), and latent (distribution shape and confidence-dependent bias). Across four 7–8 billion parameter open LLMs, the paper finds that mFARM holds up or improves under 8-bit and 4-bit quantization for most models, but collapses when clinical context is cut to the chief complaint and age. If the claim holds, mFARM gives deployment auditors a single non-compensatory score that surfaces exactly the biases parity-style metrics are blind to, and its companion FAB score prevents a model from scoring well by being fair only because it refuses to make real clinical decisions.

Core claim

A composite score from five sub-metrics—Mean Difference, Absolute Deviation, Variance Heterogeneity, KS Distributional, Correlation Difference—is claimed to catch harms that parity and equalized odds overlook in clinical LLMs. Each sub-metric runs an omnibus significance test and averages only significant effect sizes; fairness is one minus that average. Geometric-mean aggregation is non-compensatory: one zero sub-metric zeroes mFARM. Two failures parity misses: identical group averages but 200× the variance for one group; bias growing with confidence. Empirically, mFARM survives 4-bit quantization but collapses in low-context prompts (Qwen hits 0.000); fine-tuning raises accuracy and FAB.

What carries the argument

The load-bearing object is the mFARM score: the geometric mean of five [0,1]-valued fairness sub-metrics—Mean Difference, Absolute Deviation, Variance Heterogeneity, KS Distributional, and Correlation Difference—each aimed at a distinct harm. The shared mechanism inside each sub-metric is a three-stage pipeline: an omnibus test (Friedman, Levene, or a direct KS/correlation test) asks whether demographic groups differ; significant results (p ≤ 0.05) trigger Bonferroni-corrected post-hoc pairwise tests; the unfairness score averages only the significant effect sizes (Cliff's delta, normalized variance ratio, KS statistic, Spearman's ρ). Fairness is 1 − unfairness. The geometric mean makes the

Load-bearing premise

The mFARM score counts a disparity as real only if it passes a statistical significance test with a fixed p ≤ 0.05 cutoff, so the same underlying effect looks more unfair in a larger test set and less unfair in a smaller one; the scores across the two differently-sized tasks are not directly comparable.

What would settle it

Build a synthetic cohort where group mean predictions are identical but one group's prediction variance is a known multiple of the others', then check whether the Variance Heterogeneity sub-metric—and hence the mFARM composite—decreases monotonically with that multiple while statistical parity stays at exactly 1.0. If mFARM does not track the engineered magnitude, the claim that it 'captures subtle biases' fails. A second check: recompute mFARM on random halves of each test set; if scores swing by more than the reported model-to-model differences, the fixed-alpha gating, not the underlying eff

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Auditors who rely on statistical parity or equalized odds can certify a model that is behaviorally unstable for one demographic group; mFARM is designed to fail that model even when per-group average predictions match.
  • Quantization to 8-bit or 4-bit is not a fairness risk in these experiments: mFARM mostly holds or improves, with BioLlama's Opioid Analgesics score rising from 0.674 to 0.956, so memory-efficient deployment can coexist with fairness.
  • The dominant fairness risk is context scarcity: when prompts are cut to chief complaint and age, mFARM collapses, with Qwen's ED score falling from 0.690 to 0.000, so low-information triage settings are where bias is most likely to emerge.
  • LoRA fine-tuning removes the base models' single-answer default (accuracy skew near zero) and lifts FAB scores substantially—Mistral's Opioid Analgesics score goes from 0.585 to 0.875—making lightweight fine-tuning an alignment lever.
  • Because mFARM is a geometric mean and FAB a harmonic mean, fairness is treated as non-compensatory and is rewarded only alongside accuracy; a clinically inert model that is 'fair' by refusing to decide cannot earn a high deployability score.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The sub-metrics average effect sizes only over comparisons that pass a p ≤ 0.05 significance gate, so mFARM is sample-size dependent: the ED test set (N=1,020) can make small real disparities significant that the OA test set (N=273) cannot detect. As published, the two tasks' scores are not directly comparable; an effect-only variant or confidence intervals would fix this.
  • The omnibus → post-hoc → effect-size pipeline is not medicine-specific: applied to any LLM that outputs probabilities over a neutral baseline, the same five metrics could audit hiring, lending, or content-moderation models once the test-power issue is addressed.
  • The Correlation Difference sub-metric carries a testable behavioral prediction: these models are most biased precisely when they are most confident. Auditing high-confidence disagreements against clinician-adjudicated outcomes would validate or refute that mechanism.
  • The paper's own conclusion flags a limitation: in rare cases fine-tuning caused slight fairness degradation and marginal drops in deployability, so the 'fine-tuning preserves fairness' result is not universal across all runs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces mFARM, a composite fairness assessment framework for clinical decision-support LLMs. It defines five submetrics—Mean Difference, Absolute Deviation, Variance Heterogeneity, KS Distributional, and Correlation Difference—intended to capture allocational, stability, and latent harms, and aggregates them with a geometric mean. A Fairness-Accuracy Balance (FAB) score combines mFARM with accuracy via a harmonic mean. The authors construct two MIMIC-IV-derived benchmarks (ED-Triage and Opioid Analgesic Recommendation) with counterfactual demographic prompt variants and three context tiers, and evaluate four 7–8B LLMs under quantization, context reduction, and LoRA fine-tuning. The central claims are that mFARM surfaces subtle biases missed by traditional metrics, that quantization does not harm fairness, and that fine-tuning improves accuracy while maintaining fairness.

Significance. The benchmark construction is a concrete contribution: controlled counterfactual prompts, a public release of code, and a multi-metric evaluation template are valuable to the clinical LLM alignment community. The fairness scores are not circular: they are computed from model outputs by stated statistical conventions, and the geometric/harmonic aggregations are explicit choices. However, the paper's central empirical claims are currently undermined by the significance-gated construction of every submetric: scores become functions of sample size and test power, which affects cross-task comparability and explains the universal 1.0 Variance/KS results. If the authors report raw effect sizes and revise the aggregation and claims accordingly, the framework and benchmarks could be useful; as presented, the evidence for the headline claims is not yet solid.

major comments (4)
  1. [Fairness Score Calculation, Eq. (1)] The unfairness score U averages effect sizes only over comparisons with I(c)=1 (Bonferroni-significant), and a non-significant omnibus test sets the submetric to 1.0. This makes every mFARM submetric a function of statistical power, not just effect size. ED has N=1,020 test cases while OA has N=273 (Table 1); a moderate real disparity can therefore be penalized on ED but produce a perfect 1.0 on OA. The cross-task comparisons in Tables 5, 7, and 8 and the RQ4 robustness claims are consequently not directly interpretable. Please report raw effect sizes (and confidence intervals) for every comparison, and either replace the hard significance gate with a continuous aggregation (e.g., average effect size with uncertainty) or clearly restrict claims to within-task, same-N comparisons.
  2. [RQ2 / Table 6 / Table 11] The paper's central claim that mFARM 'captures subtle biases more effectively' is not supported by the Variance Heterogeneity and KS Distributional submetrics in the high-context setting: all models in Tables 6 and 11 receive exactly 1.00 on both metrics. The RQ2 text interprets this as 'models are highly stable,' but the construction of Eq. (1) means a non-significant omnibus test also yields 1.00 regardless of effect size. The illustrative advantage in Table 4 (RQ1) is therefore not empirically demonstrated in the evaluated benchmark. Please report the underlying variance ratios and KS statistics for all runs, not only the significance-gated scores, so readers can see whether the 1.00 values reflect true invariance or lack of power.
  3. [Tables 13–15 / Variance Heterogeneity methodology] For Variance Heterogeneity, the post-hoc set C contains 12 BASE-vs-group and 66 group-vs-group comparisons (78 total). With α=0.05 after Bonferroni correction, the per-comparison threshold is ≈0.00064. At OA's N=273, even a variance ratio R=2 (Evar=0.333) is unlikely to be flagged as significant, so the metric returns 1.0. The same power issue applies to the KS and Correlation Difference metrics. This is a load-bearing flaw because the framework's output is meant to be an audit of disparities; suggesting variance and KS are always perfect in high-context settings is not a meaningful fairness audit.
  4. [RQ3 / Table 5 / Table 11 / Conclusion] The contribution bullet claims LoRA fine-tuning 'boosts accuracy while maintaining fairness,' and the conclusion states that fairness degradation is 'rare.' However, Table 11 (and mostly Table 5) shows that fine-tuning decreases the mFARM score in 4 of 8 model-task pairs: ED Qwen (0.690→0.628), ED Mistral (0.916→0.675 in Table 11), OA BioLlama (0.674→0.672), and OA BioMistral (0.795→0.670). Some of these are substantial drops. The text should acknowledge this pattern and soften the 'rare instances' claim; otherwise the reported results contradict the narrative.
minor comments (5)
  1. [Table 5] The ED Mistral fairness row is inconsistent with Table 11: Table 5 lists base/FT fairness as 0.716/0.675, while Table 11 lists ED Mistral fairness as 0.916 (base) and 0.675 (FT). Please reconcile all reported numbers between tables.
  2. [Appendix: Composite Metric Definitions] Equation (3) has an incomplete sentence: 'where m and M are....' This should be completed or removed.
  3. [A Multi-faceted Fairness Framework] The phrase 'five statistically independent metrics' is imprecise. The metrics use overlapping data and tests; RQ2 only shows low empirical pairwise correlation, not statistical independence. Please rephrase.
  4. [Figure 13 / FAB terminology] Figure 13's caption refers to 'H-Score' while the text consistently uses 'FAB score.' Unify the terminology.
  5. [Throughout] The paper contains several typographical issues with the framework name ('mF arm', 'F AB') and inconsistent spacing in equations. Please proofread the camera-ready version.

Circularity Check

0 steps flagged

No significant circularity: mFARM metrics are defined directly from model outputs; no fitted parameter, self-citation chain, or definitional reduction supports the paper's claims.

full rationale

The paper's derivation chain is self-contained. The five mFARM submetrics are defined directly from model output probabilities via explicit statistical tests and effect sizes (Eq. 1 and Table 3), and the aggregate mFARM and FAB scores are stated conventions (geometric/harmonic means, Eqs. 3-5). No parameter is fitted to any subset of the data and then renamed as a prediction; no result is imported from the authors' prior work as load-bearing evidence; and no uniqueness theorem or ansatz is smuggled in via citation. The significance-gating in Eq. 1 does make the scores sensitive to sample size and statistical power, which is a measurement-validity concern rather than a circularity concern under the given criteria. The illustrative example in RQ1 merely demonstrates the behavior of the metric by construction, not an empirical prediction that reduces to its inputs. The low inter-metric correlations are an empirical observation about the metric values, not a circular derivation. Overall, the claimed findings are empirical evaluations of a proposed measurement framework, not a derivation that assumes what it purports to show.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claim rests on treating model output probabilities as comparable across prompt variants, on statistical significance as a valid gate for scoring harm, and on ground-truth labels (ESI, actual opioid prescription) as unbiased clinical-need proxies. The alpha=0.05 threshold is a free design choice that materially affects scores. No new physical or conceptual entities are introduced beyond the score definitions.

free parameters (1)
  • Significance threshold alpha = 0.05
    The fairness score is set to 1.0 if the omnibus test p > alpha, and effect sizes are averaged only over significant post-hoc comparisons. This choice directly affects mFARM values and is not derived from any principle; it interacts with sample size to make scores across tasks non-comparable.
axioms (4)
  • domain assumption Model output probability P_i is a meaningful measure of clinical decision tendency and is comparable across prompt variants.
    All five fairness metrics operate on these probabilities; if probabilities are not meaningful or are not comparable across the 13 variants, the entire framework collapses. Invoked throughout Section 'A Multi-faceted Fairness Framework'.
  • standard math Standard non-parametric statistical tests (Friedman, Levene, Wilcoxon, KS, Spearman) are valid for comparing LLM output distributions on paired cases.
    The omnibus and post-hoc procedures assume exchangeability and independence across cases; this is standard but unstated in detail. Invoked in the metric definitions.
  • domain assumption Ground-truth labels (ESI in ED Triage, actual opioid prescription in OA) are unbiased proxies for clinical need.
    Accuracy and FAB rely on these labels; if ESI assignment or prescription behavior encodes human bias, accuracy scores are contaminated. The paper does not discuss this confound. Invoked in the benchmark construction and accuracy computation.
  • domain assumption The BASE prompt with no demographic information is a neutral reference point.
    Absolute Deviation, KS, and Correlation Difference metrics compare each group to BASE; if the BASE prompt is not neutral (e.g., the model infers a 'default' demographic), the reference is biased. Invoked in the metric definitions.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of mFARM: Towards Multi-Faceted Fairness Assessment based on HARMs in Clinical Decision Support." pith.science (2026). https://pith.science/paper/NFLHX6XV

@misc{pith2026250902007,
  author       = {Pith},
  title        = {Pith review of: mFARM: Towards Multi-Faceted Fairness Assessment based on HARMs in Clinical Decision Support},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NFLHX6XV}},
  note         = {Machine review of arXiv:2509.02007}
}
Share X Bluesky LinkedIn Reddit HN
abstract

The deployment of Large Language Models (LLMs) in high-stakes medical settings poses a critical AI alignment challenge, as models can inherit and amplify societal biases, leading to significant disparities. Existing fairness evaluation methods fall short in these contexts as they typically use simplistic metrics that overlook the multi-dimensional nature of medical harms. This also promotes models that are fair only because they are clinically inert, defaulting to safe but potentially inaccurate outputs. To address this gap, our contributions are mainly two-fold: first, we construct two large-scale, controlled benchmarks (ED-Triage and Opioid Analgesic Recommendation) from MIMIC-IV, comprising over 50,000 prompts with twelve race x gender variants and three context tiers. Second, we propose a multi-metric framework - Multi-faceted Fairness Assessment based on hARMs ($mFARM$) to audit fairness for three distinct dimensions of disparity (Allocational, Stability, and Latent) and aggregate them into an $mFARM$ score. We also present an aggregated Fairness-Accuracy Balance (FAB) score to benchmark and observe trade-offs between fairness and prediction accuracy. We empirically evaluate four open-source LLMs (Mistral-7B, BioMistral-7B, Qwen-2.5-7B, Bio-LLaMA3-8B) and their finetuned versions under quantization and context variations. Our findings showcase that the proposed $mFARM$ metrics capture subtle biases more effectively under various settings. We find that most models maintain robust performance in terms of $mFARM$ score across varying levels of quantization but deteriorate significantly when the context is reduced. Our benchmarks and evaluation code are publicly released to enhance research in aligned AI for healthcare.

Figures

Figures reproduced from arXiv: 2509.02007 by Balaraman Ravindran, Gokul S Krishnan, Krithi Shailya, Shreyash Adappanavar, Sriraam Natarajan.

Figure 1
Figure 1. Figure 1: Overview of proposed fairness assessment with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Data preprocessing pipeline for fairness analysis [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: visualizes this positive-sum relationship. The ideal model would occupy the top-right corner, signify￾ing perfect accuracy and fairness. After fine-tuning (orange markers), every model moves to the right, indicating univer￾sal accuracy gains. For the ED Triage task (Figure 3a), the models’ vertical positions remain stable, showing that fair￾ness is preserved. For the OA task (Figure 3b), the models also ma… view at source ↗
Figure 4
Figure 4. Figure 4: FAB sensitivity to quantization in ED Triage. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example: Prediction Shift Due to Demographics (Qwen2.5-7B-Instruct) [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Counterfactual prompt design. The base prompt is [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt Template for ED Triage Task Inter-Metric Independence To validate that the five fairness metrics capture distinct phe￾nomena, we compute pairwise correlations across all mod￾els. As shown in the [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: Pairwise correlation heatmap of fairness sub [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy Skew of models before and after fine [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Five fairness metrics per model (high context, 16- [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Comparison of information density. The Medium-Context prompt is created by systematically re￾moving clinical fields from the High-Context version. (a) ED Triage (b) Opioid Analgesics [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: H-Score per model across context levels (Low, [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AgentFairBench: Do LLM Agents Discriminate When They Act?

    cs.AI 2026-06 unverdicted novelty 6.0

    AgentFairBench is a multi-domain benchmark for demographic disparity in LLM agent actions, with a pilot showing no significant effect for Claude Haiku 4.5 after arity-matched noise correction.

Reference graph

Works this paper leans on

43 extracted references · 35 canonical work pages · cited by 1 Pith paper · 2 internal anchors

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Barocas, S.; Hardt, M.; and Narayanan, A. 2017. Fairness and Machine Learning. Https://fairmlbook.org

    work page 2017

  4. [4]

    Black, E.; Grgić-Hlača, N.; Binns, R.; et al. 2022. Grounding Algorithmic Fairness in Lay Justice Norms. In FAccT

    work page 2022

  5. [5]

    Y.; Saligrama, V.; and Kalai, A

    Bolukbasi, T.; Chang, K.-W.; Zou, J. Y.; Saligrama, V.; and Kalai, A. T. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. arXiv preprint arXiv:1607.06520

    Pith/arXiv arXiv 2016

  6. [6]

    Bullen, P. S. 2003. Handbook of Means and Their Inequalities, volume 560 of Mathematics and Its Applications. New York: Springer

    work page 2003

  7. [7]

    H.; Zeng, Q.; and Yu, C.-H

    Chen, S.; Li, X.; Zhang, M.; Jiang, E. H.; Zeng, Q.; and Yu, C.-H. 2025. CARES : Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs . ArXiv:2505.11413 [cs]

    Pith/arXiv arXiv 2025

  8. [8]

    T.; and Li, X

    Chen, X.; Qiu, J.; Li, B.; Wang, S.; Li, Z.; Wang, Z.; Wang, R.; Luu, A. T.; and Li, X. 2024. BioLlama3: Advancing Open-Source Biomedical Language Models with Llama3. arXiv:2407.03154

    Pith/arXiv arXiv 2024

  9. [9]

    Cliff, N. 1993. Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions. Psychological Bulletin, 114(3): 494--509

    work page 1993

  10. [10]

    L.; Choma, M

    Cross, J. L.; Choma, M. A.; and Onofrey, J. A. 2024. Bias in medical AI : Implications for clinical decision-making. PLOS Digital Health, 3(11): e0000651. Publisher: Public Library of Science (PLoS)

    work page 2024

  11. [11]

    F.; and Fiske, S

    Dovidio, J. F.; and Fiske, S. T. 2012. Under the Radar : How Unexamined Biases in Decision - Making Processes in Clinical Interactions Can Contribute to Health Care Disparities . American Journal of Public Health, 102(5): 945--952. Publisher: American Public Health Association

    work page 2012

  12. [12]

    Dwork, C.; Hardt, M.; Pitassi, T.; Reingold, O.; and Zemel, R. 2012. Fairness Through Awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS '12, 214–226. New York, NY, USA: Association for Computing Machinery

    work page 2012

  13. [13]

    A.; Scheidegger, C.; and Venkatasubramanian, S

    Friedler, S. A.; Scheidegger, C.; and Venkatasubramanian, S. 2016. On the (Im)possibility of Fairness. In FAT/ML

    work page 2016

  14. [14]

    A.; Scheidegger, C.; and Venkatasubramanian, S

    Friedler, S. A.; Scheidegger, C.; and Venkatasubramanian, S. 2019. A Comparative Study of Fairness-Enhancing Interventions in Machine Learning. FAccT

    work page 2019

  15. [15]

    Friedman, M. 1937. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance. Journal of the American Statistical Association, 32(200): 675--701

    work page 1937

  16. [16]

    Gonçalves, V.; and Strubell, E. 2023. Quantization and Distillation Reduce Social Bias in Language Models. arXiv preprint arXiv:2312.05662

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Graber, M. 2005. Diagnostic Errors in Medicine : A Case of Neglect . The Joint Commission Journal on Quality and Patient Safety, 31(2): 106--113. Publisher: Elsevier BV

    work page 2005

  18. [18]

    Hardt, M.; Price, E.; and Srebro, N. 2016. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems 29 (NIPS 2016), 3315--3323

    work page 2016

  19. [19]

    Hasnain-Wynia, R. 2007. Disparities in Health Care Are Driven by Where Minority Patients Seek Care : Examination of the Hospital Quality Alliance Measures . Archives of Internal Medicine, 167(12): 1233. Publisher: American Medical Association (AMA)

    work page 2007

  20. [20]

    Jacobs, A.; Barocas, S.; et al. 2021. Measurement and Fairness. FAccT

    work page 2021

  21. [21]

    Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; Lavaud, L. R.; Lachaux, M.-A.; Stock, P.; Scao, T. L.; Lavril, T.; Wang, T.; Lacroix, T.; and Rekdworak, V. 2023. Mistral 7B. arXiv:2310.06825

    Pith/arXiv arXiv 2023

  22. [22]

    Jin, D.; Pan, E.; Oufattole, N.; Weng, W.-H.; Fang, H.; and Szolovits, P. 2021. What Disease does this Patient Have? A Large-scale OPEN Medical Domain Question Answering Dataset . In Proceedings of the 20th Workshop on Biomedical Language Processing, 138--147. Online: Association for Computational Linguistics

    work page 2021

  23. [23]

    Jin, Q.; Dhingra, B.; Liu, Z.; Cohen, W.; and Lu, X. 2019. PubMedQA : A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2567--2577. Hong Kong, China: Association for Computa...

    work page 2019

  24. [24]

    A.; and Mark, R

    Johnson, A.; Bulgarelli, L.; Pollard, T.; Horng, S.; Celi, L. A.; and Mark, R. 2020. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), 49--55

    work page 2020

  25. [25]

    E.; Bulgarelli, L.; Shen, L.; Gayles, A.; Shammout, A.; Horng, S.; Pollard, T

    Johnson, A. E.; Bulgarelli, L.; Shen, L.; Gayles, A.; Shammout, A.; Horng, S.; Pollard, T. J.; Hao, S.; Moody, B.; Gow, B.; et al. 2023. MIMIC-IV, a freely accessible electronic health record dataset. Scientific data, 10(1): 1

    work page 2023

  26. [26]

    Johnson, A. E. W.; Bulgarelli, L.; Pollard, T. J.; Gow, B.; Moody, B.; Horng, S.; Celi, L. A.; and Mark, R. G. 2024. MIMIC-IV (version 3.1). PhysioNet. RRID:SCR\_007345

    work page 2024

  27. [27]

    Jung, C.; Concannon, C.; Zimmerman, J.; et al. 2019. Simple rules for complex decisions. In AAAI

    work page 2019

  28. [28]

    J.; Loftus, J.; Russell, C.; and Silva, R

    Kusner, M. J.; Loftus, J.; Russell, C.; and Silva, R. 2017. Counterfactual Fairness. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems 30, 4066--4076. Curran Associates, Inc

    work page 2017

  29. [29]

    Labrak, Y.; Bazoge, A.; Morin, E.; Rouvier, M.; and Gourraud, P.-A. 2024. BioMistral: A Collection of Open-Source Bio-medical Large Language Models. arXiv:2402.10373

    Pith/arXiv arXiv 2024

  30. [30]

    Levene, H. 1960. Robust Tests for Equality of Variances. In Olkin, I., ed., Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, 278--292. Stanford University Press

    work page 1960

  31. [31]

    Logé, C.; Ross, E.; Dadey, D. Y. A.; Jain, S.; Saporta, A.; Ng, A.; and Rajpurkar, P. 2021. Q-Pain: A Question Answering Dataset to Measure Social Bias in Pain Management (version 1.0.0). PhysioNet. RRID:SCR\_007345

    work page 2021

  32. [32]

    Massey, F. J. 1951. The Kolmogorov-Smirnov Test for Goodness of Fit. Journal of the American Statistical Association, 46(253): 68--78

    work page 1951

  33. [33]

    Meta, A. . 2024. The Llama 3 Herd of Models. arXiv:2404.11225

    Pith/arXiv arXiv 2024

  34. [34]

    Mitchell, M.; Wu, S.; Zaldivar, A.; et al. 2021. Model Cards for Model Reporting. CACM

    work page 2021

  35. [35]

    E.; Nassery, N.; Schaffer, A

    Newman-Toker, D. E.; Nassery, N.; Schaffer, A. C.; Yu-Moe, C. W.; Clemens, G. D.; Wang, Z.; Zhu, Y.; Saber Tehrani, A. S.; Fanai, M.; Hassoon, A.; and Siegal, D. 2024. Burden of serious harms from diagnostic error in the USA . BMJ Quality & Safety, 33(2): 109--120. Publisher: BMJ

    work page 2024

  36. [36]

    L.; Apakama, D

    Omar, M.; Soffer, S.; Agbareia, R.; Bragazzi, N. L.; Apakama, D. U.; Horowitz, C. R.; Charney, A. W.; Freeman, R.; Kummer, B.; Glicksberg, B. S.; Nadkarni, G. N.; and Klang, E. 2025. Sociodemographic biases in medical decision making by large language models. Nature Medicine, 31(6): 1873--1881. Publisher: Springer Science and Business Media LLC

    work page 2025

  37. [37]

    Qwen Team, A. G. 2024. Qwen2: A Family of Strong and General Open-source Large Language Models. arXiv:2406.16781

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [38]

    Russell, C.; Kusner, M.; and Loftus, J. 2017. When Worlds Collide: Integrating Different Counterfactual Assumptions in Fairness. In NeurIPS Workshop on Fairness

    work page 2017

  39. [39]

    Sheng, E.; Chang, K.-W.; Natarajan, P.; and Peng, N. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3407--3412

    work page 2019

  40. [40]

    Spearman, C. 1904. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 15: 72--101

    work page 1904

  41. [41]

    Suriyakumar, D.; Subramanian, D.; and Narayanan, A. 2023. Fairness under Demographic Perturbations: Learning with Dynamic Group Membership. In ICML

    work page 2023

  42. [42]

    Topol, E. J. 2019. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine, 25(1): 44--56

    work page 2019

  43. [43]

    Wilcoxon, F. 1945. Individual Comparisons by Ranking Methods. Biometrics Bulletin, 1(6): 80--83

    work page 1945

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.