{"id":"97ef3aa6-54ab-48d4-adca-3880f29704d3","arxiv_id":"2412.00554","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An evaluation of GPT-4, Claude-3, and LLaMA-3 on six healthcare tasks finds low accuracy and demographic unfairness, with less favorable predictions for African American patients.","lead":"Large language models were tested on six healthcare prediction tasks, and they often performed near random chance while showing consistent racial disparities. The results suggest that deploying LLMs in clinical decision support without subgroup-level auditing is risky.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fairness estimates in Table 3 are computed from 60–500 test examples without confidence intervals; standard errors for EOD reach 6–14 percentage points, so the word 'persistent' is not supported by the reported statistics alone.","rationale":"The paper's strongest claim has two parts. The performance claim—that LLMs struggle and often barely surpass random guessing—is directly supported by Table 2 (e.g., readmission accuracies of 48–57% against a 50% random baseline; Neighbor/Landlord accuracies of 33–41% against a 33.3% baseline). That part is credible. The load-bearing weakness is the fairness claim, because it depends on small-sample differences in conditional proportions that are reported without any uncertainty quantification. The reader's weakest_assumption identifies exactly this issue, and I agree. The concern is not that the authors are wrong; it is that the reported statistics cannot carry the inferential weight placed on them. The Health Coaching test set of 60 examples and the Neighbor/Landlord sets of 261 examples produce standard errors of roughly 7–14 percentage points for the fairness metrics, which is large relative to most point estimates in Table 3. The cross-row pattern of negative DPDs for African Americans is striking, but the paper never tests whether that pattern could arise by chance under a null of no disparity, nor does it present intervals that would let a reader judge stability. Adding confidence intervals and a formal sign test is a minimal, standard remedy that would either strengthen the claim or require it to be qualified. Because the reader already recommends CONDITIONAL and this concern is the same one, the verdict remains UNCHANGED. There is no reason to reject the paper outright: the data collection and experimental design are reasonable, the qualitative examples are illustrative, and the performance results provide independent support for the general message that real-world healthcare tasks remain difficult for LLMs. But the strongest fairness wording should be conditional on statistical verification.","tokens_in":14654,"tokens_out":4713,"duration_ms":50616,"concrete_test":"For every row of Table 3, reconstruct per-group counts from the stated test sizes and balanced sampling design, then compute exact Clopper-Pearson 95% confidence intervals for DPD and EOD and a bootstrap interval for each difference. Next, run a one-sided sign test across the African-American-vs-White rows under the null that each true disparity is zero. If the pooled effect remains significantly negative and the sign consistency survives, the 'persistent' wording is supported; if not, the paper must downgrade to 'suggestive' and report the intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central fairness claim—that LLMs 'consistently predict less favorable outcomes for African American patients' and show 'a lower Equality of Opportunity for African Americans in most tasks'—rests entirely on the DPD and EOD point estimates in Table 3. These are differences of conditional proportions computed on test sets with n=60 (Health Coaching), n=261 (Neighbor/Landlord), n=500 (MIMIC tasks), and n=175 (MedQA, reported separately). No confidence intervals, bootstrap estimates, or significance tests are provided. For a balanced MIMIC split with approximately 250 African-American and 250 White patients and a favorable-outcome rate near 0.5, the standard error of DPD is sqrt(0.25/250 + 0.25/250) ≈ 4.5 percentage points; a reported DPD of -8.2 has a 95% interval that includes zero. EOD is conditioned on Y=1, which roughly halves the per-group counts, raising the SE to about 6.3 points; the mortality EOD of -3.5 is therefore within one standard error of zero. For Health Coaching, with n=60 split into subgroups of about 20, the DPD SE is approximately 13.7 points, so an estimate of -4.2 is statistically indistinguishable from zero. Even the Neighbor/Landlord DPDs (SE ≈ 6.6 points) are mostly within the noise. The qualitative pattern of negative DPDs for African Americans is suggestive and may survive a formal sign test across the 13 rows, but the reported point estimates alone do not justify the strength of the words 'persistent' and 'consistently'. The claim is plausible, but the evidence as presented is underpowered for the conclusion drawn.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates GPT-4, Claude-3, and LLaMA-3 across six healthcare tasks (MIMIC mortality, MIMIC readmission, health coaching goal completion, schizophrenia/bipolar neighbor and landlord scene classification, and MedQA) using three frameworks: in-context learning with chain-of-thought, LoRA fine-tuning, and an LLM-as-agent retrieval pipeline. Performance is measured by global accuracy and by two fairness metrics, Demographic Parity Difference (DPD) and Equal Opportunity Difference (EOD), reported for race and gender subgroups. The central claims are that LLMs struggle with real-world healthcare classification (some implementations barely exceeding random guessing), that they show persistent demographic disparities (especially less favorable predictions and lower equality of opportunity for African American patients), that explicit demographic prompting gives mixed results, and that access to up-to-date guidelines via agents does not guarantee accurate predictions.","tokens_in":14998,"tokens_out":4013,"duration_ms":43313,"significance":"If the quantitative claims are reliable, this is a useful and timely cautionary result for the deployment of LLMs in healthcare. The paper compares three model families and three learning paradigms on six tasks, including realistic dialogue-based and clinical-note tasks, which is broader than many single-benchmark evaluations. The qualitative examples and the sociolinguistic consultation on race inference are valuable and give concrete evidence of stereotyped reasoning. However, the headline fairness conclusions rest entirely on point estimates from small test sets with no confidence intervals or significance tests, and some accuracy comparisons are statistically indistinguishable from random baselines at the reported sample sizes. The paper's strength is therefore the breadth and plausibility of the patterns, not yet the statistical support for the word 'persistent.'","major_comments":[{"comment":"The central fairness claim that LLMs 'consistently predict less favorable outcomes for African American patients' and show 'persistent' disparities is not supported by the reported statistics alone, because all DPD and EOD values are point estimates without confidence intervals, bootstrap estimates, or significance tests. With the test sizes in Table 1, the standard errors are large: for a balanced MIMIC split of 500 (roughly 250 per group), the SE of a DPD is about 4.5 percentage points, so a value of -8.2 is within a 95% interval that includes zero; EOD, being conditioned on Y=1, has a still larger SE. For Health Coaching (n=60), the subgroup sizes are about 20, giving a DPD SE of roughly 13.7 points, so the reported -4.2 and -12.5 values are statistically indistinguishable from zero. For Neighbor and Landlord (n=261), the SE is about 6.6 points, so most reported DPDs are within noise. Please report per-group counts, standard errors or bootstrap confidence intervals, and significance tests for the DPD/EOD estimates, or explicitly weaken the abstract and Section 6.2 wording from 'persistent' and 'consistently' to directional patterns. A sign test across the 13 rows would be a minimal additional check.","section":"Section 6.2, Table 3"},{"comment":"The accuracy claims that some implementations 'barely surpass random guessing' also need uncertainty quantification. For example, the ICL GPT-4 readmission accuracy of 55.3 on n=500 has an approximate 95% confidence interval of [50.9, 59.7] under the usual normal approximation, which includes the 50% random baseline; the same applies to several other values in the 52-55% range on MIMIC tasks. In addition, Section 4 says 'We report the best performance between Zero-Shot and N-Shot in this setting,' but the paper does not state which variant produced each number or how many shots were used, and selecting the best of several prompted configurations without multiple-comparison adjustment tends to inflate apparent performance. Please report the exact prompt settings per task and provide confidence intervals or error bars for the accuracy values that are near the random baseline.","section":"Section 6.2, Table 2 and Section 4"},{"comment":"The paper states that each dataset was sampled so that classes, demographic attributes, and P(C=c|Z=z) are 'roughly balanced,' but it does not report the actual demographic counts in the test sets, which are necessary both for interpreting the fairness metrics and for computing their precision. Without exact subgroup sizes, a reader cannot verify the standard-error calculations or know whether the 'roughly balanced' condition holds for the small Health Coaching and Schizophrenia/Bipolar datasets. Please include a supplementary table with the per-subgroup test counts for each task and framework, or at least for each task, and use those counts in the uncertainty analysis requested above.","section":"Section 3 and Table 1"}],"minor_comments":[{"comment":"The MedQA column reports ICL results only inside parentheses (with explicit demographics), with the outside cell left as '-'. It would be clearer to state explicitly that MedQA was evaluated only with the demographic-prompt variant, and to give the sample size and the fact that these are percentages from n=175.","section":"Table 2 caption"},{"comment":"The text defines 'Demographic Parity Difference (DPD)' but the Table 3 caption and several places in Section 6.2 use 'PDP' instead of 'DPD.' Please make the acronym consistent throughout.","section":"Section 5 and Table 3 caption"},{"comment":"The race-inference results for Health Coaching (LLaMA-3 40.0 vs random 33.3) and the Claude-3 refusals are reported without any uncertainty or note on sample size. Given n=60, the 40.0% value is within one standard error of the random baseline; please add a caveat or a confidence interval.","section":"Section 6.3, Table 6"},{"comment":"The phrase 'four to eight-shot in-context examples' is vague; please specify the exact number of shots used for each task, and whether the same examples were used for all demographic-prompt conditions.","section":"Section 4, ICL baselines"},{"comment":"The Limitations section does not mention the small-sample precision issue that affects the main fairness and accuracy comparisons. Adding a sentence acknowledging that the DPD/EOD estimates are noisy and that the conclusions are preliminary would improve the paper's self-assessment.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad empirical survey of LLM behavior in healthcare, and the qualitative findings are timely. The main barrier is statistical: the central fairness and accuracy claims are stated much more strongly than the reported sample sizes and lack of uncertainty quantification can support. I think the paper can be made acceptable by adding confidence intervals or significance tests and softening the abstract and Section 6.2 claims, hence major_revision rather than reject. The self-citations to the authors' own health coaching dataset and chain-of-thought variant are disclosed and do not by themselves affect the central conclusion, but the authors should make sure the dependency of the health coaching results on non-public demographic annotations is clear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth engaging, but the headline claim outruns the evidence. What's actually new: a multi-task (six) comparison of ICL, LoRA fine-tuning, and an agent pipeline on healthcare prediction with accuracy and two fairness metrics, plus a race-inference-from-dialogue study with a sociolinguist review. That is a genuinely useful benchmark formulation, and the qualitative examples of stereotyped rationales are the most convincing part. The consistent pattern of lower favorable predictions for African-American patients across models and tasks is suggestive and not explained away by the self-citations (the health coaching dataset and one CoT baseline come from the authors' own work, but the central claim doesn't depend on them).\n\nThe soft spots are real and mostly about statistical power. Table 3 reports DPD/EOD point estimates from test sets as small as 60 to 500 examples, with no confidence intervals or significance tests. For a balanced MIMIC split, a DPD of -8.2 has a standard error around 4.5 points; the EOD for mortality at -3.5 is about one SE from zero. The Health Coaching numbers are computed from roughly 20 per subgroup, so SEs around 14 points. The word 'persistent' in the abstract and Section 6.2 is not supported by the reported numbers alone. What would support it: a sign test across the 13 rows, or bootstrap intervals, or simply softer language. The paper also says it reports the best performance between zero-shot and N-shot CoT without giving the selection rule or per-setting numbers, which makes the accuracy comparisons hard to audit. Code and two of the four datasets are not released, which is a practical limitation for a benchmark paper.\n\nNone of this kills the paper. The qualitative claims about LLM underperformance and biased reasoning hold; the race-inference section is careful and the sociolinguistic consultation is a nice touch. The conditional verdict is right. I'd send it to review with a request for CIs or bootstrap, clearer reporting of the best-of selection, and either a sign test or softened categorical claims. A serious referee would make it better, and the community would benefit from this benchmark being available with numbers that can actually be compared.","headline":"Solid multi-task fairness benchmark in low-resource healthcare, but the headline 'persistent disparities' claim needs confidence intervals before it can be taken at face value.","tokens_in":15520,"tokens_out":1952,"would_cite":true,"duration_ms":20145,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art LLMs, evaluated on six real-world healthcare prediction tasks under three learning frameworks, often barely outperform random guessing and consistently predict less favorable outcomes for African American patients.","keywords":["large language models","healthcare fairness","demographic parity","equality of opportunity","low-resource healthcare","in-context learning","parameter-efficient fine-tuning","LLM agents"],"falsifier":"Bootstrap the reported DPD and EOD estimates from the same test sets: if the 95% confidence interval for the African-American-versus-White gap on, say, the 500-example MIMIC mortality test contains zero, then the paper's central claim of persistent racial disparity on that task would not survive.","tokens_in":14447,"feed_emoji":"⚕️","tokens_out":8588,"duration_ms":127775,"temperature":0.7,"pith_summary":"This paper sets out to test whether state-of-the-art large language models can handle real-world healthcare prediction when data is scarce, and whether their predictions are fair across demographic groups. Across six tasks—mortality, readmission, health coaching goal completion, schizophrenia/bipolar diagnosis, and medical exam questions—the authors evaluate GPT-4, Claude-3, and LLaMA-3 under in-context learning, parameter-efficient fine-tuning, and an agent framework that searches for current guidelines. They report that many setups barely beat random guessing, and that the models consistently predict less favorable outcomes for African American patients and show lower equality of opportunity for that group in most tasks. The paper also reports that adding demographic information to prompts gives mixed results, and that models can infer race from conversations using stereotyped linguistic cues. If these findings hold, LLMs are not yet reliable or equitable enough for patient-facing deployment without dedicated fairness research.","feed_headline":"LLMs slip on real healthcare tasks, skew against Black patients","feed_subtitle":"Across six clinical tasks and three learning setups, top models near random guessing and favor White patients.","key_machinery":"The argument is carried by a benchmark suite of six healthcare tasks built from four datasets, paired with the standard fairness metrics Demographic Parity Difference (DPD) and Equal Opportunity Difference (EOD). DPD measures how much more or less often a group receives the favorable prediction compared with everyone else; EOD measures the same gap restricted to patients whose true outcome is favorable. The evaluation deliberately balances classes and demographic attributes so that each subgroup contributes roughly equally, then applies the same three frameworks—chain-of-thought in-context learning, LoRA fine-tuning, and a web-searching agent—across the tasks. This setup is what lets the authors attribute differences in accuracy and fairness to the models and frameworks rather than to skewed test distributions.","core_discovery":"On its own terms, the paper claims that the general-domain success of large language models does not transfer to low-resource healthcare classification. In the authors' six benchmarks, implementations such as zero-shot and few-shot chain-of-thought, LoRA fine-tuning, and a ReAct-style agent frequently hover near the random-guess baseline, with the weakest cases in readmission and the two schizophrenia/bipolar diagnosis tasks. Fairness measurements using Demographic Parity Difference and Equal Opportunity Difference show a consistent direction: relative to White patients, African American patients receive favorable predictions less often and are correctly classified as favorable less often in most tasks, with racial gaps larger than gender gaps. The paper further claims that explicitly inserting demographic attributes into prompts does not reliably improve accuracy or fairness, and that the LLM-as-agent approach can retrieve current guidelines yet still reach wrong conclusions by misapplying them. Finally, the authors show that models can infer patients' race from dialogue and that the reasoning used is often stereotyped, which they take as evidence of a hidden pathway for biased predictions in conversational health tasks.","pith_inferences":["The reported fairness gaps are point estimates from test sets of 60 to 500 examples; a bootstrap or confidence-interval analysis could reveal that several gaps are statistically indistinguishable from zero, so the 'persistent' pattern should be treated as provisional until quantified with uncertainty.","A natural testable extension is to rerun the mortality and readmission evaluations on naturalistic (unbalanced) patient distributions; the balanced sampling used here may understate or overstate the real-world disparity depending on how base rates interact with model bias.","The agent's failure mode—retrieving correct guidelines but misapplying them to spoken dialogue—suggests a concrete intervention: require the model to quote the specific guideline clause it used and justify its application, which could be evaluated as a direct follow-up.","Because models infer race from dialect features like 'ain't' and 'gonna', a targeted stress test could use dialogues with matched content but systematically varied dialect markers to measure how much of the diagnosis disparity is driven by linguistic stereotyping."],"forward_implications":["Claimed few-shot competence of LLMs on generic classification does not extend to these real-world healthcare tasks; accuracy can sit at or near random guessing even with chain-of-thought examples.","Racial disparity, not gender disparity, is the dominant fairness pattern: African American patients receive less favorable predictions and lower true positive rates in most tasks and frameworks.","Explicitly providing demographic information is not a dependable fairness intervention; its effects on accuracy, DPD, and EOD vary by task, model, and metric.","Giving an LLM web access to current guidelines does not guarantee correct health predictions; retrieved facts can be relevant while the model's reasoning from them remains wrong.","Race inference from conversational text is feasible and biased, so dialogue-based health tasks carry a risk of hidden demographic bias even when no demographics are supplied."],"supporting_citations":[{"why":"provides the MIMIC-IV clinical notes used to build the mortality and readmission tasks.","marker":"Johnson et al., 2020"},{"why":"provides the text-message health coaching corpus used for goal-completion prediction.","marker":"Gupta et al., 2020"},{"why":"provides the schizophrenia and bipolar interview transcripts used for the two diagnosis tasks.","marker":"Aich et al., 2022"},{"why":"provides MedQA, the medical exam question set used as the controlled QA comparison.","marker":"Jin et al., 2020"},{"why":"introduces chain-of-thought prompting, the core of the in-context learning framework.","marker":"Wei et al., 2022"},{"why":"introduces LoRA, the parameter-efficient fine-tuning method applied to LLaMA-3.","marker":"Hu et al., 2021"},{"why":"introduces ReAct, the basis of the agent pipeline's search-then-answer loop.","marker":"Yao et al., 2023"},{"why":"introduces Reflexion, the basis of the agent's self-evaluation and retry steps.","marker":"Shinn et al., 2023"},{"why":"introduces demographic parity, the source of the DPD fairness metric.","marker":"Zemel et al., 2013"},{"why":"supplies the demographically balanced prompting scheme and prior evidence that explicit demographic prompts do not reliably improve fairness.","marker":"Wang et al., 2023"}],"fun_headline_variants":["LLMs guess on clinical tasks, skew white","AI health models bias against Black patients","LLMs infer race, leading to biased health predictions","Agent LLMs fail low-resource healthcare fairness","Race gaps in LLM health tasks: Black patients lose"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions rest on the assumption that the small, demographically balanced test sets (60 to 500 examples per task) give stable estimates of how often each group receives favorable predictions and true positives, so sampling noise could change the size or even the sign of some reported disparities.","fun_headline_variants_meta":{"raw":{"variants":["LLMs guess on clinical tasks, skew white","AI health models bias against Black patients","LLMs infer race, leading to biased health predictions","Agent LLMs fail low-resource healthcare fairness","Race gaps in LLM health tasks: Black patients lose"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1375,"prompt_tokens":866,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":482,"tokens_out":509,"duration_ms":66484,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:13:27.647023+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Bootstrap the reported DPD and EOD estimates from the same test sets: if the 95% confidence interval for the African-American-versus-White gap on, say, the 500-example MIMIC mortality test contains zero, then the paper's central claim of persistent racial disparity on that task would not survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the MIMIC-IV clinical notes used to build the mortality and readmission tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the text-message health coaching corpus used for goal-completion prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"introduces demographic parity, the source of the DPD fairness metric."}],"review_version":1}