Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Clinical safety measured in English does not transfer to Hausa: locally deployable medical models score as harmful in Hausa while a frontier model stays competent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:30 UTC pith:VNPDUKHH

load-bearing objection Small local medical models drop from competent in English to harmful in Hausa; the tier-level conclusion leans on a single frontier model, but the core finding is real and worth refereeing. the 3 major comments →

arxiv 2607.17270 v1 pith:VNPDUKHH submitted 2026-07-19 cs.CL cs.CY

Safety That Does Not Transfer: Cross-Lingual Clinical Correctness Drift in Deployable Medical Language Models

Anthonio Oladimeji Gabriel , Dimeji Olawuyi , Toba Ajayi , Temilola Aderemi This is my paper
classification cs.CL cs.CY
keywords clinical correctness driftcross-lingual safetyHausalow-resource deploymentlocal language modelsmedical safety evaluationdangerous confidence rateNigerian treatment guidelines
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether clinical safety evidence obtained in English and on frontier systems applies to the small, quantised models actually run offline in low-resource settings. It answers no, on the basis of 128 matched English-Hausa responses scored against Nigerian national treatment guidelines. Five locally deployable models fell from a mean clinical correctness of 1.57 in English to -0.03 in Hausa, crossing into the harmful range on average; the single frontier reference model dropped only from 2.00 to 1.75 and never gave a harmful answer. Because every model answered competently in English and the frontier model answered competently in Hausa, the authors attribute the failure to the deployment tier rather than to the language or the clinical task. The result matters because the users most likely to consult these models in Hausa are also the least likely to detect a fluent but wrong answer.

Core claim

The central claim is that clinical correctness in English does not transfer to Hausa for the deployable tier, and that the locus of failure is the class of model, not the language or the clinical material. In the authors' data, mean correctness among five locally deployable 4-9 billion parameter models fell from 1.57 in English to -0.03 in Hausa, while a frontier system moved from 2.00 to 1.75. Harmful responses among deployable models rose from 5% of English items to 38% of Hausa items under raw flags (25% after adjudication); the frontier model produced none. Drift occurred in every condition - malaria, sickle cell disease, and tuberculosis - and across all five local models, so the result

What carries the argument

The instrument is a matched English-Hausa benchmark of twelve items, four each for malaria, sickle cell disease, and tuberculosis, with four question forms per condition: knowledge recall, emergency triage, a leading question inviting a contraindicated action, and a traditional-remedy claim. Correctness is anchored to Nigerian national treatment guidelines, with explicit prohibited statements, so a recommendation of chloroquine or a two-month tuberculosis stop is harmful by rule. The Dangerous Confidence Rate - the proportion of responses that are unhedged, confident, and clinically wrong - is introduced because refusal-based safety metrics cannot distinguish a safe answer from a confidently

Load-bearing premise

The attribution of the whole deficit to the deployable tier rests on a single frontier API model being representative of frontier systems, and on the subset of its calls that succeeded being unbiased; if another frontier model answered incorrectly in Hausa, the conclusion would collapse to a one-model versus five-model comparison.

What would settle it

Run the same twelve matched English-Hausa items under the same rubric with several additional frontier systems, and also re-score the released outputs with clinically qualified raters. If any second frontier model produces a harmful or guideline-violating Hausa answer on the emergency or leading-question items, the claim that the deficit is a property of the deployable tier fails: the result would be a difference among individual models, not between tiers. The released dataset makes this check immediately performable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety evaluation must be conducted at the tier of deployment; frontier results do not transfer downward.
  • Safety evaluation must be conducted in the language of use; English results do not transfer outward.
  • For clinical applications, safety evaluation must score correctness, not refusal, because a refusal-based test cannot catch fluent wrong answers.
  • Models below the frontier tier should not be relied on for clinical guidance in languages in which they have not been evaluated in that tier and that language.
  • Language-routing failures, such as answering Hausa prompts in Swahili, indicate a distinct failure mode with consequences beyond the clinical case.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the tier-attribution argument, if sound, generalises to any low-resource language and any safety-critical domain: a safety property verified in one language and tier is not a property of the model but of the evaluation setting.
  • A direct testable extension is to run the same twelve Hausa items through several additional frontier systems; if any fails the emergency or leading-question items, the 'deployable tier' conclusion would need to be weakened to 'the particular frontier model tested differs from the particular small models tested.'
  • The observed loss of medical fine-tuning benefit in Hausa suggests a hypothesis the authors could not test: English medical fine-tuning may narrow, rather than broaden, multilingual clinical competence. One could test this by comparing fine-tuned and base versions of the same architecture on the Hausa items.
  • The adjudication rule that silent failure in an emergency counts as harm could be formalised into an automatic metric, which would make the harm estimate replicable without relying on rater judgment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates whether clinical safety established in English transfers to Hausa for the class of small, locally deployable language models used in low-resource health settings. Six models were tested: five locally deployable systems (4–9B parameters, two medically fine-tuned) and one frontier API model. Twelve matched English–Hausa items across three conditions (malaria, sickle cell disease, tuberculosis) and four question forms were scored against Nigerian national treatment guidelines by two blind fluent Hausa raters. The central finding is that locally deployable models' mean clinical correctness fell from 1.57 in English to −0.03 in Hausa, while the frontier model moved from 2.00 to 1.75 and produced no harmful responses in either language. The paper concludes that the deficit is a property of the deployable tier, not of the language or the clinical material, and argues that safety assurance must be conducted at the tier and in the language of intended use.

Significance. If the central finding holds, this is an important contribution to multilingual safety evaluation for medical LLMs. The study is anchored to an external standard (Nigerian national treatment guidelines), uses blind dual rating with substantial agreement on the principal endpoint (κ=0.70), and the second rater independently reproduced the tier separation. The authors are notably transparent: they report the poor raw harm κ, describe the post-hoc adjudication rule, flag the single-frontier-model limitation, and release prompts, outputs, and code. The paper identifies a concrete failure mode—correctness drift rather than refusal drift—that is likely to be missed by existing multilingual safety benchmarks. However, the headline attribution to a 'deployable tier' rests on a single frontier model, and the small sample without uncertainty quantification tempers the strength of the conclusion.

major comments (3)
  1. [Discussion, Table 1, Fig. 7] The central conclusion—that the deficit is a property of the 'deployable tier'—rests entirely on a contrast between five local models and one frontier API model. The Discussion acknowledges this ('one system whose behaviour need not generalise'), but the Abstract and Conclusion still state the tier-level attribution as a definitive finding. As presented, the evidence supports 'five small models degraded; the one frontier model tested did not', not a general property of all frontier systems. Please either evaluate at least two or three additional frontier models (or show that different frontier models behave similarly on these items), or rephrase the central claim to refer to 'the frontier model in this study' rather than 'the frontier tier'.
  2. [Results, Fig. 7 and Methods] A subset of frontier API calls returned service errors and was excluded, but the paper does not report how many calls were excluded, for which items/languages, or whether exclusion was related to prompt difficulty. If service errors occurred disproportionately on longer or more complex Hausa prompts, the frontier's near-ceiling 1.75 mean could be inflated by selective completion. Please provide the excluded-call counts per item and language, and include a sensitivity analysis (e.g., assigning worst-case scores to excluded calls) to show that the tier separation is robust to these missing data.
  3. [Results, Table 1 and Figs. 1–3] The paper reports pooled means without any measure of uncertainty. With only five local models and one frontier model, and with multiple items per model, the difference between −0.03 and 1.75 could be sensitive to item selection or model-specific outliers. Report per-model means (Fig. 3 already does) and provide item-level bootstrap confidence intervals or a mixed-effects model with random effects for item and model. This would allow readers to assess whether the tier contrast is credible beyond this specific item set, rather than relying on the raw mean difference.
minor comments (5)
  1. [Methods, Results] The Dangerous Confidence Rate is defined in Methods but never reported in Results. Either report it for each tier/language or remove the definition to avoid an unused metric.
  2. [Related work] Several statements lack citations: 'Work on Singaporean and Albanian contexts' and the benchmarks 'Med-SafetyBench, CARES' are mentioned without reference numbers. Please add the relevant citations.
  3. [References [10]–[14]] The reference list contains untracked '[VERIFY: edition and year]' placeholders. These must be replaced with complete bibliographic details before publication.
  4. [Fig. 6] The axes 'silent failure' and 'harmful failure' are not formally defined in the text or caption. Define the thresholds used to place models on these axes, since the figure is central to the failure-mode discussion.
  5. [Abstract, Fig. 7] The abstract states 'All 128 responses were scored', but the total possible responses would be 144 (6 models × 12 items × 2 languages) and some frontier calls were excluded. Clarify how 128 is reached (e.g., excluded calls, timeouts) and state the exact number of excluded calls in the text and figure caption.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark against external national-treatment guidelines; the tier attribution is an experimental contrast, not a construction.

full rationale

The paper's derivation chain is empirical rather than definitional. It builds matched English–Hausa items, anchors correctness to an external standard (Nigerian national treatment guidelines, refs [10]–[14]), runs six models, and scores all 128 responses blind by two fluent Hausa speakers. The central claim — that the deficit is a property of the deployable tier rather than of Hausa or the clinical material — rests on two observed contrasts: local models degrade from 1.57 to −0.03 in Hausa, while the frontier model moves only from 2.00 to 1.75 and is fluent and guideline-consistent in Hausa. Neither the frontier result nor the local drift is assumed by construction; both are measured outputs. There is no fitted parameter later relabeled as a prediction, no quantity defined in terms of the conclusion, and no uniqueness theorem or rescaling ansatz imported from prior same-author work. The only self-reference is ref [16], the authors' own data and code repository, which supplies the raw material for replication rather than supporting the inference itself. Acknowledged limitations — a single frontier API model, excluded service-error calls, poor pre-adjudication harm kappa, the Yoruba 'agbo' item, and non-clinician raters — are threats to generalizability and measurement precision, and the paper states them explicitly; they do not make any step reduce to its own input. The comparison is therefore self-contained against an external benchmark and contains no circular step.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central empirical result is measured against an external standard (national guidelines) and does not fit any numeric free parameter to the data. The load-bearing non-empirical choices are the guideline standard, matched-item equivalence, lay-rater validity, the single-model frontier proxy, the post hoc harm rule, and the quantised-deployment proxy. All are disclosed in the paper.

axioms (6)
  • domain assumption Nigerian national treatment guidelines are the correct external standard for clinical correctness in the three studied conditions.
    Methods: correctness is anchored to NMEP, national sickle cell guideline, and NTBLCP. If these guidelines are misapplied or out of date, all clinical correctness scores are mis-anchored.
  • domain assumption The 12 English-Hausa item pairs are matched in clinical content and difficulty, so observed drift is attributable to language.
    Methods: Hausa items were authored by one fluent speaker and back-translated, but there is no independent validation of item difficulty equivalence. The sickle-cell traditional-remedy item using Yoruba 'agbo' is an acknowledged violation.
  • domain assumption Two fluent Hausa speakers without formal clinical training can validly score open-ended medical responses against written guidelines.
    Discussion: 'correctness was scored by two speakers of Hausa without formal clinical qualification'; a clinically trained rater might weigh partial answers differently. Inter-rater agreement is substantial but not perfect (κ=0.70).
  • domain assumption The single frontier model accessed by API is representative of the frontier deployment tier.
    Discussion: 'the frontier tier is represented by one system whose behaviour need not generalise to others.' This is load-bearing for the paper's conclusion that the deficit is a property of the deployable tier rather than of the specific models compared.
  • ad hoc to paper Harm in emergency scenarios includes failure to deliver the required urgent action, even when the output is unintelligible or evasive.
    Results: the adjudication rule was adopted to resolve the 22 divergent harm items and was 'formulated after the disagreement was observed rather than before data collection.' The 25% Hausa harm figure depends on this rule.
  • domain assumption Ollama default quantisation and temperature-zero decoding represent the 'locally deployable tier' as actually encountered.
    Methods: models were 'served through Ollama at default quantisation.' Quantisation level affects small-model behaviour and is not varied, so the exact tier boundary is a chosen proxy.

pith-pipeline@v1.3.0-alltime-deepseek · 6604 in / 11935 out tokens · 115880 ms · 2026-08-01T18:30:20.755701+00:00 · methodology

0 comments
read the original abstract

Safety evaluation of large language models is conducted predominantly in English and predominantly on frontier systems. Neither condition describes how such models are encountered in low-resource health settings, where small quantised systems are run locally and queried in local languages. We ask whether clinical safety established in English transfers to Hausa, and whether any failure is attributable to the language, the clinical task, or the class of model that low-resource deployment admits. Matched English-Hausa question pairs were built for three conditions of high burden in northern Nigeria: malaria, sickle cell disease, and tuberculosis, probing knowledge recall, emergency triage, a leading question inviting a contraindicated action, and a traditional-remedy claim. Six models were evaluated: five locally deployable systems of 4-9 billion parameters, two medically fine-tuned, and one frontier system. All 128 responses were scored against Nigerian national treatment guidelines by two fluent Hausa speakers working independently and blind to one another. Among locally deployable models, mean clinical correctness fell from 1.57 in English to -0.03 in Hausa, on a scale where 2 denotes a correct answer and -1 an actively harmful one. The frontier model moved from 2.00 to 1.75 and produced no response judged harmful in either language. Drift was consistent across all three conditions. Inter-rater agreement was substantial for clinical correctness (kappa = 0.70); agreement on harm was initially poor (kappa = 0.22) and is examined in detail. Because a frontier model answers the same questions competently in Hausa, the deficit is a property neither of the language nor of the clinical material, but of the deployable tier.

Figures

Figures reproduced from arXiv: 2607.17270 by Anthonio Oladimeji Gabriel, Dimeji Olawuyi, Temilola Aderemi, Toba Ajayi.

Figure 1
Figure 1. Figure 1: Mean clinical correctness by condition and language, pooled across all six models. The shaded region below zero denotes responses judged actively harmful on average. Pooling conditions, the five locally deployable models fell from a mean correctness of 1.57 in English to −0.03 in Hausa, crossing from competent to harmful on average. The frontier model moved from 2.00 to 1.75 ( [PITH_FULL_IMAGE:figures/ful… view at source ↗
Figure 2
Figure 2. Figure 2: Clinical correctness by deployment tier and language. Locally deployable models cross into the harmful range in Hausa; the frontier model does not. Tier Correctness (EN) Correctness (HA) Change Locally deployable (n = 5 models) 1.57 −0.03 −1.60 Frontier (n = 1 model) 2.00 1.75 −0.25 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-model trajectories between languages. The frontier model is drawn dashed. Convergence of the deploy￾able models toward and below zero is uniform in direction, not the property of a single outlier. Disaggregating by condition and question form ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Hausa clinical correctness by model, condition, and question form. Cell values are means over small numbers of observations and should be read as indicative rather than precise [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Proportion of responses judged actively harmful, by tier and language, under raw flags. Adjudicated labels yield 25% for the deployable tier in Hausa. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Silent failure against harmful failure in Hausa. The axes are not exclusive: under the adjudication rule adopted here, a response may be both unintelligible and harmful when it occurs in an emergency scenario, which is why several models score highly on both. The medically fine-tuned models occupy the high-silent-failure region, medgemma:4b timing out on the majority of Hausa prompts. The general-purpose m… view at source ↗
Figure 7
Figure 7. Figure 7: Language of response to Hausa prompts. The frontier bar is shorter because a subset of its calls returned service errors during collection and were excluded, as described in the limitations. 0.82), and substantial for clinical correctness (κ = 0.70) by conventional thresholds [2]. Agreement on the harm flag was poor (κ = 0.22) despite 82.8% raw concordance. The discrepancy is the familiar consequence of ma… view at source ↗
Figure 8
Figure 8. Figure 8: Agreement between two blind raters. Panel (b) is illustrative of the effect of adjudication and is not itself an independent reliability estimate; see text. correctness. The two raters differ on the exact position of the deployable tier relative to zero; they do not differ on its position relative to the frontier. Discussion The result is best stated by what it rules out. Hausa is not the obstacle: a front… view at source ↗
Figure 7
Figure 7. Figure 7: The frontier estimates are therefore less precisely determined than the local ones, and the frontier [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 3 linked inside Pith

  1. [1]

    A coefficient of agreement for nominal scales

    Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20(1):37–46

    1960

  2. [2]

    The measurement of observer agreement for categorical data

    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74

    1977

  3. [3]

    High agreement but low kappa: I

    Feinstein AR, Cicchetti DV . High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology. 1990;43(6):543–9

    1990

  4. [4]

    High agreement but low kappa: II

    Cicchetti DV , Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology. 1990;43(6):551–8

    1990

  5. [5]

    Low-resource languages jailbreak GPT-4

    Yong ZX, Menghini C, Bach SH. Low-resource languages jailbreak GPT-4. arXiv:2310.02446 [cs.CL]. 2023

    Pith/arXiv arXiv 2023

  6. [6]

    All languages matter: on the multilingual safety of large language models

    Wang W, Tu Z, Chen C, Yuan Y , Huang JT, Jiao W, et al. All languages matter: on the multilingual safety of large language models. Findings of the Association for Computational Linguistics: ACL 2024

    2024

  7. [7]

    IndicSafe: a benchmark for evaluating multilingual LLM safety in South Asia

    Pattnayak P, Chowdhuri S. IndicSafe: a benchmark for evaluating multilingual LLM safety in South Asia. arXiv:2603.17915 [cs.CL]. 2026

    Pith/arXiv arXiv 2026

  8. [8]

    arXiv:2508.12733 [cs.CL]

    LinguaSafe: a comprehensive multilingual safety benchmark for large language models. arXiv:2508.12733 [cs.CL]. 2025

    Pith/arXiv arXiv 2025

  9. [9]

    HealthBench: evaluating large language models towards improved human health

    Arora RK, Wei J, Soskin Hicks R, Bowman P, et al. HealthBench: evaluating large language models towards improved human health. OpenAI. 2025

    2025

  10. [10]

    National guidelines for the diagnosis and treatment of malaria

    Federal Ministry of Health, National Malaria Elimination Programme. National guidelines for the diagnosis and treatment of malaria. Abuja: Federal Ministry of Health, Nigeria. [VERIFY: edition and year]

  11. [11]

    National guideline for the control and management of sickle cell disease

    Federal Ministry of Health, Nigeria. National guideline for the control and management of sickle cell disease. Abuja: Federal Ministry of Health. [VERIFY: edition and year]

  12. [12]

    National guide- lines for the management of tuberculosis

    Federal Ministry of Health, National Tuberculosis and Leprosy Control Programme. National guide- lines for the management of tuberculosis. Abuja: Federal Ministry of Health, Nigeria. [VERIFY: edi- tion and year] 13

  13. [13]

    WHO guidelines for malaria

    World Health Organization. WHO guidelines for malaria. Geneva: World Health Organization. [VER- IFY: year of edition used]

  14. [14]

    WHO consolidated guidelines on tuberculosis, Module 4: treatment

    World Health Organization. WHO consolidated guidelines on tuberculosis, Module 4: treatment. Geneva: World Health Organization. [VERIFY: year of edition used]

  15. [15]

    Available from:https://ollama.com

    Ollama: run large language models locally. Available from:https://ollama.com

  16. [16]

    Hausa health drift: benchmark, data and analysis code

    Oladimeji Gabriel A, Olawuyi D, Ajayi T, Aderemi T. Hausa health drift: benchmark, data and analysis code. Available from:https://github.com/anthoniooladimeji11-coder/ hausa-health-drift 14