Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Aletheia shows that a 2 GB offline AI model on an ordinary 8 GB laptop can rank the correct diagnosis in its top three for all ten tested clinical cases.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An offline Qwen2.5-3B model fine-tuned on 27,000 synthetic and exam-derived clinical samples reports 80% top-1 and 100% top-3 differential-diagnosis accuracy in a low-resource setting.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A credible offline CDSS engineering report, but the 80%/100% accuracy figures are in-distribution fit from the same synthetic template pool used for training, so they should not be read as evidence of clinical generalization. the 4 major comments →

arxiv 2607.24814 v1 pith:RAPG3PB6 submitted 2026-07-14 cs.AI cs.LGq-bio.OT

Aletheia: An Offline-First Clinical Decision Support System for Differential Diagnosis in Low-Resource Healthcare Settings

Joseph Walusimbi , Ann Move Oguti , Abubakhari Sserwadda , Precious Boss Kasasira , Charles Brian Okoboi This is my paper
classification cs.AI cs.LGq-bio.OT
keywords clinical decision supportoffline inferencelarge language modelslow-resource healthcaredifferential diagnosismodel quantizationsub-Saharan AfricaQLoRA fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Aletheia is an attempt to show that clinically useful AI-assisted differential diagnosis can run entirely on an ordinary laptop, with no internet connection, in a low-resource African healthcare setting. The paper builds a 3-billion-parameter language model, fine-tuned on roughly 27,000 clinical reasoning examples weighted toward East African disease prevalence, then compresses it to a 1.93 GB file that runs within about 3.6 GB of RAM. On ten representative clinical case categories, the system places the correct diagnosis first in 80% of cases and within its top three suggestions in 100% of cases. If these numbers hold outside the evaluation set, the system would give frontline clinicians a ranked differential space to consider under severe time pressure, on hardware already present in many district facilities.

Core claim

The paper claims that a QLoRA-fine-tuned, 4-bit quantized 3B-parameter instruction model, trained on a curated Africa-weighted dataset of 27,000 clinical reasoning samples spanning 50 conditions, can deliver offline differential diagnosis on an 8 GB laptop. In the paper's evaluation across ten core case categories, the correct diagnosis appears first in 80.0% of cases and within the top three ranked suggestions in 100.0% of cases, with BERTScore-F1 of 0.909 for the accompanying rationale text. The compressed model file is 1.93 GB and estimated peak inference RAM is about 3.63 GB, leaving a comfortable margin under a 7.17 GB memory ceiling. The authors present this as evidence that LLM-based

What carries the argument

The mechanism is the combination of (1) QLoRA fine-tuning of Qwen2.5-3B-Instruct on a synthetic-plus-filtered dataset of clinical reasoning samples with eight task types (differentials, test recommendations, evidence updates, red flags, etc.), and (2) GGUF Q4_K_M quantization to shrink the merged model to 1.93 GB for CPU-only inference. The clinical output is a ranked list of differential diagnoses with probability estimates, which is why Top-3 accuracy rather than single-answer accuracy is the operationally meaningful metric.

Load-bearing premise

The reported 80% / 100% accuracy figures are computed on evaluation items drawn from the same synthetic case-template pool and filtered question sources used to build the training set, not on prospectively collected patient cases; if real clinical presentations in East Africa differ from those templates, the accuracy will not transfer.

What would settle it

Run Aletheia on a pre-registered set of 100 real patient cases from a district hospital in Eastern Uganda, with diagnoses adjudicated by a panel of clinicians; if the correct diagnosis is not in the top three in more than a small fraction of cases, the central claim fails. Alternatively, measure actual peak memory on a standard 8 GB laptop; if it exceeds 7,168 MB, the deployment claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A standard 8 GB laptop with no GPU and no internet can host a diagnostic-reasoning system whose correct answer is always in the top three in all ten evaluated case categories.
  • The fine-tuned model is substantially better than the base model at Top-1 accuracy, showing that Africa-weighted clinical fine-tuning, not the base model, carries the diagnostic capability.
  • Mixing three data sources (synthetic templates and filtered medical question datasets) raised Top-1 accuracy from 70% to 80%, suggesting that dataset diversity improves clinical reasoning even when training loss rises.
  • Given an ECE of 0.275, the paper's own recommendation is to present probability outputs as relative rankings ('more likely' / 'less likely') rather than absolute percentages.
  • The deployment pipeline (fine-tune on cloud hardware, quantize, run on device) is a template for other offline clinical tools in resource-constrained settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 100% Top-3 result transfers to real patient encounters, the dominant cloud-dependent deployment model for medical LLMs would be challenged in low-resource settings; connectivity becomes irrelevant for core decision support.
  • The evaluation set is small and drawn from the synthetic template distribution, so the headline percentages should be read as upper bounds until a prospective multi-site study with real patient cases is completed.
  • The same recipe could be applied to other structured cognitive tasks in primary care — triage, test ordering, referral decisions — since the fine-tuning already covers eight reasoning task types.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Aletheia, an offline-first clinical decision support system for differential diagnosis in low-resource sub-Saharan African settings. It fine-tunes Qwen2.5-3B-Instruct with QLoRA on 27,000 training samples (60% synthetic, 20% MedQA, 20% MedMCQA) covering 50 East African conditions, and deploys the merged model in GGUF Q4_K_M format for on-device inference. The claims are: Top-1 diagnostic accuracy 80.0%, Top-3 100.0%, BERTScore-F1 0.909, METEOR 0.467, ECE 0.275, and estimated peak inference RAM of ~3,630 MB, passing the ADTC 2026 memory ceiling of 7,168 MB on an 8 GB laptop without internet. The paper positions this as a feasibility demonstration for LLM-based clinical reasoning in hardware-constrained environments.

Significance. If the claims were externally validated, this would be a valuable contribution: an offline, low-memory CDSS targeted at a genuine need in East African primary care, with a clear deployment pipeline (QLoRA fine-tuning, GGUF quantization, CLI and local web UI) and transparent training details. The paper's focus on deployability rather than raw accuracy is appropriate, and the choice of a 3B model is well motivated. However, the current evidence does not yet support the headline numbers, because the evaluation uses in-distribution synthetic cases from the same templates used for training, and memory compliance is estimated from a formula rather than measured. The core engineering is sound but the validation is not.

major comments (4)
  1. [§III.A, §V.B, Table III] The evaluation is not an independent test of clinical reasoning. The 3,000 evaluation samples are drawn from the same combined pool after a 60/20/20 mixing split, so 60% of the evaluation pool comes from the same hand-crafted Aletheia-Synthetic template distribution used in training. The 10 'representative clinical case categories' are, according to §V.B, the basis for the reported accuracy, yet no external or prospectively collected cases are described. A perfect Top-3 score over 10 categories is exactly what template memorization would produce. The limitation in §VII.D.2 concedes that the evaluation set is small and clinician-graded data are still being collected, but this concession does not repair the validity of the reported 80%/100% figures as evidence of generalization.
  2. [§V.C, Eq. (1), Table IV] Memory compliance with the ADTC 2026 budget is not measured. Eq. (1) is an additive formula with hand-set constants (MOS=900 MB, MKV=400 MB, Mruntime=300 MB, Mapp=200 MB), and Table IV reports 'estimated peak RAM.' There is no measurement on the ADTC benchmark laptop or any process-level memory profiling. The deployment claim is therefore unverified. Additionally, the abstract states 'approximately 3,630 MB' while the Introduction states 'approximately 3,730 MB' and Table IV again says '~3,630 MB'; this inconsistency needs reconciliation.
  3. [§IV.B vs. §IV.C] The methodology is internally inconsistent about QLoRA. Section IV.B says 'we used full bfloat16 precision without quantisation' during training, which is not QLoRA (quantized low-rank adaptation). If the base model is not quantized during fine-tuning, the method is conventional LoRA; the deployment quantization in §IV.C is a post-training step. This distinction matters for reproducibility and for the claimed alignment with QLoRA-based efficient fine-tuning.
  4. [§V.B, §VI.B, Table III] The statistical basis for the accuracy claims is unclear. The paper reports a 3,000-sample evaluation set, but the headline results appear to be computed on 10 case categories. No confidence intervals, per-case breakdown, or statistical tests are provided. With n=10, an 80% Top-1 accuracy has a wide confidence interval, and a 100% Top-3 score is not informative beyond 'all ten observed cases succeeded.' Clarify how the 3,000 evaluation samples were used and report uncertainty on the key metrics.
minor comments (5)
  1. [Abstract and Introduction] The memory figure discrepancy (3,630 MB vs. 3,730 MB) should be corrected and a single consistent value used throughout.
  2. [§III.B] The 'africa weight parameter, range 3–5' is not defined precisely. State how the weights are computed and applied during sampling.
  3. [Table III] ROUGE-2 and ROUGE-L are listed as '—' despite Fig. 8 presumably depicting these metrics. Report the numerical values in the table or state why they are omitted.
  4. [§VII.A] The sentence in §VII.A discusses '90% of cases' but no 90% figure appears in the results; this appears to be a stray reference and should be corrected.
  5. [§I] The phrase 'fromthe Greek fortruthor disclosure' is malformed; fix spacing and wording. Also check the email address for the corresponding author and the reference formatting consistency.

Circularity Check

1 steps flagged

Headline accuracy is an in-distribution score on the authors' own synthetic templates, not an external clinical prediction.

specific steps
  1. fitted input called prediction [§III.A, §V.B, §VI.B]
    "After applying a 60/20/20 mixing ratio across all three sources ... the final dataset comprised 27,000 training samples and 3,000 evaluation samples, totalling 30,000 samples — an increase of 25% over the initial single-source configuration. ... Per-condition precision, recall, and F1 were computed over the 10 core evaluation case categories."

    By the paper's own composition description, the 3,000 evaluation samples are split from the same combined pool whose dominant component (60%) is Aletheia-Synthetic, generated from the 50 hand-crafted templates; no external or prospectively collected cases are introduced for evaluation. The '10 core evaluation case categories' are therefore instantiations of the same template distribution the model was fine-tuned on. The reported Top-1 80.0% / Top-3 100.0% is an in-distribution score measuring reproduction of the authors' own data generator, not an independent clinical prediction; it is the expected output of a model fit to that distribution, so the evaluation is statistically forced by the training input rather than being an external test of generalization.

full rationale

The central 'prediction' is the diagnostic-accuracy claim. It is computed on a 3,000-sample evaluation split that Section III.A draws from the same 60/20/20 mixture of Aletheia-Synthetic, MedQA, and MedMCQA used for training. Since 60% of that pool is generated from the same 50 hand-crafted templates on which the model was fine-tuned, the evaluation measures how well the model reproduces the authors' own template distribution rather than how it performs on unseen clinical presentations. The paper itself concedes in §VII.D that the quantitative evaluation was on 10 core categories with automated metrics and that a clinician-graded set is still under development, confirming that no external test set supported the headline numbers. This is a fitted-input-called-prediction circularity: the metric is determined by construction to reward memorization of the training template families. The memory-compliance claim is separately based on an additive formula with hand-set constants (Eq. 1) and is therefore an estimate, not a measurement, but I do not count that as definitional circularity; it is an unverified assumption. No self-citation chain is load-bearing. Score 6 reflects one central prediction that reduces to the training distribution by construction, while the work retains some independent content (e.g., QLoRA/GGUF deployment pipeline).

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central claims depend on unverified synthetic clinical priors, hand-set mixing ratios, and an unmeasured memory model. These are the main things the reader would have to trust without independent evidence.

free parameters (5)
  • Dataset mixing ratio = 60/20/20 (synthetic / MedQA / MedMCQA)
    Section III.A: chosen by hand to balance sources; directly determines training and evaluation distribution.
  • Africa weight parameter = range 3–5 per condition
    Section III.B: hand-set weights by estimated incidence; shapes the synthetic training data and the evaluation distribution.
  • Minimum-token quality filter = 50 tokens
    Section III.A: arbitrary cutoff applied before mixing; affects which samples enter training and evaluation.
  • Memory formula constants = MOS=900 MB, MKV=400 MB, Mruntime=300 MB, Mapp=200 MB
    Section V.C, Eq. (1): hand-assigned constants used to claim ADTC compliance without direct measurement.
  • LoRA hyperparameters = r=32, alpha=64, dropout=0.05, lr=2e-4, epochs=3
    Section IV.B, Table II: chosen by hand; these values affect model quality and the reported accuracy.
axioms (5)
  • domain assumption Hand-crafted 50-condition case templates encode clinically valid probability distributions for East African presentations
    Section III.A generates 20,000 samples from these templates; Section VII.D.1 admits the probabilities reflect the authors' clinical synthesis, not empirical cohort data.
  • domain assumption Keyword-filtered MedQA/MedMCQA items transfer to African clinical decision support
    Section III.A applies a 180-term keyword filter and mixes 12,000 such items; no evidence of contextual transfer is provided.
  • domain assumption Qwen2.5-3B-Instruct retains sufficient clinical reasoning after QLoRA fine-tuning and GGUF Q4_K_M quantization
    Section IV.A/C selects the base model and quantization; the only supporting evidence is the internal evaluation, which is in-distribution.
  • domain assumption The additive memory model in Eq. (1) with specified constants approximates real inference RAM on the ADTC laptop
    Section V.C sets MOS=900, MKV=400, Mruntime=300, Mapp=200 MB and infers ~3,630 MB without a direct measurement.
  • domain assumption Automated text-similarity metrics (ROUGE, BERTScore, METEOR) reflect clinical reasoning quality
    Section VI.E interprets BERTScore-F1 of 0.909 as semantic fidelity to expert answers, but no clinician-graded validation of reasoning quality is presented.
invented entities (1)
  • Aletheia-Synthetic (20,000 template-generated clinical reasoning samples) no independent evidence
    purpose: Primary training source encoding East African disease prevalence and probability estimates.
    Not released; generated from author-written templates; no external ground truth or prospective cohort, so the underlying epidemiology and clinical priors cannot be independently verified.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Aletheia: An Offline-First Clinical Decision Support System for Differential Diagnosis in Low-Resource Healthcare Settings." pith.science (2026). https://pith.science/paper/RAPG3PB6

@misc{pith2026260724814,
  author       = {Pith},
  title        = {Pith review of: Aletheia: An Offline-First Clinical Decision Support System for Differential Diagnosis in Low-Resource Healthcare Settings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RAPG3PB6}},
  note         = {Machine review of arXiv:2607.24814}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Access to specialist clinical expertise remains severely limited across sub-Saharan Africa, where physician-to-patient ratios can fall below 1:25,000 in rural settings. Existing AI-assisted diagnostic tools predominantly require reliable internet connectivity and high-specification hardware, rendering them impractical for frontline healthcare workers in district hospitals and health centres. This paper presents Aletheia, an offline-first clinical decision support system designed for low-resource healthcare contexts across sub-Saharan Africa. Aletheia is built upon Qwen2.5-3B-Instruct, fine-tuned using Quantised Low- Rank Adaptation (QLoRA) on a curated dataset of 27,000 clinical reasoning samples spanning 50 disease conditions with elevated prevalence in East Africa. Evaluation demonstrates a Top-1 diagnostic accuracy of 80.0%, Top-3 accuracy of 100.0%, BERTScore-F1 of 0.909, and METEOR of 0.467 across ten representative clinical case categories. The system achieves an Expected Calibration Error (ECE) of 0.275 and passes the Africa Deep Tech Challenge 2026 (ADTC 2026) memory budget constraint of 7 168 MB, achieving a peak inference RAM of approximately 3 630 MB on the standardised benchmark laptop. These results demonstrate the feasibility of deploying large language model-based clinical reasoning at the primary care level in resource-constrained settings without cloud infrastructure.

Figures

Figures reproduced from arXiv: 2607.24814 by Abubakhari Sserwadda, Ann Move Oguti, Charles Brian Okoboi, Joseph Walusimbi, Precious Boss Kasasira.

Figure 3
Figure 3. Figure 3: Top-1 (80.0%) and Top-3 (100.0%) diagnostic accuracy across 10 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Per-condition precision, F1, and recall across all 10 evaluation case [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-epoch training loss and validation loss. Epoch 1: 0.3498 / 0.369. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 7
Figure 7. Figure 7: Top-1 accuracy across five clinical reasoning task types. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 5
Figure 5. Figure 5: Confusion matrix for Top-1 differential diagnosis predictions across [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Top-1 diagnostic accuracy stratified by clinical severity (Critical, High, [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Left: Reliability diagram with ECE = 0.275. Right: Brier score stratified [PITH_FULL_IMAGE:figures/full_fig_p005_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of fine-tuned Aletheia vs base Qwen2.5-3B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Box plot distributions of ROUGE-1, ROUGE-2, and ROUGE-L [PITH_FULL_IMAGE:figures/full_fig_p006_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 6 linked inside Pith

  1. [1]

    Global Health Workforce Statistics,

    World Health Organization, “Global Health Workforce Statistics,” Geneva: WHO, 2023. [Online]. Available: https://www.who.int/data/gho/data/ themes/topics/health-workforce

    2023

  2. [2]

    AI in health and medicine,

    P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol, “AI in health and medicine,”Nature Medicine, vol. 28, no. 1, pp. 31–38, 2022

    2022

  3. [3]

    Dermatologist-level classification of skin cancer with deep neural networks,

    A. Estevaet al., “Dermatologist-level classification of skin cancer with deep neural networks,”Nature, vol. 542, no. 7639, pp. 115–118, 2017. 8

    2017

  4. [4]

    A comprehensive EHR timeseries pre- training benchmark,

    M. B. A. McDermottet al., “A comprehensive EHR timeseries pre- training benchmark,” inProc. ACM Conf. Health, Inference, and Learning, 2021, pp. 257–278

    2021

  5. [5]

    Global Strategy on Digital Health 2020–2025,

    World Health Organization, “Global Strategy on Digital Health 2020–2025,” Geneva: WHO, 2021. [Online]. Available: https://www.who.int/docs/default-source/documents/ gs4dhdaa2a9f352b0445bafbc79ca799dce4d.pdf

    2020

  6. [6]

    QLoRA: Ef- ficient finetuning of quantized LLMs,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Ef- ficient finetuning of quantized LLMs,” inAdvances in Neural Information Processing Systems, vol. 36, 2023

    2023

  7. [7]

    llama.cpp: Inference of LLaMA model in pure C/C++,

    G. Gerganov, “llama.cpp: Inference of LLaMA model in pure C/C++,” GitHub, 2023. [Online]. Available: https://github.com/ggerganov/llama. cpp

    2023

  8. [8]

    GPT-4 Technical Report,

    OpenAI, “GPT-4 Technical Report,” arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Towards expert-level medical question answering with large language models,

    K. Singhalet al., “Towards expert-level medical question answering with large language models,” arXiv:2305.09617, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    BioMedLM: A domain-specific large language model for biomedical text,

    E. Boltonet al., “BioMedLM: A domain-specific large language model for biomedical text,” Stanford CRFM, 2022

    2022

  11. [11]

    ClinicalBERT: Modeling clinical notes and predicting hospital readmission,

    K. Huang, J. Altosaar, and R. Ranganath, “ClinicalBERT: Modeling clinical notes and predicting hospital readmission,” arXiv:1904.05342, 2019

    Pith/arXiv arXiv 1904

  12. [12]

    Implementing electronic medical record systems in developing countries,

    H. S. F. Fraser, P. Biondich, D. Moodley, S. Choi, B. W. Mamlin, and P. Szolovits, “Implementing electronic medical record systems in developing countries,”Informatics in Primary Care, vol. 13, no. 2, pp. 83–95, 2005

    2005

  13. [13]

    mHealth innovations as health system strength- ening tools,

    A. B. Labriqueet al., “mHealth innovations as health system strength- ening tools,”Global Health: Science and Practice, vol. 1, no. 2, pp. 160–171, 2013

    2013

  14. [14]

    Mobile health use in low-and high-income countries,

    A. Bastawrous and M. J. Armstrong, “Mobile health use in low-and high-income countries,”Journal of the Royal Society of Medicine, vol. 106, no. 4, pp. 130–142, 2013

    2013

  15. [15]

    Ada: Your personal health guide,

    Ada Health GmbH, “Ada: Your personal health guide,” 2020. [Online]. Available: https://ada.com

    2020

  16. [16]

    Babylon health AI,

    Babylon Health, “Babylon health AI,” 2020. [Online]. Available: https: //www.babylonhealth.com

    2020

  17. [17]

    GPTQ: Accu- rate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accu- rate post-training quantization for generative pre-trained transformers,” arXiv:2210.17323, 2022

    Pith/arXiv arXiv 2022

  18. [18]

    AWQ: Activation-aware weight quantization for LLM compression and acceleration,

    J. Linet al., “AWQ: Activation-aware weight quantization for LLM compression and acceleration,” arXiv:2306.00978, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Health Statis- tics and Information Systems: African Region,

    World Health Organization Regional Office for Africa, “Health Statis- tics and Information Systems: African Region,” Brazzaville: WHO AFRO, 2023. [Online]. Available: https://www.afro.who.int/health-topics/ health-statistics

    2023

  20. [20]

    What disease does this patient have? A large-scale open domain question answering dataset from medical exams,

    D. Jinet al., “What disease does this patient have? A large-scale open domain question answering dataset from medical exams,”Applied Sciences, vol. 11, no. 14, p. 6421, 2021

    2021

  21. [21]

    MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering,

    A. Palet al., “MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering,” inProc. Conference on Health, Inference, and Learning, PMLR, vol. 174, pp. 248–260, 2022

    2022

  22. [22]

    Qwen2.5 Technical Report,

    Qwen Team, “Qwen2.5 Technical Report,” arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    PEFT: State-of-the-art parameter-efficient fine- tuning methods,

    S. Mangrulkaret al., “PEFT: State-of-the-art parameter-efficient fine- tuning methods,” GitHub, 2022. [Online]. Available: https://github.com/ huggingface/peft

    2022

  24. [24]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out, 2004, pp. 74–81

    2004

  25. [25]

    BERTScore: Evaluating text generation with BERT,

    T. Zhanget al., “BERTScore: Evaluating text generation with BERT,” inProc. ICLR, 2020

    2020

  26. [26]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” inProc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures, 2005

    2005

  27. [27]

    Obtaining well calibrated probabilities using Bayesian binning,

    M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using Bayesian binning,” inProc. AAAI, 2015, pp. 2901– 2907

    2015

  28. [28]

    Verification of forecasts expressed in terms of probability,

    G. W. Brier, “Verification of forecasts expressed in terms of probability,” Monthly Weather Review, vol. 78, no. 1, pp. 1–3, 1950

    1950

  29. [29]

    ADTC 2026: On-Device Language Model Benchmark for African Hardware Contexts,

    Africa Deep Tech Challenge, “ADTC 2026: On-Device Language Model Benchmark for African Hardware Contexts,” 2026. [Online]. Available: https://africadeeptech.org/challenge-2026

    2026

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.