Zero-Shot ATC Coding with Large Language Models for Clinical Assessments

Jimmy Lin; John-Michael Gamble; John P. Hirdes; Micaela Jantzi; Zijian Chen

REVIEW 4 major objections 4 minor 11 references

Zero-Shot ATC Coding with Large Language Models for Clinical Assessments

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automatic ATC drug coding is feasible with locally deployable open-source language models when the task is reframed as five levels of constrained multiple-choice selection among valid codes.

desk verdict A useful first paper on LLM-based ATC coding, but the abstract's accuracy numbers are overstated and the Ontario Health results only cover the codeable 166 of 200 notes. read the letter →

arxiv 2412.07743 v1 pith:TB74JPNV submitted 2024-12-10 cs.CL

Zijian Chen , John-Michael Gamble , Micaela Jantzi , John P. Hirdes , Jimmy Lin This is my paper · ORCID

classification cs.CL

keywords ATCcodinglargelanguagemodelshierarchicalclassificationlevel-by-levelpromptingclinicalfreetextzero-shotlearningprivacy-preservingNLPmedical

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that assigning WHO Anatomical Therapeutic Chemical (ATC) codes to drug mentions can be automated with large language models without sending clinical data to outside services. Its central move is to treat coding not as free-form code generation but as a five-level multiple-choice walk down the ATC hierarchy, forcing the model to pick from valid codes at each level. On two curated product-name benchmarks and on 166 unambiguous real prescription notes, this reaches 78% exact-match accuracy with GPT-4o and 60% with Llama 3.1 70B in the zero-shot setting. The paper further claims that a fine-tuned 8B Llama model matches the zero-shot 70B accuracy, which matters because an 8B model is small enough to run on local infrastructure in privacy-sensitive health care.

What carries the argument

The machinery is level-by-level prompting through the ATC ontology: the model receives the drug mention and a list of valid codes for the current level, each with its generic name or UMLS definition, and must output exactly one option; its choice determines the candidate set for the next level, repeated five times until a seven-character code is produced. This constrains the output space to at most 37 options, with an average of about 5, instead of the full 6,807-code space, and eliminates code fabrication. Knowledge grounding ablated three option formats, code only, code plus generic name, and code plus UMLS definition, showing that names are the main driver of accuracy and UMLS definitions add a small deeper-level boost.

What would settle it

Have two independent clinicians re-label the 200 real prescription notes, including the 34 excluded as ambiguous, using the same granularity scale, then score the models against the adjudicated labels; if expert agreement is low or the models' accuracy on the previously excluded notes differs sharply from the reported 60%, the claimed real-world feasibility is not reproducible.

Watch

Extended reading notes

Core claim

The discovery is that ATC coding reduces to hierarchical selection: asking the model at each of the five ATC levels to choose one option out of the valid children of the previously chosen code prevents it from inventing non-existent codes and keeps each decision small, with roughly five choices on average instead of a full 6,807-code space. Guided this way, zero-shot models perform well on familiar product names and on real prescription notes, which the authors find are often variants of product names; performance drops sharply on a benchmark with low lexical overlap between product and generic names. Grounding options with UMLS definitions gives modest gains at the deepest levels. The paper's headline capacity claim is that a fine-tuned Llama 3.1 8B reaches the same accuracy as zero-shot Llama 3.1 70B, making local, privacy-preserving deployment feasible.

Load-bearing premise

All real-world accuracy figures rest on one domain expert's ATC labels and ambiguity-level annotations for the 200 real prescription notes, with 34 notes excluded as too ambiguous, so a single labeling mistake or a debatable exclusion shifts the headline numbers.

Editorial extensions

If this is right

If the parity result holds, privacy-sensitive organizations can run ATC coding on an 8B open-source model locally and get the accuracy of a 70B zero-shot model.
In the authors' real-world notes, 166 of 200 prescriptions could be confidently assigned exact level-5 codes, and the open-source models scored above 85% at level 3 on those notes.
Because models succeed where product names resemble generic names, improving performance on sets with low brand-generic overlap is the identified next bottleneck; the authors suggest hybrid rule-based and LLM systems.
Adding UMLS definition grounding produces small but real gains at level 5, so richer per-option context can be used without changing the prompting structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The five-level multiple-choice framing is a general recipe: any hierarchical coding scheme with bounded branching, such as ICD, SNOMED, or RxNorm ingredient hierarchies, could get the same fabrication-free reduction, and the paper's ablation suggests lexical overlap, not model size, will be the bottleneck.
The granularity-level annotation points toward confidence-aware coding: a production system could return the deepest defensible code and flag ambiguous notes for human review instead of forcing a level-5 guess.
A testable extension of the string-similarity explanation is that per-code substring overlap between product and generic names should predict per-code correctness; if it does not, the proposed failure mechanism is incomplete.
Independent replication on a second site's prescription corpus would show whether fine-tuned-8B parity with zero-shot 70B generalizes beyond the specific training split used here.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here's my take on the ATC coding paper. The core idea is sound, and it's a genuinely useful first step: framing ATC coding as level-by-level selection among valid options, rather than free-form generation, is the right call and clearly builds on Boyle et al. The empirical work is honest in places — the RABBITS results are much weaker than Health Canada, and they dig into why (string similarity), which is a nice insight. The finding that a fine-tuned 8B model matches zero-shot 70B on these tasks is practically relevant for privacy-preserving deployment. I have no issue with the method or the evaluation logic inside each table.

The soft spots are real, and they're mostly about framing. The abstract and introduction say the method 'achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B' without saying these are Health Canada numbers. Table 1 shows RABBITS level-5 at 19.8% for 70B and Ontario Health at 49.4%. That's a substantial overstatement. Worse, the Ontario Health numbers are only on the 166 of 200 notes judged unambiguous at level 5. The 20 notes at granularity level 0 are exactly the hard free-text cases where an expert couldn't determine a code; omitting them inflates the 'real-world' claim. The stress-test's arithmetic is right: if you put those 34 notes back in as failures, the level-5 numbers drop to roughly 44% for the fine-tuned 8B and 41% for Llama 3.1 70B, with a ceiling of 83% even for perfect coding on the unambiguous subset. That doesn't kill the paper, but the central feasibility claim needs to be restated as 'works on the codeable subset, and here's how often a note is codeable.'

Also worth noting: the gold annotations and granularity levels come from a single expert, with no agreement check. There's no rule-based baseline, which would have been easy to run given that 43% of Health Canada names are substrings of generic names — a string matcher might be strong there. No code or data released, though they say they hope to release the 200-pair set. These are minor-to-moderate issues; the abstract overstatement is the big one.

Who's this for? Applied health NLP researchers, and anyone building local LLM pipelines for regulated data. It's worth reading because the method is simple and the failure points are instructive. It deserves peer review, but only after the authors fix the abstract, report both full-set and subset numbers, and ideally add a rule-based comparison.

My recommendation: send it to review, with a clear request for revision.

Referee Report

4 major / 4 minor

Summary. The paper proposes a level-by-level prompting method for assigning WHO ATC drug codes to prescription text. The method asks an LLM to choose among valid ATC codes at each of the five ontology levels, optionally grounding each choice with generic drug names or UMLS definitions. The authors evaluate GPT-4o and Llama 3.1 70B in zero-shot settings and a fine-tuned Llama 3.1 8B on three datasets: Health Canada product names, the RABBITS product-name benchmark, and 200 real clinical prescription notes from Ontario Health. The paper reports 78.4% exact-match accuracy with GPT-4o and 60.3% with Llama 3.1 70B on Health Canada, and claims that a fine-tuned Llama 3.1 8B matches or surpasses zero-shot Llama 3.1 70B accuracy.

Significance. If the results are stated accurately, the paper makes a useful practical contribution: it demonstrates that a locally deployable open-source model can perform ATC coding on drug mentions without sending clinical data to external APIs. The evaluation is grounded in an external ATC ontology and uses expert-verified labels, and the Code Only ablation controls for the possibility that listing generic names in the prompt simply leaks the answer. The focus on privacy-preserving deployment and the granularity-aware evaluation of ambiguous clinical notes are also strengths. However, the manuscript currently overstates the breadth of the headline numbers and excludes the most difficult real-world notes from the main exact-match results, so the central feasibility claim is not yet fully supported.

major comments (4)

[Section 5, Table 1] Section 4.1 states that 'All 200 prescription notes are used for evaluation', but Section 5 reports results only for the 166 Ontario Health notes with granularity level 5, and the Table 1 caption confirms this. The 34 excluded notes include 20 at granularity level 0. If the excluded notes are counted as incorrect, the Ontario Health level-5 accuracy becomes at most 41.0% for Llama 3.1 70B and 44.1% for the fine-tuned 8B instead of the reported 49.4% and 53.1%. Because the abstract and conclusion claim feasibility on real free-text clinical prescriptions, this exclusion is load-bearing; the paper must either evaluate all 200 notes under a clearly specified rule or explicitly subordinate the real-world claim to the unambiguous subset.
[Abstract, Section 1] The abstract presents the 78% and 60% figures as if they were general results across all datasets, but Table 1 shows that these are Health Canada level-5 numbers. On RABBITS, exact-match accuracy is 39.4% for GPT-4o and 19.8% for Llama 3.1 70B, and on Ontario Health the zero-shot Llama 3.1 70B number is 49.4%. The summary should report per-dataset results and avoid implying dataset-independent performance.
[Section 5, Fine-tuning Effectiveness] The claim that fine-tuned Llama 3.1 8B 'consistently surpasses' zero-shot Llama 3.1 70B is based on point differences with no uncertainty quantification. On Health Canada the exact-match scores are 60.5% versus 60.3%, a difference of 0.2 percentage points; only the RABBITS difference (26.4% versus 19.8%) is substantial. Without confidence intervals, bootstrapping, or repeated runs, the comparison is not established and the claim should be moderated.
[Section 4.1, Section 4.2] The gold-standard ATC labels and granularity levels for the Ontario Health notes come from a single domain expert, with no inter-annotator reliability measure. Because the granularity assignment determines which notes enter the main real-world evaluation, a second annotator or a sensitivity analysis is needed to demonstrate that the 166-note denominator is stable rather than a source of selection bias.

minor comments (4)

[Title] The title contains an extra space in 'A TC'; it should read 'Zero-Shot ATC Coding'.
[Section 4.1] The sentence 'All 200 prescription notes are used for evaluation' directly contradicts the Section 5 statement that only the 166 level-5 notes are evaluated; the wording should be revised to describe the granularity-aware evaluation exactly.
[Section 5, Table 2] The text says that the Code Only setting achieves 'meaningful accuracy', but the RABBITS level-5 Code Only result is 8.4%; this description seems overstated for that dataset.
[Section 2] The claim that this is 'the first attempt to automate ATC coding using LLMs' is presented without a systematic literature search beyond the cited ICD work; the novelty claim should be softened or supported with a broader comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: evaluation is anchored to external ontology labels and expert-annotated gold data, with no self-citation or fit-to-prediction chain.

full rationale

The derivation chain is self-contained against external anchors. ATC codes and candidate option lists are taken from the WHO ATC ontology, Health Canada's Drug Product Database, and RxNorm via RABBITS; the Ontario Health gold labels are assigned by a domain expert (JMG) independently of model outputs. The level-by-level prompting technique is an explicit adoption of Boyle et al. (2023), which is external prior work, not a self-citation, and it is presented as the method rather than as a hidden premise. The 'Code Only' ablation removes generic-name hints from the option lists, so the accuracy comparison is not purely an artifact of answer leakage in the 'With Name' setting. No parameter or conclusion is fitted to the evaluation labels, and no claim reduces to its own input by construction. The exclusion of 34 ambiguous Ontario Health notes, including 20 at granularity level 0, is a scope/reporting limitation about generalizing to noisy free-text prescriptions rather than a circular step: those notes are omitted from the denominator because their labels cannot be confidently assessed, but the model outputs and the accuracy computation for the remaining 166 notes are not constructed from those exclusions. The paper's core feasibility claim is therefore empirically grounded against external data, not definitionally guaranteed.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to the target metric. The paper's main inputs from prior literature are the ATC ontology and external expert labels. The only threshold-like choice is restricting Ontario evaluation to 166 notes with granularity level 5, which is a post hoc evaluation subset rather than a fitted parameter. No new entities are postulated.

assumptions (4)

domain assumption Ground-truth ATC labels in the Health Canada Drug Product Database and RABBITS are correct and complete.
All accuracy numbers are computed against these labels; the paper does not audit label errors (Section 4.1).
domain assumption The single domain expert's ATC and granularity annotations for the 200 Ontario Health notes are reliable.
No inter-annotator agreement is reported; all ambiguous exclusions and correct-level evaluation depend on this expert's judgment (Sections 4.1 and 4.2).
domain assumption The ATC ontology is a complete and correct classification for every evaluated drug mention, so no code outside the official list is valid.
Level-by-level prompting restricts answers to valid ontology options, so any gold label must exist in the ontology (Section 3.1).
ad hoc to paper Level-by-level choice errors do not substantially compound; each level's selection is accepted without backtracking or confidence scoring.
The method forces one path through the hierarchy, and a wrong high-level choice makes the correct level-5 code unreachable (Section 3.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Zero-Shot ATC Coding with Large Language Models for Clinical Assessments." pith.science (2026). https://pith.science/paper/TB74JPNV

@misc{pith2026241207743,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot ATC Coding with Large Language Models for Clinical Assessments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TB74JPNV}},
  note         = {Machine review of arXiv:2412.07743}
}

read the original abstract

Manual assignment of Anatomical Therapeutic Chemical (ATC) codes to prescription records is a significant bottleneck in healthcare research and operations at Ontario Health and InterRAI Canada, requiring extensive expert time and effort. To automate this process while maintaining data privacy, we develop a practical approach using locally deployable large language models (LLMs). Inspired by recent advances in automatic International Classification of Diseases (ICD) coding, our method frames ATC coding as a hierarchical information extraction task, guiding LLMs through the ATC ontology level by level. We evaluate our approach using GPT-4o as an accuracy ceiling and focus development on open-source Llama models suitable for privacy-sensitive deployment. Testing across Health Canada drug product data, the RABBITS benchmark, and real clinical notes from Ontario Health, our method achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B. We investigate knowledge grounding through drug definitions, finding modest improvements in accuracy. Further, we show that fine-tuned Llama 3.1 8B matches zero-shot Llama 3.1 70B accuracy, suggesting that effective ATC coding is feasible with smaller models. Our results demonstrate the feasibility of automatic ATC coding in privacy-sensitive healthcare environments, providing a foundation for future deployments.

Figures

Figures reproduced from arXiv: 2412.07743 by the authors.

**Figure 2.** Examples of drug mentions and their corre [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 2 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Olivier Bodenreider. 2004. https://doi.org/10.1093/nar/gkh061 The Unified Medical Language System ( UMLS ): Integrating biomedical terminology . Nucleic Acids Research, 32:D267--70

doi:10.1093/nar/gkh061 2004
[4]

Boyle, Antanas Kascenas, Pat Lok, Maria Liakata, and Alison Q

Joseph S. Boyle, Antanas Kascenas, Pat Lok, Maria Liakata, and Alison Q. O'Neil. 2023. http://arxiv.org/abs/2310.06552 Automated clinical coding using off-the-shelf large language models . arXiv :2310.06552

arXiv 2023
[5]

Kolbinger, Hannah Sophie Muti, Zunamys I

Jan Clusmann, Fiona R. Kolbinger, Hannah Sophie Muti, Zunamys I. Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia L \"o ffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, Sophia J. Wagner, and Jakob Nikolas Kather. 2023. https://doi.org/10.1038/s43856-023-00370-1 The future landscape of large language models ...

doi:10.1038/s43856-023-00370-1 2023
[6]

Jack Gallifant, Shan Chen, Pedro Moreira, Nikolaj Munch, Mingye Gao, Jackson Pond, Leo Anthony Celi, Hugo Aerts, Thomas Hartvigsen, and Danielle Bitterman. 2024. http://arxiv.org/abs/2406.12066 Language models are surprisingly fragile to drug names in biomedical benchmarks . arXiv :2406.12066

arXiv 2024
[7]

Chao-Wei Huang, Shang-Chi Tsai, and Yun-Nung Chen. 2022. https://doi.org/10.18653/v1/2022.clinicalnlp-1.2 PLM - ICD : Automatic ICD coding with pretrained language models . In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 10--20, Seattle, WA. Association for Computational Linguistics

doi:10.18653/v1/2022.clinicalnlp-1.2 2022
[8]

Kellmann, Pauline Lanting, Lude Franke, Esther J

Alexander J. Kellmann, Pauline Lanting, Lude Franke, Esther J. van Enckevort, and Morris A. Swertz. 2023. https://doi.org/10.1093/database/baad019 Semi-automatic translation of medicine usage data (in Dutch , free-text) from lifelines COVID -19 questionnaires to ATC codes . Database, 2023:baad019

work page doi:10.1093/database/baad019 2023

Show all 11 references

[9]

van der Velde, Nynke Smidt, Rolf Sijmons, Hans Hillege, and Morris A

Chao Pang, Annet Sollie, Anna Sijtsma, Dennis Hendriksen, Bart Charbon, Mark de Haan, Tommy de Boer, Fleur Kelpin, Jonathan Jetten, Joeri K. van der Velde, Nynke Smidt, Rolf Sijmons, Hans Hillege, and Morris A. Swertz. 2015. https://doi.org/10.1093/database/bav089 SORTA : a sy...

2015 doi
[10]

Glicksberg, Eyal Zimlichman, Yiftach Barash, Robert Freeman, Alexander W

Ali Soroush, Benjamin S. Glicksberg, Eyal Zimlichman, Yiftach Barash, Robert Freeman, Alexander W. Charney, Girish N Nadkarni, and Eyal Klang. 2024. https://doi.org/10.1056/AIdbp2300040 Large language models are poor medical coders — benchmarking of medical code querying . NEJ...

2024 doi
[11]

Dukyong Yoon, Changho Han, Dong Won Kim, Songsoo Kim, SungA Bae, Jee An Ryu, and Yujin Choi. 2024. https://doi.org/10.2196/56614 Redefining health care data interoperability: Empirical exploration of large language models in information exchange . Journal of Medical Internet R...

2024 doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Olivier Bodenreider. 2004. https://doi.org/10.1093/nar/gkh061 The Unified Medical Language System ( UMLS ): Integrating biomedical terminology . Nucleic Acids Research, 32:D267--70

doi:10.1093/nar/gkh061 2004

[4] [4]

Boyle, Antanas Kascenas, Pat Lok, Maria Liakata, and Alison Q

Joseph S. Boyle, Antanas Kascenas, Pat Lok, Maria Liakata, and Alison Q. O'Neil. 2023. http://arxiv.org/abs/2310.06552 Automated clinical coding using off-the-shelf large language models . arXiv :2310.06552

arXiv 2023

[5] [5]

Kolbinger, Hannah Sophie Muti, Zunamys I

Jan Clusmann, Fiona R. Kolbinger, Hannah Sophie Muti, Zunamys I. Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia L \"o ffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, Sophia J. Wagner, and Jakob Nikolas Kather. 2023. https://doi.org/10.1038/s43856-023-00370-1 The future landscape of large language models ...

doi:10.1038/s43856-023-00370-1 2023

[6] [6]

Jack Gallifant, Shan Chen, Pedro Moreira, Nikolaj Munch, Mingye Gao, Jackson Pond, Leo Anthony Celi, Hugo Aerts, Thomas Hartvigsen, and Danielle Bitterman. 2024. http://arxiv.org/abs/2406.12066 Language models are surprisingly fragile to drug names in biomedical benchmarks . arXiv :2406.12066

arXiv 2024

[7] [7]

Chao-Wei Huang, Shang-Chi Tsai, and Yun-Nung Chen. 2022. https://doi.org/10.18653/v1/2022.clinicalnlp-1.2 PLM - ICD : Automatic ICD coding with pretrained language models . In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 10--20, Seattle, WA. Association for Computational Linguistics

doi:10.18653/v1/2022.clinicalnlp-1.2 2022

[8] [8]

Kellmann, Pauline Lanting, Lude Franke, Esther J

Alexander J. Kellmann, Pauline Lanting, Lude Franke, Esther J. van Enckevort, and Morris A. Swertz. 2023. https://doi.org/10.1093/database/baad019 Semi-automatic translation of medicine usage data (in Dutch , free-text) from lifelines COVID -19 questionnaires to ATC codes . Database, 2023:baad019

work page doi:10.1093/database/baad019 2023