{"id":"1e991ad7-4fe6-4d37-8d0a-85caa74c6675","arxiv_id":"2412.07743","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A five-level prompting method lets local LLMs assign ATC drug codes, reaching 78% exact matches with GPT-4o and 60% with Llama 3.1 70B on Health Canada product names, with a fine-tuned 8B model matching the 70B zero-shot result.","lead":"The authors automated ATC drug coding by prompting LLMs level by level through the WHO drug ontology, reaching 78% exact codes with GPT-4o and 60% with Llama 3.1 70B on Health Canada product names, while a fine-tuned 8B model matched the 70B's zero-shot accuracy. The work matters because it suggests privacy-sensitive health systems could run coding locally rather than sending clinical text to external APIs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 166-note Ontario Health subset may evade the paper's central claim: at finer ATC levels the excluded notes dominate coding difficulty, so the claimed 60% Llama and fine-tuned 8B results are not established for real free-text prescriptions.","rationale":"I read the paper in good faith. The methodological contribution — level-by-level prompting with constrained option sets, plus knowledge-grounding ablation — is coherent, and the relative comparison between GPT-4o, Llama 3.1 70B, and fine-tuned Llama 3.1 8B is a fair baseline setup. The authors are also transparent in Section 4.2 about the granularity metric and in Section 5 about the 166-note evaluation set. The central claim, however, is not just that the method works on curated product names; it is that automatic ATC coding is feasible on real clinical free-text prescriptions (abstract, introduction, conclusion). That claim rests on the Ontario Health numbers, and those numbers are computed on 166 of 200 notes, excluding exactly the ambiguous prescriptions that separate feasible deployment from manual expert review. The reader's weakest_assumption identified single-expert labels and the excluded 34 notes; I regard the exclusion issue as the more load-bearing of the two, because the granularity exclusion is implicitly a selection of easy cases. Even before questioning label quality, the paper does not report accuracy on the full 200-note set, and no reasonable rescoring of the 34 excluded notes will change the qualitative conclusion that the headline free-text accuracies are inflated relative to the full data distribution. The paper is still acceptable conditionally: the authors could fix the analysis by reporting a lower-bound accuracy on all 200 notes (treating level-0 notes as incorrect), by providing the per-granularity breakdown, and by reporting inter-annotator reliability on even a 50-note subsample. Absent that, the central real-world feasibility claim is not quantitatively supported for the most challenging notes. I also note the paper has no code/data release, but that is a reproducibility concern rather than a correctness concern. The recommended verdict is CONDITIONAL rather than REJECT because the underlying method has independent support from the Health Canada and RABBITS experiments and because the Ontario Health claim could be repaired with additional reporting rather than additional methodology.","tokens_in":7538,"tokens_out":2341,"duration_ms":19018,"concrete_test":"Recalculate the Ontario Health results with all 200 notes in the denominator and with granularity-sensitive per-note scoring: for each note, define accuracy_vs_granularity = 1 only if the model's prediction is correct at the note's annotated granularity level (and 0 for the 20 level-0 notes). If a second domain expert independently annotates granularity levels on a random sample of 50 of the 200 notes, compute Cohen's kappa for the level-5 vs. below-level-5 distinction. Report the resulting Llama 3.1 70B and fine-tuned 8B accuracies on the full 200-note set; if the recalculated accuracies fall below the reported 60% and 53.1% by the expected margin (approximately 60*166/200 = 49.8% and 53.1*166/200 = 44.1% before any correction for the expert's uncertainty), the central real-world free-text claim is not supported by the reported numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 states that all 200 Ontario Health notes are used for evaluation, but Section 5 reveals that only 166 of 200 notes are assigned granularity level 5 and are evaluated; the remaining 34 notes, including 20 scored at granularity level 0, are excluded from the accuracy numbers. The paper's abstract and conclusions claim real-world free-text feasibility, yet the headline 60% Llama 3.1 70B and 53.1% fine-tuned 8B numbers in Table 1 are computed only on the 166 unambiguous notes. Because the 20 notes at granularity level 0 are, by the authors' own granularity definition, prescriptions for which no confident ATC level can be determined even by a domain expert, keeping those notes in the denominator and assigning them an accuracy of zero would reduce the level-5 Ontario Health accuracy substantially: even under the most optimistic assumption that the model is correct on all 166 level-5 notes, the accuracy becomes at most 166/200 = 83%, and the reported numbers drop to 166*0.531/200 = 44.1% for the fine-tuned 8B and 166*0.494/200 = 41.0% for Llama 3.1 70B. This is not merely a reporting issue: the central feasibility claim for free-text clinical prescriptions depends on the excluded ambiguous notes, precisely the hard cases that decide whether local open-source models can replace manual expert coding. The single-expert granularity annotations (Section 4.2) are also unvalidated; if a second expert would assign level 5 to many of the 20 level-0 notes, the reported 60% is not the true accuracy on the full set. The paper provides no per-note error analysis on the excluded notes, so a reader cannot tell whether the exclusion reflects model limitations or genuine text ambiguity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a level-by-level prompting method for assigning WHO ATC drug codes to prescription text. The method asks an LLM to choose among valid ATC codes at each of the five ontology levels, optionally grounding each choice with generic drug names or UMLS definitions. The authors evaluate GPT-4o and Llama 3.1 70B in zero-shot settings and a fine-tuned Llama 3.1 8B on three datasets: Health Canada product names, the RABBITS product-name benchmark, and 200 real clinical prescription notes from Ontario Health. The paper reports 78.4% exact-match accuracy with GPT-4o and 60.3% with Llama 3.1 70B on Health Canada, and claims that a fine-tuned Llama 3.1 8B matches or surpasses zero-shot Llama 3.1 70B accuracy.","tokens_in":7944,"tokens_out":5776,"duration_ms":52571,"significance":"If the results are stated accurately, the paper makes a useful practical contribution: it demonstrates that a locally deployable open-source model can perform ATC coding on drug mentions without sending clinical data to external APIs. The evaluation is grounded in an external ATC ontology and uses expert-verified labels, and the Code Only ablation controls for the possibility that listing generic names in the prompt simply leaks the answer. The focus on privacy-preserving deployment and the granularity-aware evaluation of ambiguous clinical notes are also strengths. However, the manuscript currently overstates the breadth of the headline numbers and excludes the most difficult real-world notes from the main exact-match results, so the central feasibility claim is not yet fully supported.","major_comments":[{"comment":"Section 4.1 states that 'All 200 prescription notes are used for evaluation', but Section 5 reports results only for the 166 Ontario Health notes with granularity level 5, and the Table 1 caption confirms this. The 34 excluded notes include 20 at granularity level 0. If the excluded notes are counted as incorrect, the Ontario Health level-5 accuracy becomes at most 41.0% for Llama 3.1 70B and 44.1% for the fine-tuned 8B instead of the reported 49.4% and 53.1%. Because the abstract and conclusion claim feasibility on real free-text clinical prescriptions, this exclusion is load-bearing; the paper must either evaluate all 200 notes under a clearly specified rule or explicitly subordinate the real-world claim to the unambiguous subset.","section":"Section 5, Table 1"},{"comment":"The abstract presents the 78% and 60% figures as if they were general results across all datasets, but Table 1 shows that these are Health Canada level-5 numbers. On RABBITS, exact-match accuracy is 39.4% for GPT-4o and 19.8% for Llama 3.1 70B, and on Ontario Health the zero-shot Llama 3.1 70B number is 49.4%. The summary should report per-dataset results and avoid implying dataset-independent performance.","section":"Abstract, Section 1"},{"comment":"The claim that fine-tuned Llama 3.1 8B 'consistently surpasses' zero-shot Llama 3.1 70B is based on point differences with no uncertainty quantification. On Health Canada the exact-match scores are 60.5% versus 60.3%, a difference of 0.2 percentage points; only the RABBITS difference (26.4% versus 19.8%) is substantial. Without confidence intervals, bootstrapping, or repeated runs, the comparison is not established and the claim should be moderated.","section":"Section 5, Fine-tuning Effectiveness"},{"comment":"The gold-standard ATC labels and granularity levels for the Ontario Health notes come from a single domain expert, with no inter-annotator reliability measure. Because the granularity assignment determines which notes enter the main real-world evaluation, a second annotator or a sensitivity analysis is needed to demonstrate that the 166-note denominator is stable rather than a source of selection bias.","section":"Section 4.1, Section 4.2"}],"minor_comments":[{"comment":"The title contains an extra space in 'A TC'; it should read 'Zero-Shot ATC Coding'.","section":"Title"},{"comment":"The sentence 'All 200 prescription notes are used for evaluation' directly contradicts the Section 5 statement that only the 166 level-5 notes are evaluated; the wording should be revised to describe the granularity-aware evaluation exactly.","section":"Section 4.1"},{"comment":"The text says that the Code Only setting achieves 'meaningful accuracy', but the RABBITS level-5 Code Only result is 8.4%; this description seems overstated for that dataset.","section":"Section 5, Table 2"},{"comment":"The claim that this is 'the first attempt to automate ATC coding using LLMs' is presented without a systematic literature search beyond the cited ICD work; the novelty claim should be softened or supported with a broader comparison.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a usable empirical core, but the abstract and the real-world feasibility claims are not currently matched by the reported experiments. The most important issue is the 166-note denominator for Ontario Health: the authors' own granularity design selects the easiest 83% of the notes, and the excluded ambiguous notes are exactly the cases that decide whether local models can replace manual coding. I would welcome a revision that reports the full-200 analysis or clearly and consistently limits the claim to unambiguous notes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on the ATC coding paper. The core idea is sound, and it's a genuinely useful first step: framing ATC coding as level-by-level selection among valid options, rather than free-form generation, is the right call and clearly builds on Boyle et al. The empirical work is honest in places — the RABBITS results are much weaker than Health Canada, and they dig into why (string similarity), which is a nice insight. The finding that a fine-tuned 8B model matches zero-shot 70B on these tasks is practically relevant for privacy-preserving deployment. I have no issue with the method or the evaluation logic inside each table.\n\nThe soft spots are real, and they're mostly about framing. The abstract and introduction say the method 'achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B' without saying these are Health Canada numbers. Table 1 shows RABBITS level-5 at 19.8% for 70B and Ontario Health at 49.4%. That's a substantial overstatement. Worse, the Ontario Health numbers are only on the 166 of 200 notes judged unambiguous at level 5. The 20 notes at granularity level 0 are exactly the hard free-text cases where an expert couldn't determine a code; omitting them inflates the 'real-world' claim. The stress-test's arithmetic is right: if you put those 34 notes back in as failures, the level-5 numbers drop to roughly 44% for the fine-tuned 8B and 41% for Llama 3.1 70B, with a ceiling of 83% even for perfect coding on the unambiguous subset. That doesn't kill the paper, but the central feasibility claim needs to be restated as 'works on the codeable subset, and here's how often a note is codeable.'\n\nAlso worth noting: the gold annotations and granularity levels come from a single expert, with no agreement check. There's no rule-based baseline, which would have been easy to run given that 43% of Health Canada names are substrings of generic names — a string matcher might be strong there. No code or data released, though they say they hope to release the 200-pair set. These are minor-to-moderate issues; the abstract overstatement is the big one.\n\nWho's this for? Applied health NLP researchers, and anyone building local LLM pipelines for regulated data. It's worth reading because the method is simple and the failure points are instructive. It deserves peer review, but only after the authors fix the abstract, report both full-set and subset numbers, and ideally add a rule-based comparison.\n\nMy recommendation: send it to review, with a clear request for revision.","headline":"A useful first paper on LLM-based ATC coding, but the abstract's accuracy numbers are overstated and the Ontario Health results only cover the codeable 166 of 200 notes.","tokens_in":8473,"tokens_out":5161,"would_cite":false,"duration_ms":45637,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatic ATC drug coding is feasible with locally deployable open-source language models when the task is reframed as five levels of constrained multiple-choice selection among valid codes.","keywords":["ATC coding","large language models","hierarchical classification","level-by-level prompting","clinical free text","zero-shot learning","privacy-preserving NLP","medical coding"],"falsifier":"Have two independent clinicians re-label the 200 real prescription notes, including the 34 excluded as ambiguous, using the same granularity scale, then score the models against the adjudicated labels; if expert agreement is low or the models' accuracy on the previously excluded notes differs sharply from the reported 60%, the claimed real-world feasibility is not reproducible.","tokens_in":7380,"feed_emoji":"💊","tokens_out":7285,"duration_ms":61508,"temperature":0.7,"pith_summary":"The paper sets out to show that assigning WHO Anatomical Therapeutic Chemical (ATC) codes to drug mentions can be automated with large language models without sending clinical data to outside services. Its central move is to treat coding not as free-form code generation but as a five-level multiple-choice walk down the ATC hierarchy, forcing the model to pick from valid codes at each level. On two curated product-name benchmarks and on 166 unambiguous real prescription notes, this reaches 78% exact-match accuracy with GPT-4o and 60% with Llama 3.1 70B in the zero-shot setting. The paper further claims that a fine-tuned 8B Llama model matches the zero-shot 70B accuracy, which matters because an 8B model is small enough to run on local infrastructure in privacy-sensitive health care.","feed_headline":"Small local language model matches a 70B model at drug coding","feed_subtitle":"A five-level multiple-choice prompt keeps clinical drug notes private and still assigns WHO codes accurately enough for real use.","key_machinery":"The machinery is level-by-level prompting through the ATC ontology: the model receives the drug mention and a list of valid codes for the current level, each with its generic name or UMLS definition, and must output exactly one option; its choice determines the candidate set for the next level, repeated five times until a seven-character code is produced. This constrains the output space to at most 37 options, with an average of about 5, instead of the full 6,807-code space, and eliminates code fabrication. Knowledge grounding ablated three option formats, code only, code plus generic name, and code plus UMLS definition, showing that names are the main driver of accuracy and UMLS definitions add a small deeper-level boost.","core_discovery":"The discovery is that ATC coding reduces to hierarchical selection: asking the model at each of the five ATC levels to choose one option out of the valid children of the previously chosen code prevents it from inventing non-existent codes and keeps each decision small, with roughly five choices on average instead of a full 6,807-code space. Guided this way, zero-shot models perform well on familiar product names and on real prescription notes, which the authors find are often variants of product names; performance drops sharply on a benchmark with low lexical overlap between product and generic names. Grounding options with UMLS definitions gives modest gains at the deepest levels. The paper's headline capacity claim is that a fine-tuned Llama 3.1 8B reaches the same accuracy as zero-shot Llama 3.1 70B, making local, privacy-preserving deployment feasible.","pith_inferences":["The five-level multiple-choice framing is a general recipe: any hierarchical coding scheme with bounded branching, such as ICD, SNOMED, or RxNorm ingredient hierarchies, could get the same fabrication-free reduction, and the paper's ablation suggests lexical overlap, not model size, will be the bottleneck.","The granularity-level annotation points toward confidence-aware coding: a production system could return the deepest defensible code and flag ambiguous notes for human review instead of forcing a level-5 guess.","A testable extension of the string-similarity explanation is that per-code substring overlap between product and generic names should predict per-code correctness; if it does not, the proposed failure mechanism is incomplete.","Independent replication on a second site's prescription corpus would show whether fine-tuned-8B parity with zero-shot 70B generalizes beyond the specific training split used here."],"forward_implications":["If the parity result holds, privacy-sensitive organizations can run ATC coding on an 8B open-source model locally and get the accuracy of a 70B zero-shot model.","In the authors' real-world notes, 166 of 200 prescriptions could be confidently assigned exact level-5 codes, and the open-source models scored above 85% at level 3 on those notes.","Because models succeed where product names resemble generic names, improving performance on sets with low brand-generic overlap is the identified next bottleneck; the authors suggest hybrid rule-based and LLM systems.","Adding UMLS definition grounding produces small but real gains at level 5, so richer per-option context can be used without changing the prompting structure."],"supporting_citations":[{"why":"Supplies the hierarchical level-by-level prompting method that the paper adapts from ICD to ATC coding.","marker":"Boyle et al. (2023)"},{"why":"Provides the RABBITS benchmark of expert-verified product-name and ATC-code pairs used as a test set and for fine-tuning.","marker":"Gallifant et al. (2024)"},{"why":"Defines UMLS, the source of the drug definitions used for knowledge grounding.","marker":"Bodenreider (2004)"},{"why":"Shows direct code generation by LLMs is unreliable, motivating the constrained selection design.","marker":"Soroush et al. (2024)"},{"why":"Provides background evidence that fine-tuning pretrained language models improves ICD coding, the prior art this work builds on.","marker":"Huang et al. (2022)"},{"why":"Provides background evidence for LLMs translating between ontological frameworks, another ICD-era technique the paper cites.","marker":"Yoon et al. (2024)"},{"why":"Describe the existing manual, rule-based, and hybrid ATC coding practices that this work aims to automate.","marker":"Pang et al. (2015); Kellmann et al. (2023)"},{"why":"Supports the premise that LLMs hold enough drug and therapeutic knowledge to benefit from knowledge grounding.","marker":"Clusmann et al. (2023)"}],"fun_headline_variants":["Fine-tuned Llama 8B ties 70B on WHO drug codes","Five-choice prompts let smaller LLM match 70B drug-coding","Local Llama 8B reaches same ATC accuracy as 70B","Hierarchical ATC prompts: 8B model equals 70B, cuts errors","Small LLM matches 70B at drug coding, stays on-premise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All real-world accuracy figures rest on one domain expert's ATC labels and ambiguity-level annotations for the 200 real prescription notes, with 34 notes excluded as too ambiguous, so a single labeling mistake or a debatable exclusion shifts the headline numbers.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Llama 8B ties 70B on WHO drug codes","Five-choice prompts let smaller LLM match 70B drug-coding","Local Llama 8B reaches same ATC accuracy as 70B","Hierarchical ATC prompts: 8B model equals 70B, cuts errors","Small LLM matches 70B at drug coding, stays on-premise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000365,"raw_usage":{"total_tokens":1966,"prompt_tokens":952,"completion_tokens":1014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":912}},"tokens_in":568,"tokens_out":1014,"duration_ms":8278,"temperature":1.0,"reasoning_tokens":912,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:31:47.646631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent clinicians re-label the 200 real prescription notes, including the 34 excluded as ambiguous, using the same granularity scale, then score the models against the adjudicated labels; if expert agreement is low or the models' accuracy on the previously excluded notes differs sharply from the reported 60%, the claimed real-world feasibility is not reproducible.","supporting_citations":[{"cited_title":"van der Velde, Nynke Smidt, Rolf Sijmons, Hans Hillege, and Morris A","cited_arxiv_id":null,"evidence_quote":"Describe the existing manual, rule-based, and hybrid ATC coding practices that this work aims to automate."}],"review_version":1}