Pith. sign in

REVIEW 5 major objections 7 minor 39 references

Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding

T0 review · 5 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ALIGN, a three-stage LLM coding system, assigns ATC medication codes with 72-73% accuracy and defers uncertain cases to reach about 90%.

desk verdict A well-engineered zero-shot medical coding system whose headline numbers all depend on a noisy majority-vote gold standard that the paper itself documents as inconsistent. read the letter →

arxiv 2411.13163 v2 pith:Y5Y7EUFT submitted 2024-11-20 cs.LG

Nabeel Seedat , Caterina Tozzi , Andrea Hita Ardiaca , Mihaela van der Schaar , James Weatherall , Adam Taylor This is my paper · ORCID
classification cs.LG
keywords medicalcodingATCclassificationMedDRAlargelanguagemodelszero-shotlearninguncertaintyquantificationhuman-in-the-loopclinicaltrialdatainteroperability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ALIGN, a zero-shot system that automates medical coding of historical clinical trial data by composing several large language model steps. It claims ALIGN outperforms direct prompting and retrieval-augmented baselines on ATC medication coding, especially at the most specific chemical level, while matching RAG on MedDRA medical-history coding. It also claims ALIGN's uncertainty estimates let a human reviewer take over the hardest cases, raising accuracy to about 90% with 30% deferral. If true, ALIGN would shorten the months-long manual harmonization needed to combine trial datasets and make cross-trial reuse practical.

What carries the argument

The load-bearing mechanism is a three-stage composition: (1) candidate generation by dense retrieval, BM25 sparse retrieval, and LLM reasoning with query contextualization and synonym generation; (2) self-evaluation that filters candidates through natural-language entailment against official ATC/MedDRA descriptions; (3) confidence scoring that reformulates coding as a multiple-choice question, uses logit biasing to constrain answers, extracts token log-probabilities, applies temperature-scaled softmax, and estimates uncertainty as predictive entropy. The entropy threshold drives deferral to human reviewers, and the authors note the same confidence scores can feed conformal prediction for coverage guarantees.

What would settle it

Take a random sample of coded medication and medical-history terms from these 22 trials, have independent expert coders re-adjudicate the majority-vote labels without seeing them, and compare ALIGN's predictions against that fresh gold standard; if accuracy on ATC Level 4 falls below the reported 72-73% or the 7-22% margin over baselines disappears, the central claim is undermined.

Watch

Extended reading notes

Core claim

The paper's central claim is that a compositional, multi-step LLM pipeline can code clinical trial terms accurately in a zero-shot setting without labeled examples. For ATC medication codes, ALIGN achieves 72-73% overall accuracy at Level 4 and 86-89% on the most common medications, beating LLM and RAG baselines by 7-22 percentage points; for MedDRA it matches RAG at all hierarchy levels and reaches 87-90% at the finest reported level. The system's predictive entropy flags uncertain predictions, and simulated deferral of about 30% of cases to human coders brings ATC accuracy to roughly 90%, with the largest gains on uncommon medications.

Load-bearing premise

The evaluation assumes that the majority vote of the pre-existing codes in the trial data is the correct gold standard, even though the paper documents that codes for the same term vary within and between studies.

Editorial extensions

If this is right

  • ATC coding at the most specific chemical level improves from roughly 50-66% for baselines to 72-73% overall, making automated coding usable for common medications.
  • Common medications reach 86-89% accuracy at ATC Level 4, so the majority of real prescriptions can be coded automatically with high confidence.
  • With 30% human deferral, accuracy approaches 90%, and deferral helps most on uncommon medications; this gives a practical human-in-the-loop workflow.
  • Cost of $0.0007 per code with a cost-efficient backbone makes the pipeline cheap enough for large-scale application to hundreds of thousands of records.
  • The zero-shot design means the same system can be pointed at new coding dictionaries or therapeutic areas without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the accuracy holds on independently re-adjudicated labels, ALIGN-style compositional coding could generalize to ICD-10 and adverse-event coding, where one-to-many mappings are even more common.
  • The self-evaluation step's heavy pruning (31-53% of candidates) suggests retrieval-only pipelines waste most of their candidates; composition may be needed whenever coding decisions hinge on route, dose, or indication rather than name matching.
  • The entropy-based deferral curve implies a deployment recipe: set the deferral threshold by available expert hours, not by a fixed accuracy target, since the gain per deferred case is concentrated in the long tail.
  • The conformal prediction extension, if applied with a small labeled calibration set, could turn ALIGN's confidence scores into set-valued predictions with a formal coverage guarantee, a step the paper describes but does not evaluate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces ALIGN, a compositional large language model (LLM) system for zero-shot medical coding of concomitant medications into ATC codes and medical history terms into MedDRA codes. ALIGN operates in three stages: diverse candidate generation (dense retrieval, BM25, and LLM reasoning), LLM self-evaluation via natural-language-inference filtering against external code descriptions, and confidence scoring with entropy-based uncertainty estimation to support human deferral. The system is evaluated on 22 immunology trials (10 RA, 12 SLE) and compared against vanilla LLM prompting, chain-of-thought prompting, and retrieval-augmented generation (RAG). The authors report that ALIGN matches RAG on MedDRA coding and outperforms all baselines on ATC coding, particularly at the more specific ATC levels, and that uncertainty-based deferral improves accuracy to about 90% at 30% deferral. The paper also provides cost estimates of $0.0007 and $0.02 per code for GPT-4o-mini and GPT-4o backbones, respectively.

Significance. If the reported results are reliable, ALIGN is a practically useful zero-shot system for harmonizing historical clinical trial data, with a meaningful contribution in combining retrieval and LLM reasoning with explicit verification and uncertainty-driven human-in-the-loop deferral. Strengths of the paper include the compositional system design, the use of external grounding for self-evaluation, repeated experiments (five repeats) with mean and standard deviation, and a concrete cost analysis. The main risk is the evaluation's dependence on a noisy majority-vote gold standard, which the paper itself documents as internally and externally inconsistent. Because the central claim is comparative (ALIGN outperforms RAG on ATC), the label noise issue is load-bearing and must be addressed before the quantitative conclusions can be accepted.

major comments (5)
  1. [Dataset construction; Table S1] The gold standard is the majority of pre-existing codes, but Table S1 documents the same term receiving different codes within and across studies (e.g., Anovlar coded as G03AA, G03AB, or G03FA) and legitimate one-to-many mappings (e.g., acetylcysteine assigned R05CB01, S01XA08, or V03AB23 depending on context). No inter-annotator agreement is reported, so for many terms the majority label may be arbitrary. Since every accuracy figure in Tables 1-2 and the deferral simulation in Figure 3 is scored against this label, label noise is load-bearing. Please report the distribution of label agreement and either perform sensitivity analyses treating any of the observed codes as correct, or validate on a manually adjudicated subset.
  2. [Dataset construction; Figure 1] The evaluation set consists only of records with pre-existing codes, while the stated motivation is missing codes (Figure 1). Missingness may be non-random, so the reported accuracies may not transfer to the missing-code population. In addition, the deferral simulation (Figure 3) assumes deferred cases are corrected to the majority-vote gold standard, meaning it measures agreement with a potentially noisy label rather than clinical truth. Please address this population mismatch or explicitly state it as a limitation.
  3. [Abstract; Results, Table 2] The abstract claims ALIGN 'outperforming baselines by 7-22%' for ATC coding, but the tables do not clearly identify the comparison supporting this range. For ATC level 4 overall, ALIGN is about 7 percentage points above RAG (e.g., 72.20 vs 65.27 in Table 2a for RA), and the 22% figure does not match any reported margin. Please specify the exact baseline and metric used for the 7-22% claim and report effect sizes with confidence intervals.
  4. [Results, Table 2 (ALIGN (Set))] The set-based coverage results (ALIGN (Set)) are reported without the average prediction set size. Without this information, high coverage is not interpretable, since a set containing many candidate codes would trivially achieve high coverage. Please report the average number of codes in the prediction set (or a size distribution) alongside the set-based accuracy.
  5. [Results, Table 2(b)-(c)] The common/uncommon split is defined on occurrences (Pareto 80/20), but the evaluation is performed on unique query-code pairs; the number of unique pairs in each subset is not reported. The 'common medications' subset could be very small, and the reported 86-89% accuracies may have wide confidence intervals that the five-repeat standard deviation does not capture. Please report the number of unique pairs in each subset.
minor comments (7)
  1. [Introduction] In the second paragraph, 'standardized medical coding systems play are essential' is ungrammatical; should read 'play an essential role' or 'are essential'.
  2. [Figure 1] The caption refers to 'RA vs SLR'; the intended abbreviation appears to be 'SLE'.
  3. [Abstract; Results] The phrase '±90% accuracy' appears in the abstract and Results; the plus-minus symbol is likely a typo for 'to 90%'.
  4. [Supplementary, Table S2] Table S2 column headers list the MedDRA levels as 'SOC HLT HLGT', while Table 1 uses 'SOC HLGT HLT'; please make the ordering consistent.
  5. [Results, Cost efficiency] The cost figures ($0.0007 and $0.02 per code) are quoted without details of token usage or the cost model; please clarify how these values were computed.
  6. [Supplementary, Conformal Prediction] The conformal prediction extension is described but not evaluated. If it is presented as a feature of ALIGN, please include at least a small worked example or state clearly that it is a proposal without empirical validation.
  7. [Experimental Setup] The statement that temperature 0.7 showed 'no sensitivity' is not supported by a reported sensitivity analysis; please include such an analysis or soften the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ALIGN is evaluated against an external majority-vote benchmark, and no fitted quantity is renamed as a prediction.

full rationale

Walking the derivation chain, ALIGN's outputs are compared to externally defined ATC/MedDRA code labels constructed by majority vote from pre-existing human codings, not to any function of ALIGN's own outputs or parameters. The system's components (dense retrieval, BM25, LLM reasoning, self-evaluation via entailment against ATC/DDD and MedDRA descriptions, MCQ logit scoring) are all defined independently of the evaluation labels; no parameter is fitted to the test set. Hyperparameters (k=10, logit bias 100, temperature 0.7, and the softmax temperature scaling) are stated as fixed choices rather than learned from the benchmark, and the deferral analysis is explicitly a simulation that assumes correct human annotation for deferred cases, with the entropy-based ranking compared against random deferral rather than against an oracle. The 'ALIGN (Set)' results are prediction-set coverage, not a hidden reuse of the labels. The self-citations (refs 8, 9, and 15) appear only as background context and are not load-bearing: no uniqueness theorem or prior-work ansatz is imported to force ALIGN's design or conclusions. The paper's own Table S1 and the Discussion's limitation paragraph do document real threats to validity, notably that the majority-vote gold standard inherits intra- and inter-study coding variability, so absolute accuracies and baseline gaps could be affected by label noise, and that performance on uncommon codes remains limited. However, those are evaluation-quality and generalizability concerns, not circularity of derivation: no equation, fitted parameter, or self-citation chain is shown to reduce a claimed prediction to its own input.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rely on a small number of hand-set hyperparameters (top-k, logit bias, softmax temperature, LLM temperature, Pareto split) and on domain assumptions about the validity of coded ground truth, the reliability of LLM entailment verification, and the calibration of token-level probabilities. No invented physical or conceptual entities are introduced; ALIGN is a pipeline of existing components. The most consequential assumption is that majority-vote pre-existing codes are a trustworthy gold standard, since the paper itself documents coding variability.

free parameters (5)
  • top-k retrieval candidates = 10
    Number of candidate codes retrieved by dense and BM25 retrieval; chosen by hand, not tuned, but affects candidate diversity and cost.
  • logit bias magnitude = 100
    Fixed value added to logits of MCQ option tokens to constrain output; chosen by hand.
  • softmax temperature for confidence scaling = not reported
    The paper states temperature scaling is applied to prediction-set log-probabilities but does not report the temperature value or how it was selected.
  • LLM sampling temperature = 0.7
    Generation temperature for all experiments; authors state they 'found no sensitivity' but do not show the sweep.
  • common/uncommon Pareto split = 80/20
    Threshold defining common medications as 80% of occurrences; choice affects the 'common' accuracy results.
assumptions (5)
  • domain assumption The ATC/DDD and MedDRA dictionaries used for indexing and grounding are authoritative and complete for the coding tasks.
    Self-evaluation grounds candidate codes by scraping these sources; if the dictionaries are incomplete for uncommon drugs (e.g., Epikajutsuto), accuracy is capped.
  • domain assumption Majority-vote pre-existing codes constitute a valid ground truth for accuracy evaluation.
    Dataset construction selects the most frequently assigned ATC/MedDRA code for each unique term as the label, despite Table S1 documenting substantial intra- and inter-study coding variability.
  • domain assumption LLM textual entailment judgments (supported/not supported) reliably identify valid codes.
    The self-evaluation stage filters candidates based on LLM NLI predictions; no validity analysis of the NLI filter itself is provided beyond its effect on final accuracy.
  • domain assumption Token-level log-probabilities from the LLM are well-calibrated confidence estimates for this task.
    The confidence scoring relies on cited evidence that LLM token-level probabilities are calibrated; no task-specific calibration analysis is reported.
  • domain assumption Five repeated runs at temperature 0.7 capture the stochastic variability of the systems.
    All results are reported as mean plus/minus standard deviation over five runs; no significance testing or assessment of between-run stability across the full pipeline is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding." pith.science (2026). https://pith.science/paper/Y5Y7EUFT

@misc{pith2026241113163,
  author       = {Pith},
  title        = {Pith review of: Unlocking Historical Clinical Trial Data with ALIGN: A Compositional Large Language Model System for Medical Coding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5Y7EUFT}},
  note         = {Machine review of arXiv:2411.13163}
}
abstract

The reuse of historical clinical trial data has significant potential to accelerate medical research and drug development. However, interoperability challenges, particularly with missing medical codes, hinders effective data integration across studies. While Large Language Models (LLMs) offer a promising solution for automated coding without labeled data, current approaches face challenges on complex coding tasks. We introduce ALIGN, a novel compositional LLM-based system for automated, zero-shot medical coding. ALIGN follows a three-step process: (1) diverse candidate code generation; (2) self-evaluation of codes and (3) confidence scoring and uncertainty estimation enabling human deferral to ensure reliability. We evaluate ALIGN on harmonizing medication terms into Anatomical Therapeutic Chemical (ATC) and medical history terms into Medical Dictionary for Regulatory Activities (MedDRA) codes extracted from 22 immunology trials. ALIGN outperformed the LLM baselines, while also providing capabilities for trustworthy deployment. For MedDRA coding, ALIGN achieved high accuracy across all levels, matching RAG and excelling at the most specific levels (87-90% for HLGT). For ATC coding, ALIGN demonstrated superior performance, particularly at lower hierarchy levels (ATC Level 4), with 72-73% overall accuracy and 86-89% accuracy for common medications, outperforming baselines by 7-22%. ALIGN's uncertainty-based deferral improved accuracy by 17% to 90% accuracy with 30% deferral, notably enhancing performance on uncommon medications. ALIGN achieves this cost-efficiently at \$0.0007 and \$0.02 per code for GPT-4o-mini and GPT-4o, reducing barriers to clinical adoption. ALIGN advances automated medical coding for clinical trial data, contributing to enhanced data interoperability and reusability, positioning it as a promising tool to improve clinical research and accelerate drug development.

Figures

Figures reproduced from arXiv: 2411.13163 by the authors.

Figure 1
Figure 1. Missingness of codes is pervasive across different trials: There is significant variability in the presence of standardized medical codes (ATC and MedDRA) across both Rheumatoid Arthritis (RA) and Systematic Lupus Erythematosus (SLE) studies. The missingness of codes in studies can be seen both within a therapeutic indication (RA vs SLR) and across indications, highlighting the need for automated medical coding solu… view at source ↗
Figure 2
Figure 2. ALIGN is a compositional Large Language Model (LLM) system for medical coding. ALIGN takes a receives a query such as medication name or medical history and proceeds through three main stages: (1) Diverse candidate code generation, (2) LLM self-evaluation and code filtering to automatically filter spurious and inconsistent codes; and (3) Confidence scoring and uncertainty estimation, which provides a confidence scor… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 23 canonical work pages

  1. [1]

    Minimizing Patient Burden Through the Use of Historical Subject-Level Data in Innovative Confirmatory Clinical Trials: Review of Methods and Opportunities

    Lim J, Walley R, Yuan J, et al. Minimizing Patient Burden Through the Use of Historical Subject-Level Data in Innovative Confirmatory Clinical Trials: Review of Methods and Opportunities. Ther Innov Regul Sci. 2018;52(5):546-559. doi:10.1177/2168479018778282

    work page doi:10.1177/2168479018778282 2018

  2. [2]

    Clinical Trial Data Sharing for COVID-19–Related Research

    Dron L, Dillman A, Zoratti MJ, Haggstrom J, Mills EJ, Park JJH. Clinical Trial Data Sharing for COVID-19–Related Research. J Med Internet Res. 2021;23(3):e26718. doi:10.2196/26718

    work page doi:10.2196/26718 2021

  3. [3]

    Sharing Historical Trial Data to Accelerate Clinical Development

    Yin PT , Desmond J, Day J. Sharing Historical Trial Data to Accelerate Clinical Development. Clin Pharmacol Ther. 2019;106(6):1177-1178. doi:10.1002/cpt.1608

    work page doi:10.1002/cpt.1608 2019

  4. [4]

    Unlocking the potential of patient data through responsible sharing – has anyone seen my keys? Med Writ

    Cullinan P , Roberts L. Unlocking the potential of patient data through responsible sharing – has anyone seen my keys? Med Writ. 2020;29:46-51

    work page 2020

  5. [5]

    Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact

    Balagopalan A, Baldini I, Celi LA, et al. Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact. PLOS Digit Health. 2024;3(4):e0000474. doi:10.1371/journal.pdig.0000474

    work page doi:10.1371/journal.pdig.0000474 2024

  6. [6]

    Why digital medicine depends on interoperability

    Lehne M, Sass J, Essenwanger A, Schepers J, Thun S. Why digital medicine depends on interoperability. Npj Digit Med. 2019;2(79). Accessed August 13, 2024. https://www.nature.com/articles/s41746-019-0158-1

    work page 2019

  7. [7]

    Augmented non-hallucinating large language models as medical information curators

    Gilbert S, Kather JN, Hogan A. Augmented non-hallucinating large language models as medical information curators. Npj Digit Med. 2024;7(1):1-5. doi:10.1038/s41746-024- 01081-0

    doi:10.1038/s41746-024- 2024

  8. [8]

    Navigating Data-Centric Artificial Intelligence With DC-Check: Advances, Challenges, and Opportunities

    Seedat N, Imrie F, Schaar M van der. Navigating Data-Centric Artificial Intelligence With DC-Check: Advances, Challenges, and Opportunities. IEEE Trans Artif Intell. 2024;5(6):2589-2603. doi:10.1109/TAI.2023.3345805

    arXiv 2024

Show all 39 references
  1. [9]

    Data science in pharmaceutical R&D: the DISRUPT-DS industry roundtable

    Khan NS, Senderovitz T , Weatherall J, et al. Data science in pharmaceutical R&D: the DISRUPT-DS industry roundtable. Nat Rev Drug Discov. Published online July 15, 2024. doi:10.1038/d41573-024-00104-5

    2024 doi

  2. [10]

    Accessed September 23, 2024

    Anatomical Therapeutic Chemical (ATC) Classification. Accessed September 23, 2024. https://www.who.int/tools/atc-ddd-toolkit/atc-classification

    2024

  3. [11]

    The Medical Dictionary for Regulatory Activities (MedDRA)

    Brown EG, Wood L, Wood S. The Medical Dictionary for Regulatory Activities (MedDRA). Drug Saf. 1999;20(2):109-117. doi:10.2165/00002018-199920020-00002

    1999 doi

  4. [12]

    PaLM: Scaling Language Modeling with Pathways

    Chowdhery A, Narang S, Devlin J, et al. PaLM: Scaling Language Modeling with Pathways. arXiv.org. April 5, 2022. Accessed August 18, 2024. https://arxiv.org/abs/2204.02311v5

    2022 arXiv

  5. [13]

    Large language models encode clinical knowledge

    Singhal K, Azizi S, Tu T , et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. doi:10.1038/s41586-023-06291-2 15

    2023 doi

  6. [14]

    Large Language Models as Optimizers

    Yang C, Wang X, Lu Y , et al. Large Language Models as Optimizers. In: ; 2023. Accessed August 18, 2024. https://openreview.net/forum?id=Bb4VGOWELI

    2023

  7. [15]

    Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes

    Seedat N, Huynh N, Breugel B van, Schaar M van der. Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes. In: ; 2024. Accessed August 19, 2024. https://openreview.net/forum?id=9cG1oRnqNd

    2024

  8. [16]

    Accessed September 23, 2024

    ICD-10 Version:2019. Accessed September 23, 2024. https://icd.who.int/browse10/2019/en

    2019

  9. [17]

    Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying

    Soroush A, Glicksberg BS, Zimlichman E, et al. Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying. NEJM AI. 2024;1(5). doi:10.1056/AIdbp2300040

    2024 doi

  10. [18]

    Benchmarking Large Language Models for Extraction of International Classification of Diseases Codes from Clinical Documentation

    Simmons A, Takkavatakarn K, McDougal M, et al. Benchmarking Large Language Models for Extraction of International Classification of Diseases Codes from Clinical Documentation. Published online May 3, 2024:2024.04.29.24306573. doi:10.1101/2024.04.29.24306573

    2024 doi

  11. [19]

    Can Large Language Models abstract Medical Coded Language? Published online June 6, 2024

    Lee SA, Lindsey T. Can Large Language Models abstract Medical Coded Language? Published online June 6, 2024. doi:10.48550/arXiv.2403.10822

    2024 doi arXiv

  12. [20]

    CliBench: Multifaceted Evaluation of Large Language Models in Clinical Decisions on Diagnoses, Procedures, Lab Tests Orders and Prescriptions

    Ma MD, Ye C, Yan Y , et al. CliBench: Multifaceted Evaluation of Large Language Models in Clinical Decisions on Diagnoses, Procedures, Lab Tests Orders and Prescriptions. Published online June 14, 2024. doi:10.48550/arXiv.2406.09923

    2024 doi arXiv

  13. [21]

    Redefining Health Care Data Interoperability: Empirical Exploration of Large Language Models in Information Exchange

    Yoon D, Han C, Kim DW, et al. Redefining Health Care Data Interoperability: Empirical Exploration of Large Language Models in Information Exchange. J Med Internet Res. 2024;26:e56614. doi:10.2196/56614

    2024 doi

  14. [22]

    Large language models are good medical coders, if provided with tools

    Kwan K. Large language models are good medical coders, if provided with tools. Published online July 6, 2024. doi:10.48550/arXiv.2407.12849

    2024 doi arXiv

  15. [23]

    Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendation

    Wang X, Mercer R, Rudzicz F. Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendation. In: Duh K, Gomez H, Bethard S, eds. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag...

    2024 doi

  16. [24]

    The Shift from Models to Compound AI Systems

    Gupta R, Ghodsi MZ Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, Ali. The Shift from Models to Compound AI Systems. The Berkeley Artificial Intelligence Research Blog. Accessed September 2...

    2024

  17. [25]

    BM25S: Orders of magnitude faster lexical search via eager sparse scoring

    Lù XH. BM25S: Orders of magnitude faster lexical search via eager sparse scoring. Published online July 4, 2024. doi:10.48550/arXiv.2407.03618 16

    2024 doi arXiv

  18. [26]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Wei J, Wang X, Schuurmans D, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In: ; 2022. Accessed August 18, 2024. https://openreview.net/forum?id=_VjQlMeSB_J

    2022

  19. [27]

    Accessed October 1, 2024

    ATCDDD - ATC/DDD Index. Accessed October 1, 2024. https://atcddd.fhi.no/atc_ddd_index/

    2024

  20. [28]

    Accessed October 1, 2024

    MedDRA. Accessed October 1, 2024. https://www.meddra.org/

    2024

  21. [29]

    A large annotated corpus for learning natural language inference

    Bowman SR, Angeli G, Potts C, Manning CD. A large annotated corpus for learning natural language inference. In: Màrquez L, Callison-Burch C, Su J, eds. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguis...

    2015 doi

  22. [30]

    Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification

    Sanyal S, Xiao T , Liu J, Wang W, Ren X. Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification. Published online February 5, 2024. doi:10.48550/arXiv.2402.03686

    2024 doi arXiv

  23. [31]

    Self-Evaluation Improves Selective Generation in Large Language Models

    Ren J, Zhao Y , Vu T , Liu PJ, Lakshminarayanan B. Self-Evaluation Improves Selective Generation in Large Language Models. In: ; 2023. Accessed August 18, 2024. https://openreview.net/forum?id=OptKBWmreP

    2023

  24. [32]

    Language Models (Mostly) Know What They Know

    Kadavath S, Conerly T , Askell A, et al. Language Models (Mostly) Know What They Know. Published online November 21, 2022. Accessed August 18, 2024. http://arxiv.org/abs/2207.05221

    2022 arXiv

  25. [33]

    Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

    Tian K, Mitchell E, Zhou A, et al. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In: Bouamor H, Pino J, Bali K, eds. Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  26. [34]

    Algorithmic Learning in a Random World

    Vovk V, Gammerman A, Shafer G. Algorithmic Learning in a Random World. Springer International Publishing; 2022. doi:10.1007/978-3-031-06649-8

    2022 doi

  27. [35]

    Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks

    Lewis P , Perez E, Piktus A, et al. Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks. In: Advances in Neural Information Processing Systems. Vol 33. Curran Associates, Inc.; 2020:9459-9474. Accessed September 23, 2024. https://proceedings.neurips.cc/paper/2020...

    2020

  28. [36]

    DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines

    Khattab O, Singhvi A, Maheshwari P , et al. DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines. In: ; 2023. Accessed August 18, 2024. https://openreview.net/forum?id=sY5N0zY5Od 17 Supplementary: Table of Contents Challenges of medical coding.........

    2023

  29. [37]

    Let 𝑝(𝑦true|𝑥) represent the ALIGN code confidence score (i.e

    Nonconformity Scoring : For each instance in the calibration set, we calculate the nonconformity score based on the model's confidence in the correct code. Let 𝑝(𝑦true|𝑥) represent the ALIGN code confidence score (i.e. from the softmax output) for the true code label 𝑦true. Th...

  30. [38]

    This quantile q̂ represents the threshold at which we can ensure the desired coverage (e.g., 95%)

    Quantile Calculation : We then determine a critical non -conformity threshold by computing the ϵ-quantile of these scores from the calibration set. This quantile q̂ represents the threshold at which we can ensure the desired coverage (e.g., 95%). i.e. the quantile such that at...

  31. [39]

    The prediction set 𝐶(𝑥test) is constructed by including all predicted candidate codes 𝑦i where the confidence score exceeds the threshold 1− q̂ i.e

    Prediction Set Generation : At test or deployment time, the model computes the confidence scores for a test query (𝑥test) . The prediction set 𝐶(𝑥test) is constructed by including all predicted candidate codes 𝑦i where the confidence score exceeds the threshold 1− q̂ i.e. 𝐶(𝑥t...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.