{"id":"b0acf005-aad0-46c9-9580-d8ee15d30ef8","arxiv_id":"2411.13163","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ALIGN, a compositional zero-shot LLM system with candidate generation, self-evaluation, and uncertainty-based human deferral, improves ATC and MedDRA coding accuracy on 22 historical immunology trials compared with prompting and RAG baselines.","lead":"This paper introduces ALIGN, a three-stage AI system that automatically assigns standard medical codes (ATC and MedDRA) to medications and medical history terms in historical clinical trial data, without needing labeled training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Majority-vote gold standard is the load-bearing risk: the paper's own Table S1 documents intra- and inter-study code variability, yet every accuracy and deferral claim is scored against a single majority label per term.","rationale":"The reader's weakest assumption is the same one I would flag: the evaluation labels are a majority vote over historical codes that the paper itself demonstrates are variable. I agree with that assessment; my read does not change the CONDITIONAL verdict. I considered whether a different issue was more load-bearing, such as the per-unique-term evaluation or the restriction to records with existing codes. Those are real limitations: per-unique-term accuracy can differ substantially from per-record accuracy, and the missing-code population is the intended deployment target but is excluded from the benchmark. However, all of those issues act through the same evaluation pipeline, and even they would be partly addressable if the labels were trustworthy. The gold-standard concern is upstream: every accuracy value, every baseline comparison, and every deferral simulation is scored against a label set whose reliability the paper itself calls into question. A corrected gold standard would simultaneously resolve the label-noise problem and give a cleaner test of whether ALIGN's relative advantage is genuine. The paper has other strengths: the compositional design is clearly specified, the comparison to RAG/LLM baselines is reasonable within the benchmark, experiments are repeated five times, and the cost analysis is transparent. These do not, however, remove the need for external validation of the labels, which is why the verdict should remain CONDITIONAL rather than ACCEPT.","tokens_in":16550,"tokens_out":5785,"duration_ms":60699,"concrete_test":"Draw a stratified random sample of ~300 unique ATC term-route pairs (oversampling uncommon terms) and ~200 MedDRA terms from the RA and SLE evaluation sets. Have two independent certified medical coders, blinded to both the historical codes and ALIGN's outputs, assign codes and flag cases where multiple codes are genuinely acceptable; adjudicate disagreements to form a reference set. Then recompute Table 2 accuracy for ALIGN, RAG, and LLM against this reference, and recompute the entropy-based deferral curves. If ALIGN retains a comparable margin over baselines and deferral still improves accuracy, the central claim survives; if the margin shrinks or reverses, the reported superiority is an artifact of majority-vote label noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline number in ALIGN is computed against a gold standard built by taking, for each unique medication/route or MedDRA term, the most frequently assigned pre-existing code from variable historical annotations. The paper itself supplies evidence that this label is not a stable truth: Table S1 lists intra-study variability ('Anovlar coded as G03AA, G03AB, or G03FA in the same study') and inter-study variability, plus one-to-many ATC mappings that are legitimately context-dependent (acetylcysteine: R05CB01/S01XA08/V03AB23). For ATC specifically, multiple codes can be correct for the same drug depending on indication, dose, and route; collapsing them to a single majority code will mark valid predictions as errors and may also encode systematic biases from the original studies. The deferral simulation makes this worse: it assumes a deferred case is corrected to the majority-vote gold standard, so it measures agreement with the noisy label, not with clinical truth. In addition, the evaluation set is restricted to records that already have codes, whereas ALIGN's motivating use case is missing codes; whether the reported 72-73% and 86-89% accuracies transfer to the missing-code population is untested. Since the 7-22% margin over baselines is a comparison on this same noisy benchmark, label noise is the single load-bearing assumption: if the majority labels are wrong or arbitrary, the absolute accuracies and the claimed superiority over baselines may not reflect true coding correctness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ALIGN, a compositional large language model (LLM) system for zero-shot medical coding of concomitant medications into ATC codes and medical history terms into MedDRA codes. ALIGN operates in three stages: diverse candidate generation (dense retrieval, BM25, and LLM reasoning), LLM self-evaluation via natural-language-inference filtering against external code descriptions, and confidence scoring with entropy-based uncertainty estimation to support human deferral. The system is evaluated on 22 immunology trials (10 RA, 12 SLE) and compared against vanilla LLM prompting, chain-of-thought prompting, and retrieval-augmented generation (RAG). The authors report that ALIGN matches RAG on MedDRA coding and outperforms all baselines on ATC coding, particularly at the more specific ATC levels, and that uncertainty-based deferral improves accuracy to about 90% at 30% deferral. The paper also provides cost estimates of $0.0007 and $0.02 per code for GPT-4o-mini and GPT-4o backbones, respectively.","tokens_in":16763,"tokens_out":7519,"duration_ms":72160,"significance":"If the reported results are reliable, ALIGN is a practically useful zero-shot system for harmonizing historical clinical trial data, with a meaningful contribution in combining retrieval and LLM reasoning with explicit verification and uncertainty-driven human-in-the-loop deferral. Strengths of the paper include the compositional system design, the use of external grounding for self-evaluation, repeated experiments (five repeats) with mean and standard deviation, and a concrete cost analysis. The main risk is the evaluation's dependence on a noisy majority-vote gold standard, which the paper itself documents as internally and externally inconsistent. Because the central claim is comparative (ALIGN outperforms RAG on ATC), the label noise issue is load-bearing and must be addressed before the quantitative conclusions can be accepted.","major_comments":[{"comment":"The gold standard is the majority of pre-existing codes, but Table S1 documents the same term receiving different codes within and across studies (e.g., Anovlar coded as G03AA, G03AB, or G03FA) and legitimate one-to-many mappings (e.g., acetylcysteine assigned R05CB01, S01XA08, or V03AB23 depending on context). No inter-annotator agreement is reported, so for many terms the majority label may be arbitrary. Since every accuracy figure in Tables 1-2 and the deferral simulation in Figure 3 is scored against this label, label noise is load-bearing. Please report the distribution of label agreement and either perform sensitivity analyses treating any of the observed codes as correct, or validate on a manually adjudicated subset.","section":"Dataset construction; Table S1"},{"comment":"The evaluation set consists only of records with pre-existing codes, while the stated motivation is missing codes (Figure 1). Missingness may be non-random, so the reported accuracies may not transfer to the missing-code population. In addition, the deferral simulation (Figure 3) assumes deferred cases are corrected to the majority-vote gold standard, meaning it measures agreement with a potentially noisy label rather than clinical truth. Please address this population mismatch or explicitly state it as a limitation.","section":"Dataset construction; Figure 1"},{"comment":"The abstract claims ALIGN 'outperforming baselines by 7-22%' for ATC coding, but the tables do not clearly identify the comparison supporting this range. For ATC level 4 overall, ALIGN is about 7 percentage points above RAG (e.g., 72.20 vs 65.27 in Table 2a for RA), and the 22% figure does not match any reported margin. Please specify the exact baseline and metric used for the 7-22% claim and report effect sizes with confidence intervals.","section":"Abstract; Results, Table 2"},{"comment":"The set-based coverage results (ALIGN (Set)) are reported without the average prediction set size. Without this information, high coverage is not interpretable, since a set containing many candidate codes would trivially achieve high coverage. Please report the average number of codes in the prediction set (or a size distribution) alongside the set-based accuracy.","section":"Results, Table 2 (ALIGN (Set))"},{"comment":"The common/uncommon split is defined on occurrences (Pareto 80/20), but the evaluation is performed on unique query-code pairs; the number of unique pairs in each subset is not reported. The 'common medications' subset could be very small, and the reported 86-89% accuracies may have wide confidence intervals that the five-repeat standard deviation does not capture. Please report the number of unique pairs in each subset.","section":"Results, Table 2(b)-(c)"}],"minor_comments":[{"comment":"In the second paragraph, 'standardized medical coding systems play are essential' is ungrammatical; should read 'play an essential role' or 'are essential'.","section":"Introduction"},{"comment":"The caption refers to 'RA vs SLR'; the intended abbreviation appears to be 'SLE'.","section":"Figure 1"},{"comment":"The phrase '±90% accuracy' appears in the abstract and Results; the plus-minus symbol is likely a typo for 'to 90%'.","section":"Abstract; Results"},{"comment":"Table S2 column headers list the MedDRA levels as 'SOC HLT HLGT', while Table 1 uses 'SOC HLGT HLT'; please make the ordering consistent.","section":"Supplementary, Table S2"},{"comment":"The cost figures ($0.0007 and $0.02 per code) are quoted without details of token usage or the cost model; please clarify how these values were computed.","section":"Results, Cost efficiency"},{"comment":"The conformal prediction extension is described but not evaluated. If it is presented as a feature of ALIGN, please include at least a small worked example or state clearly that it is a proposal without empirical validation.","section":"Supplementary, Conformal Prediction"},{"comment":"The statement that temperature 0.7 showed 'no sensitivity' is not supported by a reported sensitivity analysis; please include such an analysis or soften the claim.","section":"Experimental Setup"}],"recommendation":"major_revision","confidential_remarks":"The manuscript uses proprietary TransCelerate data and does not mention code or data availability; please consider whether the evaluation is reproducible. The label-noise issue is the main risk to the quantitative claims, and I would encourage the editor to ask for the sensitivity analysis and population-mismatch analysis described in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you're tracking LLM-based medical coding or any system that promises to harmonize historical trial data. ALIGN is a zero-shot pipeline that combines dense retrieval, BM25, LLM chain-of-thought candidate generation, NLI-based self-evaluation, and MCQ-style confidence scoring with entropy-based deferral, evaluated on ATC and MedDRA coding across 22 immunology trials. The composition isn't novel component-wise, but the whole thing is well-engineered, and the ablation is genuinely useful: the LLM proposes ~80% of the correct codes, retrieval adds a few percent, and the self-evaluation prunes 30-50% of candidates. Cost numbers are sensible ($0.0007 per code on GPT-4o-mini).\n\nThe main weakness is the ground truth. Every accuracy figure is computed against a majority vote of the pre-existing codes that the paper itself shows to be inconsistent — Table S1 gives Anovlar coded as G03AA, G03AB, and G03FA within a single study. So ALIGN's scores measure agreement with a possibly arbitrary majority label, not clinical truth. This affects the absolute accuracies and the claimed 7-22% margin over baselines, since all methods are scored on the same noisy benchmark. The deferral simulation also assumes a deferred case is corrected to the majority label, so the 17% improvement partly reflects agreement with that label. On top of that, the evaluation only uses terms that already have codes — the motivating use case is missing codes, and transfer is untested. Minor but real: the abstract says 87-90% for HLGT when that's actually HLT (HLGT is ~95%), and the 'overall accuracy' is per unique term, not per record, which makes common medications count once.\n\nThat said, the central claim is plausible: ALIGN beats prompting and RAG on ATC and matches RAG on MedDRA. Given the ground-truth risk, I'd treat the absolute numbers as agreement-with-historical-codes rather than true clinical accuracy. This deserves a serious referee. The right fix is a small manually adjudicated gold standard, or at least a sensitivity analysis that counts reasonable alternative codes as correct. If that supports the majority label, the paper is a solid contribution. If not, the claims need scaling back.\n\nI'd send it to peer review and make the ground-truth validation the central request. No code or data is released, which hurts reproducibility but doesn't undercut the system's design.","headline":"A well-engineered zero-shot medical coding system whose headline numbers all depend on a noisy majority-vote gold standard that the paper itself documents as inconsistent.","tokens_in":17420,"tokens_out":3593,"would_cite":true,"duration_ms":34571,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ALIGN, a three-stage LLM coding system, assigns ATC medication codes with 72-73% accuracy and defers uncertain cases to reach about 90%.","keywords":["medical coding","ATC classification","MedDRA","large language models","zero-shot learning","uncertainty quantification","human-in-the-loop","clinical trial data interoperability"],"falsifier":"Take a random sample of coded medication and medical-history terms from these 22 trials, have independent expert coders re-adjudicate the majority-vote labels without seeing them, and compare ALIGN's predictions against that fresh gold standard; if accuracy on ATC Level 4 falls below the reported 72-73% or the 7-22% margin over baselines disappears, the central claim is undermined.","tokens_in":16263,"feed_emoji":"💊","tokens_out":5315,"duration_ms":49054,"temperature":0.7,"pith_summary":"The paper introduces ALIGN, a zero-shot system that automates medical coding of historical clinical trial data by composing several large language model steps. It claims ALIGN outperforms direct prompting and retrieval-augmented baselines on ATC medication coding, especially at the most specific chemical level, while matching RAG on MedDRA medical-history coding. It also claims ALIGN's uncertainty estimates let a human reviewer take over the hardest cases, raising accuracy to about 90% with 30% deferral. If true, ALIGN would shorten the months-long manual harmonization needed to combine trial datasets and make cross-trial reuse practical.","feed_headline":"Three-step LLM system codes trial drugs to 73% accuracy","feed_subtitle":"Uncertainty-based deferral lifts ATC coding to about 90%, with human review on the hardest 30% of cases.","key_machinery":"The load-bearing mechanism is a three-stage composition: (1) candidate generation by dense retrieval, BM25 sparse retrieval, and LLM reasoning with query contextualization and synonym generation; (2) self-evaluation that filters candidates through natural-language entailment against official ATC/MedDRA descriptions; (3) confidence scoring that reformulates coding as a multiple-choice question, uses logit biasing to constrain answers, extracts token log-probabilities, applies temperature-scaled softmax, and estimates uncertainty as predictive entropy. The entropy threshold drives deferral to human reviewers, and the authors note the same confidence scores can feed conformal prediction for coverage guarantees.","core_discovery":"The paper's central claim is that a compositional, multi-step LLM pipeline can code clinical trial terms accurately in a zero-shot setting without labeled examples. For ATC medication codes, ALIGN achieves 72-73% overall accuracy at Level 4 and 86-89% on the most common medications, beating LLM and RAG baselines by 7-22 percentage points; for MedDRA it matches RAG at all hierarchy levels and reaches 87-90% at the finest reported level. The system's predictive entropy flags uncertain predictions, and simulated deferral of about 30% of cases to human coders brings ATC accuracy to roughly 90%, with the largest gains on uncommon medications.","pith_inferences":["If the accuracy holds on independently re-adjudicated labels, ALIGN-style compositional coding could generalize to ICD-10 and adverse-event coding, where one-to-many mappings are even more common.","The self-evaluation step's heavy pruning (31-53% of candidates) suggests retrieval-only pipelines waste most of their candidates; composition may be needed whenever coding decisions hinge on route, dose, or indication rather than name matching.","The entropy-based deferral curve implies a deployment recipe: set the deferral threshold by available expert hours, not by a fixed accuracy target, since the gain per deferred case is concentrated in the long tail.","The conformal prediction extension, if applied with a small labeled calibration set, could turn ALIGN's confidence scores into set-valued predictions with a formal coverage guarantee, a step the paper describes but does not evaluate."],"forward_implications":["ATC coding at the most specific chemical level improves from roughly 50-66% for baselines to 72-73% overall, making automated coding usable for common medications.","Common medications reach 86-89% accuracy at ATC Level 4, so the majority of real prescriptions can be coded automatically with high confidence.","With 30% human deferral, accuracy approaches 90%, and deferral helps most on uncommon medications; this gives a practical human-in-the-loop workflow.","Cost of $0.0007 per code with a cost-efficient backbone makes the pipeline cheap enough for large-scale application to hundreds of thousands of records.","The zero-shot design means the same system can be pointed at new coding dictionaries or therapeutic areas without retraining."],"supporting_citations":[{"why":"Supplies the historical clinical trial data source used to build and evaluate the coding datasets.","marker":"[3]"},{"why":"Defines the ATC classification system that the medication-coding task targets.","marker":"[10]"},{"why":"Defines the MedDRA dictionary and hierarchy used for medical-history coding.","marker":"[11]"},{"why":"Provides the compound AI system framing that motivates ALIGN's multi-component architecture.","marker":"[24]"},{"why":"Supplies the chain-of-thought prompting technique used in candidate generation and reasoning.","marker":"[26]"},{"why":"Provides the natural language inference task formulation used for self-evaluation of candidate codes.","marker":"[29]"},{"why":"Supports the claim that LLMs give well-calibrated token-level uncertainty, which ALIGN's confidence scoring builds on.","marker":"[32]"},{"why":"Provides the conformal prediction framework the paper extends ALIGN with for coverage guarantees.","marker":"[34]"},{"why":"Defines retrieval-augmented generation, the main baseline ALIGN is compared against.","marker":"[35]"}],"fun_headline_variants":["LLM pipeline codes clinical trials at 90% with human help","Zero-shot LLM coding hits 90% accuracy after 30% human deferral","ALIGN: three-step LLM boost for medical coding accuracy","Uncertainty-guided deferral lifts LLM medical coding to 90%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the majority vote of the pre-existing codes in the trial data is the correct gold standard, even though the paper documents that codes for the same term vary within and between studies.","fun_headline_variants_meta":{"raw":{"variants":["LLM pipeline codes clinical trials at 90% with human help","Zero-shot LLM coding hits 90% accuracy after 30% human deferral","ALIGN: three-step LLM boost for medical coding accuracy","Uncertainty-guided deferral lifts LLM medical coding to 90%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1647,"prompt_tokens":1046,"completion_tokens":601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":662,"tokens_out":601,"duration_ms":5887,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:45:33.360777+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of coded medication and medical-history terms from these 22 trials, have independent expert coders re-adjudicate the majority-vote labels without seeing them, and compare ALIGN's predictions against that fresh gold standard; if accuracy on ATC Level 4 falls below the reported 72-73% or the 7-22% margin over baselines disappears, the central claim is undermined.","supporting_citations":[{"cited_title":"Sharing Historical Trial Data to Accelerate Clinical Development","cited_arxiv_id":null,"evidence_quote":"Supplies the historical clinical trial data source used to build and evaluate the coding datasets."},{"cited_title":"Accessed September 23, 2024","cited_arxiv_id":null,"evidence_quote":"Defines the ATC classification system that the medication-coding task targets."},{"cited_title":"The Shift from Models to Compound AI Systems","cited_arxiv_id":null,"evidence_quote":"Provides the compound AI system framing that motivates ALIGN's multi-component architecture."},{"cited_title":"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting technique used in candidate generation and reasoning."},{"cited_title":"Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the main baseline ALIGN is compared against."}],"review_version":1}