REVIEW 4 major objections 6 minor
Teaching agentic AI to learn expert reasoning for rare disease diagnosis
T0 review · 4 major / 6 minor · reviewed 2026-07-12 · grok-4.5
Pith's one-line read A portable clinical policy plus public biomedical tools lets one unmodified language model diagnose rare diseases at high accuracy without fine-tuning or large case banks.
desk verdict Solid systems paper: external clinician-vetted policy + public tools lifts phenotype-first rare-disease ranking without fine-tunes or case banks; SOTA numbers rest partly on a permissive LLM judge, but the direction of the effect is real. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Policy Iteration with Human Feedback (PIHF): a weight-free loop in which a frozen model runs a natural-language diagnostic policy, clinicians review scores and full reasoning traces, and the retained critiques revise the policy itself—producing a portable, inspectable eight-phase workflow that dictates tool use, evidence weighing, and reflective adjudication.
What would settle it
A prospective bedside trial that supplies the system with incomplete or evolving phenotypes plus laboratory and genetic data, then measures whether physician-adjudicated top-1 and top-5 accuracy still exceeds the same unmodified model without the policy and tools.
Extended reading notes
Core claim
LiteOdyssey shows that a single unmodified reasoning language model, guided by an eight-phase clinical-genetics policy developed through Policy Iteration with Human Feedback and eight public or cached biomedical tools, reaches state-of-the-art phenotype-first disease Recall@1 of 58.6% on LIRICAL and 59.6% on the PhenoPacket Store, with large structured-environment lifts over the identical model without tools, transfer to an open-weights model never used in development, and statistically significant gains on 515 Undiagnosed Diseases Network patients.
Load-bearing premise
That clinical features encoded as HPO terms plus public curated knowledge tools, scored mainly by a language-model judge, are a fair enough stand-in for real diagnostic performance even when laboratory, biochemical, and genetic-variant evidence are mostly absent and absolute accuracy falls sharply on hard real-world cases.
Editorial extensions
If this is right
- Diagnostic support for rare disease can be shipped as a readable policy document plus public tools rather than as fine-tuned weights or multi-gigabyte retrieval corpora.
- The same policy can be executed unchanged across closed and open model families, so improvements travel without re-training.
- Every differential comes with a step-by-step reasoning trace that a clinician can audit, contest, or revise.
- Groups without large compute or curated case banks can still field competitive phenotype-first rare-disease AI.
- PIHF offers a general pattern for turning expert diagnostic practice into portable, weight-free policy artifacts.
Reading between the lines
- If the policy is the true carrier of performance, later work can distill a stabilized PIHF document into a small adapter while keeping the original policy as the human-readable source of truth.
- The same external-policy pattern could be tried on other multi-step clinical tasks that currently rely on multi-agent orchestration or heavy retrieval, such as oncology staging or complex infectious-disease workups.
- Because the policy is natural language, institutions could maintain local forks that encode site-specific testing pathways without retraining foundation models.
- Absolute accuracy remaining low on real undiagnosed cohorts suggests the next binding constraint is richer multimodal inputs rather than further policy refinement alone.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. LiteOdyssey is a single-agent, phenotype-first rare-disease diagnostic system that guides an unmodified reasoning LLM with a natural-language eight-phase clinical genetics policy (developed via Policy Iteration with Human Feedback, PIHF) and eight public/cached biomedical tools. On LIRICAL (n=370) and the PhenoPacket Store (n=873), the system reports disease Recall@1 of 58.6% and 59.6%, large lifts over a same-backbone tool-free baseline (e.g., +23.5 and +36.7 points), transfer of the environment gain to Qwen3.6-35B without re-tuning, and statistically significant but smaller gains on 515 UDN patients (R@1 20.4% vs 16.7%, McNemar p=0.027). The authors argue that externalizing diagnostic reasoning as a portable, auditable policy can match or exceed heavier multi-agent/retrieval/fine-tuning systems while remaining deployable and inspectable.
Significance. If the core result holds under stricter, comparator-aligned scoring, the paper is significant for medical AI: it shows that much of rare-disease diagnostic yield can come from organizing inference-time reasoning and public tool use rather than from fine-tuning, multi-agent orchestration, or large solved-case banks. Strengths that should be credited include same-backbone tool ablations, development-excluded public cases, model-level holdout on an open-weights backbone, a private multi-site UDN cohort with paired McNemar tests, gene-level secondary endpoints, exact-OMIM sensitivity preserving direction, and worked reasoning-trace cases that illustrate rescues and regressions. PIHF as a weight-free, clinician-auditable policy iteration process is a useful methodological contribution for settings where model weights cannot be modified.
major comments (4)
- [Methods §5.6; Results §3.1; Appendix Figure A1 / Table A1] Methods §5.6 and Results §3.1: Primary disease Recall@1 is defined by an LLM-as-a-judge that can credit synonym/alias agreement, while deterministic exact-OMIM is only a sensitivity analysis. Appendix Figure A1 / Table A1 show material absolute drops under exact-OMIM (e.g., held-out LIRICAL full-system R@1 54.4% vs 57.5% judged). The headline SOTA claim (58.6% LIRICAL; 59.6% PhenoPacket; Figure 1 vs DeepRare 56.0%) is therefore not fully secured at identifier level. Please report exact-OMIM R@1/R@5 as co-primary (or primary) for all main tables/figures, quantify the fraction of judged-only top-1 credits, and restate absolute SOTA claims only where they survive the stricter metric.
- [Results §3.1; Figure 1–2] §3.1 comparator framing: DeepRare is the main published reasoning comparator (39.5% without retrieval; 51.6–56.0% with a 67,795-case retrieval corpus that can include benchmark-like cases). The manuscript scores LiteOdyssey with “DeepRare-style” matching but does not show that DeepRare’s published numbers were obtained under the same judge/alias protocol, nor re-score DeepRare outputs on the identical case set and metric. Without matched scoring (and an explicit statement of which DeepRare configuration/backbone is being compared), the absolute “higher still” SOTA claim is not load-bearing. Either re-evaluate under a shared protocol or demote absolute ranking language and emphasize the controlled same-model environment lift.
- [Methods §5.3–5.4; §5.8] Methods §5.3–5.4 / §5.8: PIHF and the eight-phase policy are central inventions, but only 50 LIRICAL and 50 UDN cases informed development, the actual policy artifact is not provided (code/demo “URL TBD”), and the iteration loop (how many rounds, which critiques were accepted/rejected, how phase-transition rules changed) is described at a high level. Because portability and auditability are core claims, the final natural-language policy, tool schemas, judge prompt, and a minimal PIHF change log should be released or included as supplementary material so that the weight-free policy—not only the model—can be inspected and re-run.
- [Abstract; Results §3.5–3.6; Discussion] §3.5 and Discussion: On the UDN cohort the absolute full-system disease R@1 is 20.4% (+3.7 points over baseline). The gain is statistically significant and directionally consistent on held-out cases, but the abstract and introduction lean heavily on this “external evaluation” as clinical support. Please keep absolute yield, selection for diagnostic difficulty, phenotype-only inputs, and the non-significant gene R@1 (p=0.21) co-equal with the p-values so readers do not over-read clinical readiness from a modest absolute top-1 rate.
minor comments (6)
- [Abstract / title] Naming is inconsistent across abstract/title casing (liteOdyssey vs LiteOdyssey).
- [Figure 1] Figure 1 deployment-burden axis (0.66 GB vs 100+ GB) needs a short methods note on what is counted (cached indices, tool DBs, model weights excluded/included) so the comparison is reproducible.
- [Results §3.2; Figure 2] §3.2 states PhenoPacket parametric baseline 22.9% while Figure 2 panels use subset-specific baselines (e.g., 10.7% unmapped); clarify full-corpus vs subset baselines in one place.
- [Appendix A; §3.7] Appendix A worked cases are valuable; consider one additional regression/rescue pair from PhenoPacket or UDN (de-identified) to show the trace behavior outside LIRICAL.
- [References; §5.8] References include several 2025–2026 arXiv preprints and a Nature DeepRare citation with 2026 date; verify bibliographic completeness and version pins for reproducibility manifests.
- [References] Minor prose issues: missing spaces in concatenated author/affiliation strings in references; “pointprevalenceofrare diseases” etc. in ref [1] rendering.
Circularity Check
No circularity: claims are empirical performance of a policy+tools system on external ground-truth diagnoses, with held-out ablations.
full rationale
LiteOdyssey's load-bearing claims are measured diagnostic accuracy (disease/gene Recall@k) of a single unmodified LLM under a natural-language PIHF policy plus public tools, versus the same model without tools, on LIRICAL, PhenoPacket Store, and UDN cases whose ground-truth diagnoses are external (benchmark labels or UDN-established). PIHF revises the policy using scores/traces from only 50 LIRICAL + 50 UDN development cases; the paper reports essentially identical environment lifts on the complementary held-out sets (oos320 LIRICAL, full PhenoPacket, oos465 UDN), on a never-used open-weights backbone, and under exact-OMIM sensitivity. There is no equation or definition that makes the reported Recall equal the development inputs by construction, no fitted numerical parameter re-labeled as a prediction, and no uniqueness theorem or ansatz whose sole support is an overlapping-author citation. Standard train/dev/eval separation plus external ground truth is not circular under the stated criteria.
Assumptions & free parameters
free parameters (4)
- PIHF development set size =
50 LIRICAL + 50 UDN
- Eight-phase workflow structure and phase-transition rules =
8 phases (0–7)
- Tool library composition and weighting (e.g., HPO information content) =
8 tools over Monarch/OMIM/ClinGen/gnomAD/PubMed/offline variants
- LLM-as-judge scoring prompt / acceptance criteria =
judged R@1 primary; exact-OMIM secondary
assumptions (5)
- domain assumption Public curated biomedical sources (Monarch, OMIM, ClinGen, gnomAD, PubMed) plus offline variant filters supply enough external knowledge for phenotype-first monogenic rare-disease ranking without private solved-case retrieval.
- domain assumption HPO term lists (or free-text mapped to HPO) are a sufficient clinical input representation for the evaluated diagnostic task.
- domain assumption A frozen general-purpose reasoning LLM can faithfully execute a long natural-language clinical policy and integrate tool outputs into a ranked differential.
- ad hoc to paper LLM-as-judge disease matching, supplemented by exact-OMIM cross-check, is an adequate primary accuracy metric for system comparison.
- standard math Standard statistical comparisons (paired exact McNemar tests; Bonferroni on secondary gene endpoints) appropriately support claims of environment lift.
invented entities (3)
-
Policy Iteration with Human Feedback (PIHF)
-
LiteOdyssey diagnostic policy (eight-phase workflow artifact)
-
LiteOdyssey single-agent tool-mediated diagnostic system
independent evidence
Cite this review
Pith. "Pith review of Teaching agentic AI to learn expert reasoning for rare disease diagnosis." pith.science (2026). https://pith.science/paper/7AAHRUSN
@misc{pith2026260616149,
author = {Pith},
title = {Pith review of: Teaching agentic AI to learn expert reasoning for rare disease diagnosis},
year = {2026},
howpublished = {\url{https://pith.science/paper/7AAHRUSN}},
note = {Machine review of arXiv:2606.16149}
}
read the original abstract
Rare disease diagnosis depends on expert reasoning that is scarce and difficult to transfer; off-the-shelf large language models (LLMs) rank the correct disease first in only 35.4% of benchmark cases. Here we show that this expert reasoning can be converted into a scalable AI capability through a governed learning process rather than model training alone. We developed liteOdyssey through Policy Iteration with Human Feedback (PIHF), an in-context policy-learning method adapted from generalized policy iteration in reinforcement learning, in which model failures and expert corrections consolidate into an explicit, clinician-gated policy that turns an off-the-shelf LLM into an agentic diagnostic system. We demonstrated that such a policy improved diagnostic accuracy to match the best published systems at a fraction of their deployment footprint, generalized to unseen diseases, transferred across models, and remained under clinician control. Across 1,243 public benchmark cases spanning 722 rare diseases, liteOdyssey ranked the correct disease first in 59.3% of cases versus 26.5% without the policy, with nearly identical gains on the 1,193 cases and 679 diseases excluded from policy development. Ablations showed that gains exceeded automated prompting improvement or source access alone, and the policy transferred without modification across closed- and open-weight models. In 515 Undiagnosed Diseases Network patients, liteOdyssey again improved accuracy, and blinded physicians rated its differentials more often exact and less often unhelpful. Through PIHF, expert reasoning becomes an LLM capability that experts can inspect, revise, and transfer across models.
Figures
Figures from the paper (2 more)
Reviewed July 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.