REVIEW 4 major objections 4 minor 41 references
Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A 70-row rule table compiled from published guidance can label 3,000 triage dialogues, and fine-tuning on those labels lifts agreement from 61.7% to 74.1% and emergent-case recall from 9.5% to 69.0%.
desk verdict A genuinely useful weak-supervision taxonomy held back by an evaluation that is too benchmark-coupled to support the headline gain. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the operational rule table, an ordered set of 70 rows pairing a textual trigger and symptom family with a disposition tier. A generated profile cites a nonempty set of rows and receives the maximum tier over those rows; this citation-based, maximum-tier labeling is what turns the guideline into oracle supervision. Eight construction strategies, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair, convert the table into 3,000 dialogues carrying no human labels, and the paper characterizes each strategy's evidential status. The permutation control, which relabels dialogues while preserving their texts, is what isolates the assignment as the carrier of the signal.
What would settle it
An external, frozen, multi-clinician-labeled interactive benchmark that was never used in any development step would settle the central claim: if GAO-Triage does not beat the untuned backbone on agreement and emergent recall there, the measured gains are policy-conformance artifacts rather than real triage capability.
Extended reading notes
Core claim
The central discovery is that a compiled 70-row operational rule table, used as the sole source of instance-level labels for 3,000 synthetic multi-turn dialogues, can stand in for expert annotation: fine-tuning a 9B backbone on these labels raises exact agreement with a 201-case operational reference from 61.7% to 74.1% and emergent recall from 9.5% to 69.0%. The same recipe reproduces with a second seed and a second patient simulator, and none of the seven general-purpose systems tested dominates the resulting agent on both agreement and emergent recall. Permuting which dialogue receives which label while keeping the dialogue texts fixed collapses the model to a constant-routine predictor (112/201, emergent recall 0/42), which the paper takes as evidence that the learned capability resides in the guideline-derived assignment of labels to dialogues rather than in the dialogue surface form itself.
Load-bearing premise
The 201 patient-authored narratives and the authors' adjudicated reference labels are a faithful stand-in for real ophthalmic triage decisions, even though the same cases influenced the rule table, corpus development, and model selection.
Editorial extensions
If this is right
- Training a multi-turn triage agent no longer requires expert dialogue transcripts; a compiled specialty guideline can supply all instance-level labels.
- A 9B open-weight model fine-tuned with this recipe can match or approach frontier API models on the task, so deployment does not require a frontier model at inference time.
- Because the agreement gain is carried by emergent-case recall, rising from 9.5% to 69.0%, the method specifically improves detection of sight-threatening presentations.
- Randomly permuting label–dialogue assignments collapses performance to the majority class, showing that dialogue surface form alone does not teach triage.
- Repairing contradictory labels was the only development change that lowered catastrophic emergent-to-routine misses while raising agreement, pointing to label consistency as a safety lever.
Reading between the lines
- The recipe should transfer to other triage or intake domains that have published decision protocols, since the core mechanism is a compiled rule table supplying label assignment rather than ophthalmology-specific content; a direct test would compile a general-medicine or mental-health triage guideline into a similar table and compare against a corpus without such assignment.
- The permutation result implies that synthetic dialogue corpora without a principled evidence-to-disposition assignment may teach dialogue style but not triage skill, so corpus designers should treat label assignment as the primary design variable rather than dialogue count.
- The GRPO outcome-reward collapse suggests that directly rewarding the final disposition can teach agents to stop talking instead of eliciting decisive facts; an informative next experiment would separate the outcome reward from the rule, format, and judged-process components to identify which one causes the silent early terminations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Guideline-as-Oracle (GAO), a method that compiles AAO ophthalmic guidance into a 70-row operational rule table and uses that table as the sole source of instance-level supervision to generate 3,000 multi-turn training dialogues with no human dialogue labels. Fine-tuning a 9B language model on this corpus yields GAO-Triage, which is evaluated against a 201-case operational reference derived from public Reddit narratives. The primary base-versus-SFT comparison shows agreement rising from 61.7% to 74.1% (exact McNemar p=0.0046) and emergent-case recall from 9.5% to 69.0%; the gains persist under a second seed and a second patient simulator. A label-permutation control collapses the model to a majority-class routine predictor. The paper also catalogs eight rule-to-dialogue construction strategies, explicitly marking which are the labeling mechanism, which return null results, which are confounded, and which are evaluated only as a package. The authors repeatedly state that all scores measure policy conformance to an author-defined operationalization, not clinical validity.
Significance. If the central claims were fully established, GAO would be a valuable practical alternative to expert dialogue annotation for multi-turn medical agents, and the paper's design-space taxonomy of construction strategies would be a useful reference for others building rule-derived training corpora. The manuscript's strengths include a same-day clean primary comparison with a pre-specified analysis plan, a second-seed and second-simulator replication, a label-permutation control, and unusually candid reporting of null results, confounds, corrupted historical runs, bounding analyses, and data-governance gaps. However, the unresolved possibility of train/benchmark overlap and the author-built, policy-coupled evaluation reference mean that the primary numbers do not yet isolate a generalizable guideline-derived triage capability. The paper's honest audit makes the threat explicit, but the main evidence is not yet sufficient to support the claim that the learned signal comes from guideline-derived assignment rather than from memorized evaluation content.
major comments (4)
- [Sections 3.2, 4.1, 5.1] The decontamination evidence does not support the primary interpretation of the main result. Section 4.1 reports that the 201-case benchmark 'had informed rule-table revision, corpus development, and model selection,' and Section 3.2's own audit states that its lexical screens 'establish neither isolation nor decontamination' and 'demonstrably missed targeted paraphrases'; it also found paraphrase clusters of 12–19 dialogues and a condition where 59 source dialogues recur as 118 training exposures. The clean protocol in Section 5.1 disables only the complete-profile heuristic; it does not remove potentially overlapping training material. Under the contamination scenario, the 74.1% versus 61.7% agreement gain and the 9.5% to 69.0% emergent-recall gain could reflect case-specific retrieval of memorized benchmark labels rather than a guideline-derived triage capability. The paper should either train on a corpus with verified exclusion of the 201 evaluation cases and their paraphrase variants, or make the primary evaluation an independently labeled external benchmark that was not used in any development step, and report the central comparisons on that benchmark.
- [Section 4.1] The evaluation target is an author-adjudicated operational reference, not an independent clinical gold standard. The workflow flagged only 25 of 201 labels for review, retained the other 176 by default, and revised 12 after author adjudication; two rule-conditioned LLM audits agreed on only 44.1% of 930 citation-to-tier recomputations (κ=0.285), and nine of the twelve revisions were downgrades. Since the same benchmark informed the rule table Φ, which in turn informed both adjudication and training, the primary scores measure conformance to an author-built policy. The Introduction and Conclusion statements that the signal 'lies in guideline-derived assignment' and that 'the capability comes from guideline-derived assignment' are stronger than this evidence supports; they should be restricted to 'the model conforms to the author-compiled operational policy on its own benchmark' unless an independent clinician-labeled evaluation is added.
- [Section 6] The label-permutation control does not eliminate the memorization alternative. If training dialogues overlap with the benchmark and carry the original labels, then permuting label–dialogue assignments would destroy exactly the memorized associations and produce the observed collapse to the majority class; the control therefore cannot distinguish guideline-derived assignment from benchmark memorization. The control also uses a 2,635-sample corpus rather than the 3,998-exposure headline corpus, so it does not test the exact trained condition. To support the localization claim, repeat the permutation control on a corpus verified free of benchmark content, or otherwise show that the primary effect survives decontamination.
- [Limitations and ethics] The paper itself asserts a serious data-governance gap: the authors 'retain no platform agreement, acquisition record, or de-identification audit' for Reddit-derived data, note that 'public posting is not consent to machine-learning use,' state that post-derived text reached remote endpoints with unrecorded retention policies, and report that no IRB review was sought. This is a substantial unresolved issue that must be addressed or explicitly justified before publication, particularly because the same unconsented data source feeds both the training corpus and the evaluation benchmark.
minor comments (4)
- [Section 3.3] In the paragraph on evidence gating, 'Figure 5 Section 6' should be 'Figure 5 in Section 6'.
- [Section 5.1] The primary comparison reports an exact McNemar p-value for the paired agreement outcome, while Table 1 lists p-values comparing each arm against GAO-Triage; it would help to state explicitly that all Table 1 p-values are two-sided exact tests and are not multiplicity-adjusted.
- [Section 4.2] The sentence 'The two rule views already disagree on 52 cases' should state the denominator explicitly, since it appears in the context of a distinct dialogue-revised condition with n=201.
- [Section 6] The claim that label repair 'coincides with the disappearance of a late-training safety degradation' rests on different denominators (2/64 versus 1/71) and on package-level evidence; this should be described as an association rather than an effect, as the surrounding text already acknowledges.
Circularity Check
Primary evaluation is policy-self-referential: the 201-case reference and the training labels are both products of the same Φ, and the benchmark also informed rule-table revision and model selection.
-
self definitional
[Section 4.1 (Benchmark and Reference Labels)]
"The narratives and the final reference were fixed before any comparison was run, but the same cases had informed rule-table revision, corpus development, and model selection. ... its dependencies run one way—the benchmark informed Φ, which informed both adjudication and training. Internal agreement therefore tests the construction recipe for policy conformance, not for clinical validity."
The evaluation target is the authors' own operational reference: it was adjudicated under the same Φ that generated every training label, and the benchmark cases that compose it also drove rule-table revision and model selection. Agreement between GAO-Triage and this reference therefore measures, by the paper's own account, how well the model reproduces the authors' compiled policy on cases that helped build that policy. The primary 'prediction' (74.1% agreement, 69.0% emergent recall) is not an independent outcome but a self-consistency check between two artifacts—training labels and reference—that share Φ as their common constructor.
-
fitted input called prediction
[Section 3.2 (From Table to Corpus), audit paragraph]
"These are lexical screens rather than semantic classifiers, and they demonstrably missed targeted paraphrases, so they establish neither isolation nor decontamination."
This admission follows the paper's statement that the benchmark informed rule-table revision and that some training profiles were abstracted from Reddit narratives of the same type as the benchmark. Because the decontamination audit is inconclusive by the authors' own words, the clean-protocol result cannot separate a learned guideline-to-triage mapping from memorization of evaluation-derived content. The reported prediction on the 201 cases is therefore not demonstrated to be out-of-sample; the gain could be benchmark-specific retrieval, which is a form of fitting the reported numbers to the evaluation input.
full rationale
The guideline-to-dialogue construction itself is not circular: the model is genuinely trained on labels produced from the 70-row table, and the permutation control shows the label–dialogue assignment, not surface form, carries the behavioral signal. The circularity is in the evaluation chain. Section 4.1 states the 201 benchmark cases 'had informed rule-table revision, corpus development, and model selection' and that 'the benchmark informed Φ, which informed both adjudication and training.' The operational reference is therefore the authors' own policy-derived artifact, not an independent ground truth; agreement with it measures self-consistency with Φ. Section 3.2's audit cannot rule out that benchmark content reached training ('establish neither isolation nor decontamination'), so the primary 74.1%/69.0% gain is not shown to be an out-of-sample prediction. The external Oxford-40 check is too weak (static vignettes, p=0.383) to break the loop. Because the paper explicitly frames results as policy conformance rather than clinical validity, this is partial rather than total circularity: the method's construction is a real derivation, but the headline empirical claim is evaluated against a target built from the same policy that produced the training signal.
Assumptions & free parameters
free parameters (2)
- Class resampling weights =
E:2.0, U:1.5, R:1.0
- Training corpus composition =
3,000 dialogues; 625 E, 746 U, 1,629 R; 3 epochs
assumptions (4)
- domain assumption AAO Preferred Practice Patterns can be compiled into a 70-row operational rule table mapping trigger-symptom pairs to tiers.
- domain assumption Synthetic multi-turn dialogues generated from profiles are a sufficient stand-in for real telephone triage conversations.
- domain assumption The 201 Reddit narratives and the adjudicated reference provide a valid target for measuring triage capability.
- standard math Exact McNemar tests and paired confidence intervals are appropriate given n=201 and the observed discordance rates.
Cite this review
Pith. "Pith review of Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent." pith.science (2026). https://pith.science/paper/ZNSJRLPJ
@misc{pith2026260804772,
author = {Pith},
title = {Pith review of: Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZNSJRLPJ}},
note = {Machine review of arXiv:2608.04772}
}
read the original abstract
Scaling supervision for multi-turn medical agents is difficult because expert dialogue annotation is costly and clinical conversations are privacy-restricted. We introduce Guideline-as-Oracle (GAO), which compiles American Academy of Ophthalmology guidance into a 70-row operational rule table and uses it as the sole source of instance-level supervision for 3,000 training dialogues, reserving human labeling for evaluation. Because converting rules into dialogues is itself a design problem, we catalog eight construction strategies, including cited-row tier assignment, one-fact boundary pairs, metadata-only repair, and label repair, and characterize the evidential status of each: labeling mechanism, null, confounded, or evaluated only as a package. Fine-tuning a 9B backbone on this corpus yields GAO-Triage, improving agreement with a 201-case operational reference from 61.7% to 74.1% (exact McNemar p=0.0046) and emergent-case recall from 9.5% to 69.0%; the gains persist across a second seed and patient simulator. None of the seven general-purpose systems we test dominates GAO-Triage on both metrics, and GAO-Triage requires no frontier model at inference time. Permuting label-dialogue assignments collapses the model to a constant-routine predictor, indicating that the signal lies in guideline-derived assignment rather than dialogue surface form. Label repair coincides with the disappearance of a late-training safety degradation.
Figures
Reference graph
Works this paper leans on
-
[1]
Educational and Psychological Measurement , volume =
Cohen, Jacob , title =. Educational and Psychological Measurement , volume =
-
[2]
Goguen, Joseph A. and Meseguer, Jos. Security Policies and Security Models , booktitle =. doi:10.1109/SP.1982.10014 , year =
arXiv 1982
- [3]
-
[4]
Horowitz, Joel L. and Manski, Charles F. , title =. Econometrica , volume =. doi:10.2307/2951627 , year =
-
[5]
Heitjan, Daniel F. and Rubin, Donald B. , title =. The Annals of Statistics , volume =. doi:10.1214/aos/1176348396 , year =
-
[6]
and De Sa, Christopher and Wu, Sen and Selsam, Daniel and R\'
Ratner, Alexander J. and De Sa, Christopher and Wu, Sen and Selsam, Daniel and R\'. Data Programming: Creating Large Training Sets, Quickly , booktitle =
-
[7]
and Ehrenberg, Henry and Fries, Jason and Wu, Sen and R\'
Ratner, Alexander and Bach, Stephen H. and Ehrenberg, Henry and Fries, Jason and Wu, Sen and R\'. Snorkel: Rapid Training Data Creation with Weak Supervision , journal =
-
[8]
arXiv preprint arXiv:2212.08073 , year =
Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and others , title =. arXiv preprint arXiv:2212.08073 , year =
Show all 41 references
-
[9]
arXiv preprint arXiv:1707.06347 , year =
Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg , title =. arXiv preprint arXiv:1707.06347 , year =
-
[10]
and others , title =
Ouyang, Long and Wu, Jeff and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll L. and others , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[11]
and Ermon, Stefano and Finn, Chelsea , title =
Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D. and Ermon, Stefano and Finn, Chelsea , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[12]
arXiv preprint arXiv:2402.03300 , year =
Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and others , title =. arXiv preprint arXiv:2402.03300 , year =
-
[13]
arXiv preprint arXiv:2409.19256 , year =
Sheng, Guangming and Zhang, Chi and Ye, Zilingfeng and Wu, Xibin and Zhang, Wang and Zhang, Ru and Peng, Yanghua and Lin, Haibin and Wu, Chuan , title =. arXiv preprint arXiv:2409.19256 , year =
-
[14]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (System Demonstrations) , year =
Zheng, Yaowei and Zhang, Richong and Zhang, Junhao and Ye, Yanhan and Luo, Zheyan and Feng, Zhangchi and Ma, Yongqiang , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (System Demonstrations) , year =
-
[15]
Sara and Wei, Jason and others , title =
Singhal, Karan and Azizi, Shekoofeh and Tu, Tao and Mahdavi, S. Sara and Wei, Jason and others , title =. Nature , volume =
-
[16]
arXiv preprint arXiv:2303.13375 , year =
Nori, Harsha and King, Nicholas and McKinney, Scott Mayer and Carignan, Dean and Horvitz, Eric , title =. arXiv preprint arXiv:2303.13375 , year =
-
[17]
arXiv preprint arXiv:2401.05654 , year =
Tu, Tao and Palepu, Anil and Schaekermann, Mike and Saab, Khaled and Freyberg, Jan and others , title =. arXiv preprint arXiv:2401.05654 , year =
-
[18]
Nature Medicine , volume =
Thirunavukarasu, Arun James and Ting, Darren Shu Jeng and Elangovan, Kabilan and Gutierrez, Laura and Tan, Ting Fang and Ting, Daniel Shu Wei , title =. Nature Medicine , volume =
-
[19]
and Khashabi, Daniel and Hajishirzi, Hannaneh , title =
Wang, Yizhong and Kordi, Yeganeh and Mishra, Swaroop and Liu, Alisa and Smith, Noah A. and Khashabi, Daniel and Hajishirzi, Hannaneh , title =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , year =
-
[20]
, title =
Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and Dubois, Yann and Li, Xuechen and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B. , title =
-
[21]
Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , year =
Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and others , title =. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , year =
-
[22]
arXiv preprint arXiv:2311.01964 , year =
Zhou, Kun and Zhu, Yutao and Chen, Zhipeng and Chen, Wentong and Zhao, Wayne Xin and Chen, Xu and Lin, Yankai and Wen, Ji-Rong and Han, Jiawei , title =. arXiv preprint arXiv:2311.01964 , year =
-
[23]
arXiv preprint arXiv:2407.10671 , year =
Yang, An and Yang, Baosong and Hui, Binyuan and Zheng, Bo and Yu, Bowen and others , title =. arXiv preprint arXiv:2407.10671 , year =
-
[24]
2026 , note =
Preferred Practice Pattern Guidelines , howpublished =. 2026 , note =
2026
-
[25]
Lee, Joseph and Shang, Tianqi and Baik, Jae Young and Duong-Tran, Duy and Yang, Shu and Li, Lingyao and Shen, Li , title =
-
[26]
Xu, Cheng and Guan, Shuhao and Greene, Derek and Kechadi, M-Tahar , title =
-
[27]
Yu, Ailing and Yao, Lan and Liu, Jingnan and Chen, Zhe and Yin, Jiajun and Wang, Yuan and Liao, Xinhao and Ye, Zhiling and Li, Ji and Yue, Yun and Xiao, Hansong and Zhou, Hualei and Guo, Chunxiao and Wei, Peng and Liu, Junwei and Gu, Jinjie , title =
-
[28]
arXiv preprint arXiv:2505.09388 , year =
Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and others , title =. arXiv preprint arXiv:2505.09388 , year =
-
[29]
Scientific Data , volume =
Zhao, Zhengyun and Jin, Qiao and Chen, Fangyuan and Peng, Tuorui and Yu, Sheng , title =. Scientific Data , volume =. doi:10.1038/s41597-023-02814-8 , note =
-
[30]
Johnson, Alistair and Bulgarelli, Lucas and Pollard, Tom and Celi, Leo Anthony and Horng, Steven and Mark, Roger , title =
-
[31]
Whiting, Penny and Rutjes, Anne W. S. and Reitsma, Johannes B. and Bossuyt, Patrick M. M. and Kleijnen, Jos , title =. BMC Medical Research Methodology , volume =
-
[32]
and Rutjes, Anne W
Whiting, Penny F. and Rutjes, Anne W. S. and Westwood, Marie E. and Mallett, Susan and Deeks, Jonathan J. and Reitsma, Johannes B. and Leeflang, Mariska M. G. and Sterne, Jonathan A. C. and Bossuyt, Patrick M. M. , title =. Annals of Internal Medicine , volume =
-
[33]
and Amaral, Luis A
Goldberger, Ary L. and Amaral, Luis A. N. and Glass, Leon and Hausdorff, Jeffrey M. and Ivanov, Plamen Ch. and Mark, Roger G. and Mietus, Joseph E. and Moody, George B. and Peng, Chung-Kang and Stanley, H. Eugene , title =. Circulation , volume =
-
[34]
Cureus , volume =
Mittal, Surina and Aggarwal, Yakshi , title =. Cureus , volume =
-
[35]
Agenda-Based User Simulation for Bootstrapping a
Schatzmann, Jost and Thomson, Blaise and Weilhammer, Karl and Ye, Hui and Young, Steve , booktitle =. Agenda-Based User Simulation for Bootstrapping a. 2007 , publisher =
2007
-
[36]
Proceedings of the 19th Annual
Neural User Simulation for Corpus-based Policy Optimisation of Spoken Dialogue Systems , author =. Proceedings of the 19th Annual. 2018 , publisher =
2018
-
[37]
Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =
Teaching Language Models To Gather Information Proactively , author =. Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =. 2025 , publisher =
2025
-
[38]
and Sin, Jessica M
Li, Shuyue Stella and Mun, Jimin and Brahman, Faeze and Hosseini, Pedram and Thomas, Bryceton G. and Sin, Jessica M. and Ren, Bing and Ilgen, Jonathan S. and Tsvetkov, Yulia and Sap, Maarten , booktitle =. 2025 , url =
2025
-
[39]
arXiv preprint arXiv:2405.07960 , year =
Schmidgall, Samuel and Ziaei, Rojin and Harris, Carl and Reis, Eduardo and Jopling, Jeffrey and Moor, Michael , title =. arXiv preprint arXiv:2405.07960 , year =
-
[40]
arXiv preprint arXiv:2506.22405 , year =
Nori, Harsha and Daswani, Mayank and Kelly, Christopher and Lundberg, Scott and others , title =. arXiv preprint arXiv:2506.22405 , year =
-
[41]
npj Digital Medicine , year =
Kini, Annapoorna and Vengrenyuk, Andriy and Pineda, Derek and Vengrenyuk, Yuliya and others , title =. npj Digital Medicine , year =
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.