Computation and Language · arXiv:2610.30947 v1 · submitted 22 Oct 2026 · 13 references
RetiJev: Fast and Source-Grounded Ophthalmic Decision Models
The paper says
“Ophthalmic clinical decision support requires fast, accurate, and privacy-preserving inference, yet autoregressive large language models (LLMs) incur substantial latency and computational overhead for decisions among predefined actions. We therefore investigate a decision-oriented alternative to generative reasoning.”
more of the abstract
Specifically, we first introduce RetiJev-Data, a source-grounded ophthalmology corpus comprising 78,158 clinical-decision and diagnosis questions derived from articles in PubMed Central and a MeSH vocabulary covering 531 eye diseases, and 50,074 guideline-grading questions spanning 11 grading systems, along with a 24,662-question evaluation benchmark. Based on this, we develop RetiJev, a family of 0.8B-9B decision models that directly score candidate answers without autoregressive decoding. Across all model sizes, RetiJev outperforms both a 27B LLM and the commercial Jev 1.13 decision model on guideline grading by 10.4-23.0 points. RetiJev-9B further achieves the highest accuracy on clinical decisions, improving over both baselines by 5.4-6.5 points while matching them on diagnosis. It answers a case in 158 ms on one GPU, 3.5x faster than LLM generation and 470x faster than reasoning. Models are released at https://github.com/Awenbocc/RetiJev.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 23 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A well-engineered systems paper with a genuinely new dataset, but the clinical decision results should be read as agreement with an LLM proxy, not clinical ground truth. This line is wrong
3 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 13 references say.
The paper says, at Abstract and Section 5.2
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.64):
“Limitations Decision labels are source-grounded and cross-verified, but not clinician-adjudicated, and all patient states are synthetic.”
Pith says: Objection 1 of 3 · major
The abstract states that RetiJev-9B achieves the 'highest accuracy on clinical decisions' without mentioning that the labels are LLM-derived and not clinician-verified. The central claim of a 5.4–6.5 point improvement on clinical decisions is a claim about agreement with a synthetic benchmark, not about clinical correctness.
more of objection 1
Given the Limitations section (Section 6) honestly acknowledges this, the mismatch between the abstract's unqualified language and the later caveat is a presentation issue that should be corrected. I recommend adding a sentence to the abstract such as 'On a synthetic, source-grounded benchmark with LLM-verified labels, RetiJev-9B outperforms...'.
The paper says, at Table 2 and §5.2 (Diagnosis)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.86):
“Because both diagnosis sets are small, these comparisons have limited precision.”
Pith says: Objection 2 of 3 · major
The diagnosis test sets contain only 164 (Unseen) and 224 (Seen) questions. The paper states that 'these comparisons have limited precision' but nonetheless concludes that RetiJev-9B 'matches the baselines on diagnosis.' This conclusion is fragile given the wide confidence intervals that must accompany such small sets. I ask the authors to report bootstrap confidence intervals for the diagnosis results and to phrase the conclusion as 'no significant difference was detected given the limited sample size,' rather than 'matches.'
The paper says, at Section 6
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.77):
“Limitations Decision labels are source-grounded and cross-verified, but not clinician-adjudicated, and all patient states are synthetic.”
Pith says: Objection 3 of 3 · major
The paper states that decision labels are 'source-grounded and cross-verified, but not clinician-adjudicated' and that 'RetiJev has also not been evaluated prospectively.' These are important limitations that are stated clearly. I consider this a strength of the manuscript's honesty. I do not raise this as a flaw; however, readers may miss it, so I support the authors' existing plan to restrict claims to benchmark performance.
7 minor objections
The paper says, at Figure 1
Pith says: minor objection 1 of 7
The x-axis label 'time to answer one case (log scale)' uses a log scale that compresses the 27B-with-reasoning bar (74.5 s) relative to the others. Consider adding a secondary linear axis annotation or a table of exact values for clarity.
The paper says, at Section 4.2, Eq. (3)
Pith says: minor objection 2 of 7
The RPS formula uses mk for the number of candidates. For ordered ratings the index mk should be the number of levels. Please clarify that mk is the same quantity used in the softmax, and that the denominator is mk−1 as is standard for RPS.
show 5 more
The paper says, at Table 2 caption
Pith says: minor objection 3 of 7
The caption says 'Latency is the median time per case (ms) on a single H200 GPU at batch size 1, with the 27B baseline evaluated using the letter-scoring protocol.' It would be helpful to also state that the 27B latency for letter scoring is 247 ms (from Table 3) to avoid forcing the reader to cross-reference.
The paper says, at Section 5.4, Calibration and Deferral
Pith says: minor objection 4 of 7
The risk–coverage AUC values (0.072, 0.045) and the statement that the deferred 20% contain 51% and 58% of errors are informative. Please clarify how the 20% coverage threshold was chosen (e.g., by sweeping over confidence thresholds or using a pre-specified percentile).
The paper says, at Appendix H, Table 10
Pith says: minor objection 5 of 7
The pilot models are described as 'distinct from the final models in Table 2.' The reader may wonder whether the final models also underwent the same augmentation. The text later clarifies that all final models use paraphrase augmentation, but a forward reference in this paragraph would help.
The paper says, at Appendix I, Generator-family bias
Pith says: minor objection 6 of 7
The analysis shows that all models score lower on Qwen-written questions, which contain all 961 per-candidate diagnosis questions. The authors note that 'this task imbalance prevents attributing the difference to model-family bias alone.' This is a fair point. Consider reporting results on only the non-diagnosis subsets to further isolate any bias.
The paper says, at General
Pith says: minor objection 7 of 7
Several references are to arXiv preprints or GitHub repositories without dates. Where possible, verify that the most recent version is cited and add the arXiv identifier or commit hash for replicability.
What would settle it
Have a panel of board-certified ophthalmologists independently label 200 randomly sampled cases from Decision-Unseen (each case providing the preferred action from the catalog). Compare RetiJev-9B's accuracy against these clinician labels vs. the benchmark labels. A drop of >5 points would indicate the benchmark overestimates clinical decision quality.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 23 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 3 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know two things about this paper. First, the dataset is the real contribution: 78k source-grounded clinical decision questions and 50k guideline grading questions, built from PubMed Central with traceable citations. That is a lot of work and it will be useful to the community. Second, the central claim—that RetiJev beats a 27B LLM and Jev 1.13 on clinical decisions by 5–6 points—rests on labels generated by an LLM pipeline and cross-verified by other LLMs, without clinician adjudication. The authors are honest about this in the limitations section, but a reader should not mistake those numbers for clinically validated accuracy.
What is actually new: the RetiJev-Data corpus, the multi-task training with teacher distillation and paraphrase consistency, and the speed numbers (158 ms per case on one GPU with 9B parameters). The guideline grading results are the cleanest part of the paper because those labels are computed deterministically from published rules and verified by parameter re-extraction. On that task, every RetiJev model substantially beats the large baselines, and the robustness to OOD renderers is convincing.
Where it gets softer: the clinical decision evaluation measures agreement with an LLM-derived standard. The stress-test concern is fair—without clinician labels, a 5-point improvement over a 27B model on synthetic questions does not automatically mean better clinical care. The diagnosis test sets are also small (164 and 224 questions), and the teacher is from the same model family as one baseline, though the authors test for generator-family bias and report it. All of this is disclosed, but it limits what you can conclude from the headline numbers.
Who this is for: researchers working on efficient inference for structured clinical tasks, or anyone building domain-specific decision models. The dataset alone makes this worth reading. If you are a clinician hoping to deploy this, you will need to wait for real-world validation.
Recommendation: this paper deserves a serious referee. It is a solid systems paper with a reproducible dataset, honest limitations, and a clear engineering contribution. The clinical claims need to be read with appropriate caution, but the paper earns its place in the review process.
The strongest objection
The central claim of superiority on clinical decisions (5.4–6.5 point improvement over the 27B LLM and Jev 1.13) is derived from RetiJev-Bench, where labels are generated by an LLM pipeline and cross-verified by other LLMs, with no clinician involvement (Section 3.1.2). The paper states: 'Decision labels are source-grounded and cross-verified, but not clinician-adjudicated' (Limitations, [b250]). This means the reported accuracy measures agreement with an LLM-derived standard, not clinical correctness. The distillation teacher is also from the same family as one baseline (Qwen3.8-27B). While the pipeline reduces noise, it inherits any systematic bias of the LLMs producing and verifying labels. The benchmark's construct validity for clinical decision-making is unestablished. Without clinician-adjudicated labels, the 5.4–6.5 point improvements cannot be confidently interpreted as better clinical decisions.
The reader identified synthetic data as the weakest assumption; I agree but specify that the label derivation pipeline (LLM-generated and cross-verified without clinician input) means the clinical decision benchmark is an unvalidated proxy. The condition in the reader's verdict already covers this concern, so no change to the verdict is needed. The paper itself acknowledges the limitation, and a conditional verdict is appropriate.
The referee's report
If the benchmark results transfer to clinical practice, RetiJev could provide a fast, local alternative to generative LLMs for structured ophthalmic decisions, with substantial efficiency gains. The release of a large, source-grounded dataset and the decision-model family supports further research. The thorough ablations, calibration analysis, and latency measurements are strengths. However, the clinical-decision labels are not clinician-adjudicated and all evaluation is on synthetic data, so the significance for real-world deployment remains to be established.
Pith's reading of the paper (a machine's summary, not the authors')
RetiJev introduces a family of compact decision models (0.8B-9B parameters) that directly score predefined answer options instead of generating free-text responses. The authors construct RetiJev-Data, a source-grounded corpus of 78,158 clinical-decision and diagnosis questions from PubMed Central and MeSH, plus 50,074 guideline-grading questions across 11 grading systems. Using a shared pointer head and a training objective combining cross-entropy, ordinal supervision, teacher distillation, and paraphrase consistency, the models achieve state-of-the-art accuracy on guideline grading (95.9%-97.3% on unseen renderers) and clinical action selection (84.9% on unseen rules), surpassing a 27B LLM and the commercial Jev 1.13 model. Crucially, RetiJev-9B runs in 158 ms per case on a single GPU—3.5× faster than LLM generation and 470× faster than reasoning—while RetiJev-0.8B fits in 3.3 GB of GPU memory and can run on CPU. The authors argue that direct candidate scoring is a faster, more privacy-preserving alternative to generative LLMs for structured clinical decisions, but acknowledge that all labels are synthetic and clinician-adjudicated validation is still needed.
What carries the argument. RetiJev uses a frozen Qwen3.5 backbone with trainable LoRA adapters and a shared pointer head that projects hidden states from a decision token ⟨D⟩ and option-closing tokens ⟨/O⟩ into a query-key space, scoring candidates via scaled dot-product attention. The training objective combines hard cross-entropy labels, ranked probability score (RPS) for ordinal ratings, knowledge distillation from a frozen 27B teacher for clinical decisions, and a paraphrase-consistency loss that enforces invariant predictions across reworded patient descriptions. The dataset, RetiJev-Data, is built by extracting conditional clinical rules from 65,759 PubMed Central articles, generating synthetic patient cases from those rules, and verifying labels through cross-family LLM agreement (two families agree, or three families agree with a supporting quote). For guideline grading, 11 published grading systems are encoded as Python functions that compute labels from structured parameters; clinical notes are generated from those parameters by an LLM and pass a parameter-recovery check.
Core claim. The paper establishes that for ophthalmic decision tasks with predefined answer options, a compact decision model that directly scores candidate answers can match or exceed the accuracy of much larger generative models while being orders of magnitude faster. Across 11 guideline-grading systems, every RetiJev model size (0.8B, 2B, 9B) significantly outperforms a 27B LLM and the commercial Jev 1.13 API by 10.4-23.0 percentage points. On clinical decision tasks (action selection from source-grounded rules), RetiJev-9B achieves 84.9% accuracy on unseen rule families, improving over both baselines by 5.4-6.5 points. The key insight is that by framing decisions as scoring rather than generation, the model avoids autoregressive decoding entirely, enabling single-pass inference with a shared state cache across multiple questions per case. The models require only 133-158 ms per case on one GPU and the 0.8B variant runs on eight CPU cores, supporting local deployment without transmitting patient data.
The premise it rests on. The synthetic patient cases and their labels, derived from LLM-extracted rules and verified by cross-family agreement, are sufficiently representative of real-world ophthalmic encounters to validate the model's clinical decision-making performance—this assumption is load-bearing because no prospective or clinician-adjudicated validation has been performed.
If the paper is right (Pith's inferences, not the authors')
Compact decision models can replace large generative LLMs for many structured clinical decision tasks, dramatically reducing hardware and latency requirements, and enabling point-of-care deployment on local devices without cloud connectivity.
The source-grounded dataset construction method—extracting rules from literature, generating cases, and verifying labels via cross-model agreement—could be applied to other medical specialties to create training data for decision models without expensive clinician annotation.
The paraphrase-consistency loss and teacher distillation jointly improved accuracy and calibration (ECE from 0.186 to 0.033 on Decision-Unseen), suggesting a general technique for training robust decision models from limited hard labels.
Because RetiJev reuses a shared patient state across multiple questions and scores candidates in a single prefill, it scales sublinearly with the number of questions per case (9B model goes from 138 ms at 1 question to 198 ms at 20 questions), making it practical for multi-question clinical workflows.
The model's confidence scores are well-calibrated (ECE 0.016 for 9B) and its risk–coverage curve enables selective deferral: at 80% coverage, errors drop by half, allowing uncertain cases to be escalated to a human while the model handles the rest autonomously.
The same direct-scoring paradigm could be extended to other domains with standardized decision options, such as triage in emergency medicine, drug interaction checking, or radiology reporting guidelines, using similar source-grounded data pipelines.
Because the evaluation relies entirely on synthetic cases and machine-verified labels, the true clinical edge over large LLMs remains unproven until tested on real patient records or in prospective studies. The 10-23 point advantage on grading may narrow when models face the ambiguity and incompleteness of actual clinical notes.
The finding that small models (0.8B) can match 27B LLMs on seen clinical decisions but struggle on rare diagnoses (Diagnosis-Unseen) suggests that disease coverage, not architecture, is the bottleneck—a hint that targeted data augmentation (e.g., from the 531 MeSH diseases) can further close the gap without scaling up model size.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Ophthalmic clinical decision support requires fast, accurate, and privacy-preserving inference, yet autoregressive large language models (LLMs) incur substantial latency and computational overhead for decisions among predefined actions. We therefore investigate a decision-oriented alternative to generative reasoning. Specifically, we first introduce RetiJev-Data, a source-grounded ophthalmology corpus comprising 78,158 clinical-decision and diagnosis questions derived from articles in PubMed Central and a MeSH vocabulary covering 531 eye diseases, and 50,074 guideline-grading questions spanning 11 grading systems, along with a 24,662-question evaluation benchmark. Based on this, we develop RetiJev, a family of 0.8B-9B decision models that directly score candidate answers without autoregressive decoding. Across all model sizes, RetiJev outperforms both a 27B LLM and the commercial Jev 1.13 decision model on guideline grading by 10.4-23.0 points. RetiJev-9B further achieves the highest accuracy on clinical decisions, improving over both baselines by 5.4-6.5 points while matching them on diagnosis. It answers a case in 158 ms on one GPU, 3.5x faster than LLM generation and 470x faster than reasoning. Models are released at https://github.com/Awenbocc/RetiJev.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-23
The authors' main claim, as the classifier picked it out 0.72
“Across all model sizes, RetiJev outperforms both a 27B LLM and the commercial the classifier 1.13 decision model on guideline grading by 10.4-23.0 points.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.11 · human participants 0.11 · controlled comparison 0.88 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.95
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Ophthalmic clinical decision support requires fast, accurate, and privacy-preserving inference, yet autoregressive large language models (LLMs) incur substantial latency and computational overhead for decisions among predefined actions. | background 0.93 | 0.09 | 0.04 | asserted 1.00 |
| We therefore investigate a decision-oriented alternative to generative reasoning. | contribution 0.69 | 0.28 | 0.04 | asserted 0.51 |
| Specifically, we first introduce RetiJev-Data, a source-grounded ophthalmology corpus comprising 78,158 clinical-decision and diagnosis questions derived from articles in PubMed Central and a MeSH vocabulary covering 531 eye diseases, and 50,074 guideline-grading questions spanning 11 grading systems, along with a 24,662-question evaluation benchmark. | method 0.65 | 0.94 | 0.04 | asserted 1.00 |
| Based on this, we develop RetiJev, a family of 0.8B-9B decision models that directly score candidate answers without autoregressive decoding. | method 0.75 | 0.95 | 0.08 | asserted 1.00 |
| Across all model sizes, RetiJev outperforms both a 27B LLM and the commercial Jev 1.13 decision model on guideline grading by 10.4-23.0 points. | result 1.00 | 0.95 | 0.29 | asserted 1.00 |
| RetiJev-9B further achieves the highest accuracy on clinical decisions, improving over both baselines by 5.4-6.5 points while matching them on diagnosis. | result 1.00 | 0.95 | 0.27 | asserted 1.00 |
| It answers a case in 158 ms on one GPU, 3.5x faster than LLM generation and 470x faster than reasoning. | result 1.00 | 0.94 | 0.15 | asserted 1.00 |
| Models are released at https://github.com/Awenbocc/RetiJev. | contribution 0.87 | 0.73 | 0.02 | asserted 0.97 |
What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.74
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-23: 1 request, 6,449 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (13)
- [8] Li, X., Chang, Q., Ning, J., Xu, C., Zhang, S., Zhang, Y ., Luo, L., and Lin, H. Specialized decision models vs. general-purpose llms: Benchmarking jev across knowl- edge, reasoning, and multilingual tasks.arXiv preprint arXiv:2610.11978, 2026a. Li, Y ., Miao, Y ., Krishnan, R... on PithPith's reviewarXiv
- [12] Warner, B., Chaffin, A., Clavi ´e, B., et al. Smarter, bet- ter, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference.arXiv preprint arXiv:2412.13663, on PithPith's reviewarXiv
- [13] 12 RetiJev: Fast and Source-Grounded Ophthalmic Decision Models A. Action Catalog Table 5.Clinical-decision tasks and question formats.Action catalogs define the available options, from which case-specific candidates are selected. Urgency uses an ordinal scale, while diagnosis... on Pith
- [1997] Nori, H., King, N., McKinney, S. M., Carignan, D., and Horvitz, E. Capabilities of GPT-4 on medical challenge problems.arXiv preprint arXiv:2303.13375, on PithPith's reviewarXiv
- [2013] Gao, W., Deng, Z., Niu, Z., et al. OphGLM: Train- ing an ophthalmology large language-and-vision assis- tant based on instructions and dialogue.arXiv preprint arXiv:2306.12174, on PithPith's reviewarXiv
- [2015] Wang, X., Liu, B., Wang, M., Zhang, Z., Zhu, C., Fu, H., and Liu, J. Towards clinically interpretable ophthalmic vqa via spatially-grounded lesion evidence.arXiv preprint arXiv:2605.22414, on PithPith's reviewarXiv
- [2016] Gunasekar, S., Zhang, Y ., Aneja, J., et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, on PithPith's reviewarXiv
- [2018] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capa- bility in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, on PithPith's reviewarXiv
- [2019] Kim, H., Ai, X., Srinivasan, S., Gilson, A., et al. LEME: Open large language models for ophthalmology with ad- vanced reasoning and clinical validation.arXiv preprint arXiv:2410.03740, on PitharXiv
- [2021] Madrid-Garc´ıa, A. and Merino-Barbancho, B. Jev in medicine: A benchmark evaluation. preliminary results. arXiv preprint arXiv:2609.34024, on PithPith's reviewarXiv
- [2023] Burton, M. J., Ramke, J., Marques, A. P., et al. The Lancet Global Health Commission on Global Eye Health: Vision beyond 2020.The Lancet Global Health, 9(4):e489–e551, on Pith
- [2024] Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, on PithPith's reviewarXiv
- [2025] Deka, P. From retrieval to typed decisions: Calibrated system one models from biomedical sentence encoders. arXiv preprint arXiv:2610.02486, on PithPith's reviewarXiv
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment