{"id":"794ab638-2ea8-4c6b-a6d3-0b79caa6ed28","arxiv_id":"2412.02273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Deep reinforcement learning agents can request lab tests sequentially and diagnose anemia subtypes on real hospital records, generating explainable diagnostic pathways.","lead":"Researchers trained deep reinforcement learning agents to request blood tests one at a time and reach an anemia subtype diagnosis, then tested them on records from a Paris hospital. The agents produced step-by-step diagnostic pathways with accuracy and F1 scores close to standard machine-learning classifiers that see all tests at once.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 'best DQN' is selected as the maximum over three validation-selection criteria and ten seeds—not a prespecified policy—so the headline SOTA comparison is not yet established.","rationale":"Stress-test focus. The abstract's competitive claim requires that a DRL policy chosen by a reproducible rule beats or matches batch classifiers. The reader's weakest assumption was imperfect ground-truth labels; I agree that is a limitation, but it does not differentially bias the DRL-vs-SOTA comparison because the same labels define rewards and evaluation for every method. The selection issue does differentially affect the comparison. Tables D.1 and D.3 show the protocol: ten seeds and three alternative checkpoint-selection goals; Table 2 then reports the single most favorable result with ±0.00 error. In contrast to D.1, D.2 and D.3 report zero variance, which is internally inconsistent with running ten seeds. If a fixed selection rule had been used, the DQN's F1 could be 53.03 instead of 63.48, which would invert the headline comparison with XGBoost. The paper's own limitations section acknowledges small data and class imbalance, and the code is available, which makes the requested reanalysis feasible. I would keep the reader's CONDITIONAL verdict but tighten the condition: pre-register the selection metric and report the full seed distribution; then the competitive claim can be fairly evaluated.","tokens_in":17112,"tokens_out":7557,"duration_ms":83831,"concrete_test":"Using the released code, rerun the fine-tuned Dueling DQN-PER experiment with the same 10 seeds, but with a single pre-registered checkpoint-selection metric (e.g., validation ROC-AUC) and no model/variant selection after seeing the test set. Report the mean±std over the 10 seeds of test F1, ROC-AUC, accuracy, and MEL, and state how many of the 10 runs exceed XGBoost's F1=61.77 and ROC-AUC=78.61. If the mean or a majority of runs do not meet or exceed those thresholds, Table 2's zero-variance 'best DQN' row is a selection artifact and the competitive claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that DRL methods 'perform competitively with state-of-the-art methods' is supported in Table 2 by a single row for 'Best DQN (Fine-tuned Dueling DQN-PER)' with F1=63.48, ROC-AUC=77.67, and MEL=3.19. Every value in that row has ±0.00 standard deviation. This is not a mean over the ten seeds described in Sec. 3.5; it is one selected run/checkpoint. Appendix D.3 shows the same model evaluated under three different checkpoint-selection goals ('Accuracy', 'F1', 'ROC-AUC'), and the Table 2 row corresponds to the most favorable of those three rows. With ten seeds and three selection goals, the headline is effectively the best of roughly 30 candidate selections, while the comparators (RF, XGBoost, FFNN) are reported as standard tuned models. If a fixed selection rule had been used, the comparison could change materially: the fine-tuned DQN-PER selected by validation F1 has test F1=53.03 (D.3), below XGBoost's 61.77, and the accuracy-selected row has F1=51.75. The imperfect label matcher (Sec. 5.1, 86% precision) is a real limitation, but it applies to all compared models; the selection asymmetry directly inflates the DRL side of the claimed comparison. A proof-of-concept claim could survive; the 'competitive' claim, as reported, is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formulates anemia differential diagnosis as a sequential decision-making problem. It defines an MDP with feature-query actions and diagnostic actions, trains Dueling DQN and Dueling DDQN agents with Prioritized Experience Replay, and evaluates them on a synthetic dataset generated from an expert-defined decision tree and on a real-world dataset of 1,127 patients from a Paris hospital. Four training scenarios are compared: synthetic-only training, real-world-only training, direct transfer from synthetic to real data, and fine-tuning on real data. The authors report F1, ROC-AUC, accuracy, and mean episode length, and compare against random, tree-based, and standard machine-learning baselines. The central claim is that deep reinforcement learning performs competitively with state-of-the-art methods while additionally generating interpretable, step-by-step diagnostic pathways.","tokens_in":17457,"tokens_out":5153,"duration_ms":56449,"significance":"If the central claim were established, this would be a useful contribution to clinical decision support: it extends an earlier synthetic-only pipeline to a real-world clinical data warehouse, provides public code and data-description resources, and demonstrates interpretable sequential test selection with transfer learning. The transparent pathway generation is a genuine advantage over black-box classifiers, and the real-world evaluation with ICD-10 and NLP-derived labels is valuable despite label noise. However, the headline comparison in Table 2 is currently undermined by post hoc checkpoint selection and by zero-variance reporting for the selected DQN model, so the claim that DRL methods 'perform competitively' is not yet supported by the evidence as presented.","major_comments":[{"comment":"The 'Best DQN' row in Table 2 is not a prespecified model: it is the maximum over three validation-selection criteria and ten seeds. The text states that the row is the best DQN in terms of F1 score, but Table D.3 shows the fine-tuned Dueling DQN-PER achieves test F1 = 63.48 when checkpoints are selected by ROC-AUC, while selection by F1 yields test F1 = 53.03 and selection by accuracy yields 51.75. The reported row is therefore the most favorable of several candidate selections, whereas the comparators (RF, XGBoost, FFNN) are standard tuned models reported as averages. This asymmetry invalidates the claimed comparison. The authors should prespecify a single checkpoint-selection metric and report the mean and standard deviation over the ten seeds under that fixed protocol, or alternatively report all three selection rows from D.3 as the primary comparison.","section":"§4.2 and Table D.3"},{"comment":"The real-world ground-truth labels are derived from ICD-10 codes and a Medkit-based matcher with a reported precision of 86%, and 42.9% of the matcher's errors are 'uncertain diagnoses' where the detected term does not represent a confirmed diagnosis. Because these labels define both the reward signal used to train the DQN and the evaluation metrics in Table 2, the absolute F1 and ROC-AUC values are sensitive to this label noise. The authors acknowledge the limitation in Section 5.1 but do not quantify its impact. I request a robustness check on a manually verified random subset of the test set, or at least separate reporting of matcher precision for the test labels. Without such an analysis, the numerical performance of all models, including the DQN, is not fully reliable.","section":"§5.1"},{"comment":"The DQN rows in Tables D.2 and D.3 are reported with a standard deviation of ±0.00, while the same model class in Table D.1 shows F1 standard deviations of up to 6.69 points across ten seeds. This indicates that the DQN tables report one selected checkpoint per scenario rather than the distribution over the ten training runs. Zero-variance reporting obscures the run-to-run instability of the DQN method and undermines comparisons between training scenarios, such as the claimed improvement from fine-tuning. The authors should report mean and standard deviation over all ten runs for each selection metric, as they already do in Table D.1.","section":"Tables D.2, D.3, and Table 2"}],"minor_comments":[{"comment":"There is a typo in the author list: 'Anne-Isablle Tropeanoc' should be 'Anne-Isabelle Tropeano'.","section":"Author list"},{"comment":"The sentence 'models were selected based on their F1 scores, specifically those closest to the mean F1 score reported in Table 1' is ambiguous; it is unclear whether the selection is per seed, whether the chosen checkpoint is close to the mean validation F1 or the mean test F1, and how 'closest' is defined.","section":"§4.2 and Table D.2"},{"comment":"The paper does not report the total number of training timesteps or the checkpoint interval, although these are needed to understand how many checkpoints are evaluated; adding these values would improve reproducibility.","section":"Table B.1 and §3.5"},{"comment":"The baseline 'RWD-trained DT' in Table 2 is not defined in the table or its caption; the text later uses 'RWD-trained decision tree', so please make the abbreviation explicit.","section":"Table 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-structured proof-of-concept, but the headline comparison is not yet established because of the checkpoint-selection protocol and the zero-variance reporting. The fix is straightforward: prespecify a single validation-selection rule and report mean and standard deviation across seeds, as the authors already do in Table D.1. I agree with the reader's assessment that there is no circularity between the synthetic decision tree and the real-world evaluation, since the real-world labels come from independent ICD-10 and NLP sources. If the authors address the selection protocol and add a label-noise robustness check, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate extension of the authors' synthetic-data work to real-world EHR data, and the fine-tuning result is genuinely interesting. But the headline comparison in Table 2 is not yet established, because the DQN row is the best of three validation-selection criteria (and likely the best of ten seeds), reported as a single zero-variance number while baselines are averaged. The paper deserves a serious referee, but the authors need to fix the evaluation protocol.\n\nWhat's actually new: the prior paper [3] was synthetic only. Here they build an expert-defined decision tree, generate a synthetic dataset, then evaluate on 1,127 real patients from a Paris hospital across three scenarios: train on synthetic, apply directly to real data; fine-tune on real data; train from scratch on real data. The fine-tuned Dueling DQN-PER improves recall on sickle cell anemia from 9% to 83%, which is a concrete, clinically meaningful result. The code and data-extraction queries are on GitHub, and the limitations section is honest about small class sizes and the 86% precision of the NLP label matcher.\n\nThe soft spot: Table 2's 'Best DQN' row is one run/checkpoint, not a mean over seeds. Appendix D.3 shows the same fine-tuned model under three validation-selection goals: validation-F1 selection gives test F1 53.03, accuracy selection gives 51.75, and the ROC-AUC selection used in Table 2 gives 63.48. With ten seeds and three goals, the headline is effectively the best of ~30 candidate selections, while XGBoost's 61.77 is a fixed model evaluated once. If the authors had prespecified a single selection rule, the comparison could look very different. The label noise is a real limitation, but it applies to all models equally, so it doesn't explain the asymmetry.\n\nWho this is for: researchers working on sequential clinical decision support, RL for diagnostic reasoning, and transfer learning from synthetic to real clinical data. They will get a well-described, reproducible setup and a useful discussion of pathway interpretability.\n\nBottom line: a solid proof-of-concept, not a settled competitive claim. Send to peer review, but require the authors to report a prespecified selection rule and full distributions across seeds and selection criteria, or the SOTA comparison should be softened.","headline":"A credible proof-of-concept that DRL can learn sequential test-request policies for anemia on real EHR data, but the headline 'competitive with SOTA' claim is undercut by post hoc selection of the best run/checkpoint in Table 2.","tokens_in":17972,"tokens_out":2593,"would_cite":false,"duration_ms":26077,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep reinforcement learning can guide anemia diagnosis test by test, matching batch classifiers while outputting an ordered pathway.","keywords":["Anemia","Diagnostic pathway","Clinical decision support","Deep Q-network","Reinforcement learning","Electronic health records","Transfer learning","Differential diagnosis"],"falsifier":"Take a random sample of about 200 of the 1,127 real-world patients, have two clinicians assign the final anemia sub-type from the full chart without seeing the model's output, and recompute the fine-tuned Dueling DQN-PER's F1 and ROC-AUC against that adjudicated gold standard; if the scores fall to chance level or clearly below the best batch classifier's, the paper's central claim is falsified.","tokens_in":16921,"feed_emoji":"🩺","tokens_out":13320,"duration_ms":127103,"temperature":0.7,"pith_summary":"This paper tries to establish that a deep reinforcement learning agent can act like a clinician: instead of labeling a patient from all lab values at once, it requests one test at a time and stops when it can commit to a diagnosis. On a synthetic dataset generated from an expert-defined anemia decision tree, the agent nearly matches the oracle tree, with accuracy and F1 around 99 percent. On 1,127 real electronic health record patients, the best fine-tuned model reaches a macro-F1 (a class-averaged balance of precision and recall) of 63.48 and a ROC-AUC (a rank-based measure of class separation) of 77.67 while using an average of 3.19 tests per episode, outperforming random forests on F1 and coming close to the best batch classifier on ROC-AUC. The deeper claim is that a sequential strategy can be as accurate as batch classification while also being explainable, because every episode is an ordered list of requested tests. A secondary claim is that pre-training on synthetic data and fine-tuning on a small real cohort beats training on the real cohort alone.","feed_headline":"Reinforcement learning matches batch classifiers on anemia diagnosis","feed_subtitle":"The best agent orders about 3 tests per patient and still matches batch models on F1.","key_machinery":"The load-bearing object is the Markov decision process formulation of diagnosis, together with the Dueling DQN-PER agent that solves it. In the MDP, the state is a fixed-size vector of feature values with -1 for unqueried features, the actions are either querying one of the 11 lab features or committing to a diagnosis, and the reward structure couples diagnostic accuracy with test count. Dueling DQN is a deep Q-network that splits the estimated value of a state into a state-value stream and an advantage stream for each action, and prioritized experience replay makes the agent sample the most informative past episodes; together they let the policy learn which test to request next from partial observations. The trajectory of actions is exactly the diagnostic pathway that the paper offers as the transparent counterpart to a batch classifier.","core_discovery":"The authors formulate differential anemia diagnosis as a Markov decision process in which a patient's record is revealed feature by feature. At each step the state is the vector of lab values seen so far (with -1 marking values not yet requested); the possible actions are to ask for one of 11 anemia-related measurements or to declare one of the diagnosis classes; and the reward is +1 for a correct final diagnosis, -1 for a wrong one, and a small per-test penalty so that shorter pathways are preferred. Training a Dueling DQN with prioritized experience replay on 69,879 synthetic instances, then fine-tuning on the real cohort, produces a policy that, on the held-out real test set, achieves F1 63.48 and ROC-AUC 77.67 with mean episode length 3.19. The generated pathways are the interpretable core: the model cannot see all features at once, because it is rewarded for reaching the right class with the fewest tests. The authors interpret the gains on sickle cell anemia—recall rising from 9 percent on the synthetic-only model to 83 percent after fine-tuning, against 0 percent for the expert decision tree—as evidence that learned sequential policies can outperform a static guideline tree when the tree's thresholds or missing-value handling do not match real patients.","pith_inferences":["A direct extension the paper leaves implicit is to replace the uniform per-test penalty with real monetary or time costs for each lab test; the reward function already provides the hook, and this would turn 'shorter pathway' into 'cheaper or faster pathway.'","The paper's own limitation note that laboratory tests are usually ordered as panels suggests a panel-aware action space would change both the mean episode length and the clinical usefulness of the pathways the agent produces.","Because the manual note-adjudication set was used only to evaluate term detection, a natural follow-up is to adjudicate a sample of the final diagnosis labels themselves; this would reveal whether the reported F1 advantage over batch classifiers is real or an artifact of label noise.","The expert tree's failure on sickle cell anemia cases with missing ferritin suggests the learned policy is partly compensating for missing data; a complete-case analysis would separate missing-data handling from disease-specific learning."],"forward_implications":["A clinician using this policy would receive not only a suggested diagnosis but a short, ordered list of tests, 3.19 on average in the best real-world model, so fewer unwarranted lab orders could be placed.","When real data are scarce, synthetic pre-training followed by fine-tuning is a workable route: the paper reports that this beats both training only on synthetic data and training only on the real cohort, and the improvement is largest for rare classes such as sickle cell anemia.","Sequential decision policies can beat static expert decision trees on real data: the expert-defined tree missed every sickle cell anemia case, while the fine-tuned model reached 83 percent recall, suggesting that learned thresholds handle missing and atypical values better.","Because the state and action space are defined only by a feature list and a diagnosis list, the same MDP should transfer to other lab-based differential diagnoses, which the authors name as their planned extension.","If these claims hold, an explainable sequential agent could complement batch classifiers in clinical decision support, preserving accuracy while adding step-by-step reasoning."],"supporting_citations":[{"why":"Supplies the 'wayfinding' framing that diagnostic AI should provide step-by-step guidance rather than only predicting a label, which is the objective the current paper's pathways are built to meet.","marker":"[2]"},{"why":"Provides the prior deep reinforcement learning study on anemia and systemic lupus erythematosus whose synthetic dataset methodology and Dueling DQN-PER choices the current paper extends to real-world data.","marker":"[3]"},{"why":"Supplies the Deep Q-Network algorithm that approximates the Q-value function over the large anemia state space.","marker":"[30]"},{"why":"Supplies the Double DQN stabilization used in one of the two model families tested in the current paper.","marker":"[34]"},{"why":"Supplies the dueling architecture that splits state value from action advantage and is used by the best-performing model.","marker":"[35]"},{"why":"Supplies prioritized experience replay, which the best-performing model uses to sample informative episodes during training.","marker":"[36]"},{"why":"Supplies the clinical text processing tool used to detect anemia diagnosis terms in electronic health record notes, a load-bearing part of the real-world label construction.","marker":"[31]"}],"fun_headline_variants":["DRL picks test order to match batch models on anemia diagnosis","Reinforcement learning sequences anemia tests with interpretable steps","Anemia diagnosis via RL: fewer tests, same F1, clear reasoning","Deep RL guides differential anemia diagnosis with transparent pathways","AI agent orders anemia tests stepwise, matching state-of-the-art F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result depends on the electronic health record labels actually being confirmed anemia diagnoses; the authors' own evaluation of the matching algorithm used to create those labels found only 86 percent precision, with 42.9 percent of its errors being 'uncertain diagnoses' where the term was detected but did not mean a confirmed diagnosis.","fun_headline_variants_meta":{"raw":{"variants":["DRL picks test order to match batch models on anemia diagnosis","Reinforcement learning sequences anemia tests with interpretable steps","Anemia diagnosis via RL: fewer tests, same F1, clear reasoning","Deep RL guides differential anemia diagnosis with transparent pathways","AI agent orders anemia tests stepwise, matching state-of-the-art F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1352,"prompt_tokens":917,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":533,"tokens_out":435,"duration_ms":5528,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:38:22.739788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 200 of the 1,127 real-world patients, have two clinicians assign the final anemia sub-type from the full chart without seeing the model's output, and recompute the fine-tuned Dueling DQN-PER's F1 and ROC-AUC against that adjudicated gold standard; if the scores fall to chance level or clearly below the best batch classifier's, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"Deep Reinforcement Learning for Personalized Diagnostic Decision Pathways Using Electronic Health Records: A Comparative Study on Anemia and Systemic Lupus Erythematosus","cited_arxiv_id":"2404.05913","evidence_quote":"Provides the prior deep reinforcement learning study on anemia and systemic lupus erythematosus whose synthetic dataset methodology and Dueling DQN-PER choices the current paper extends to real-world data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Deep Q-Network algorithm that approximates the Q-value function over the large anemia state space."},{"cited_title":"Van Hasselt, A","cited_arxiv_id":null,"evidence_quote":"Supplies the Double DQN stabilization used in one of the two model families tested in the current paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dueling architecture that splits state value from action advantage and is used by the best-performing model."},{"cited_title":"Facilitating phenotyping from clinical texts: the medkit library","cited_arxiv_id":"2409.00164","evidence_quote":"Supplies the clinical text processing tool used to detect anemia diagnosis terms in electronic health record notes, a load-bearing part of the real-world label construction."}],"review_version":1}