{"id":"9d5bc003-800d-4588-8982-8e8e1ad43905","arxiv_id":"1908.08796","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A structured review of reinforcement learning applications in healthcare, organized by disease area and clinical task, with an account of unresolved technical challenges.","lead":"This paper surveys how reinforcement learning has been applied to healthcare, covering treatment policies, automated diagnosis, and hospital operations. It maps the field and lists open challenges for anyone deciding where to invest research effort.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'first comprehensive survey' claim in Section IX is unsupported by any documented search protocol; an independent citation retrieval could falsify it.","rationale":"The reader's CONDITIONAL verdict is sound. The paper's central novel contribution is scope: organizing RL in healthcare into DTRs, diagnosis, and other domains, and identifying challenges. The 'first comprehensive survey' sentence in Section IX turns scope into a falsifiable historical claim. For that claim, a reproducible selection protocol is not a bureaucratic nicety; it is the only way to establish representativeness. The absence of a protocol is visible in the text: Sections III-VI list applications with no inclusion criteria, and Section VI's last domains (drug discovery, health management) get only a few references, raising the question of why those and not others. As a stress-test, I would not ask for new technical results; I would ask that the comprehensiveness claim be either operationalized (search strings, databases, dates) or removed. This concern does not attack the taxonomy or the challenge list, which are genuinely useful and largely accurate. It is a framing and evidence-support issue, hence conditional rather than reject.","tokens_in":48921,"tokens_out":6351,"duration_ms":66631,"concrete_test":"Run a PRISMA-style search in PubMed, Scopus, and DBLP for 1990-2019 using a validated boolean query such as (reinforcement learning OR 'fitted Q iteration' OR Q-learning) AND (healthcare OR clinical OR medical OR treatment OR diagnosis), with explicit inclusion/exclusion criteria, and compare the deduplicated record set against the references in Tables III-VI. Also search for earlier surveys with comparable scope. If the survey's coverage recall is below roughly 80% within its stated domains, or if any pre-2019 survey with overlapping scope exists, the 'first comprehensive' claim is falsified and the conclusion should be reframed as 'a survey' rather than 'the first comprehensive survey.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's load-bearing claim is the assertion in Section IX that it 'serves as the first comprehensive survey of RL applications in healthcare.' For a survey, comprehensiveness is a factual property of the citation universe, and 'first' is a historical claim, yet the paper nowhere specifies a search protocol (databases, query terms, inclusion/exclusion criteria, date range), so neither claim can be checked by a reader. The enumeration in Sections III-VI is organized by a sensible taxonomy, but the absence of a protocol leaves open the possibility that the cited literature is a selection biased toward work visible to the authors, omitting prior surveys or application areas. This is not cosmetic: the paper explicitly uses 'comprehensive' as the basis of its contribution claim. Section VII's challenge list is useful independent of this, but the 'first comprehensive' framing should be revised to 'a survey' or supported by a reproducible search. The paper contains no limitation statement acknowledging this, which makes the overclaim the weakest point in an otherwise useful reference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a survey of reinforcement learning (RL) applications in healthcare. It begins with a review of RL foundations (MDPs, dynamic programming, Monte Carlo and temporal-difference methods, Q-learning, SARSA, policy search, actor-critic) and of key techniques (batch RL, model-based RL, transfer RL, representation learning for values/rewards/tasks, inverse RL, POMDPs). It then organizes the application literature into three broad areas: dynamic treatment regimes for chronic diseases and critical care, automated medical diagnosis from structured and unstructured data, and other healthcare domains such as resource allocation, process control, drug discovery, and health management. The final sections discuss challenges (state/action engineering, reward formulation, off-policy evaluation, model learning, exploration, credit assignment) and future directions (interpretability, prior knowledge, small data, ambient intelligence, and in-vivo validation). The conclusion states that the paper 'serves as the first comprehensive survey of RL applications in healthcare.'","tokens_in":49036,"tokens_out":4259,"duration_ms":48815,"significance":"If its coverage is accepted, the survey is a useful entry point for researchers new to RL in healthcare. Its strengths are the breadth of cited work, the clear application taxonomy, and the compact summary tables that map references to base methods, efficiency/representation techniques, data sources, and stated limitations. The treatment of off-policy evaluation, reward formulation, and the need for in-vivo validation is well aligned with the current state of the field. However, the paper's central contribution claim rests on 'first comprehensive survey,' and the manuscript provides no search protocol, inclusion/exclusion criteria, or comparison with prior surveys, so neither the priority nor the comprehensiveness can be independently verified. The paper also contains no limitation statement acknowledging this gap. These issues are load-bearing because comprehensive coverage is the stated basis of the survey's contribution, not merely a stylistic flourish.","major_comments":[{"comment":"The claim that the paper 'serves as the first comprehensive survey of RL applications in healthcare' is unsupported as written. The manuscript does not document a search protocol (databases, query terms, date range), inclusion/exclusion criteria, or any comparison with prior surveys, so a reader cannot check either 'first' or 'comprehensive.' Because this claim is the stated contribution of the paper, this is a load-bearing issue rather than a cosmetic one. I recommend either adding a methodology paragraph that makes the citation selection reproducible, or softening the claim to 'a survey' and explicitly stating the limitations of the coverage.","section":"Section IX, with implications for Sections I and III-VI"},{"comment":"The paper nowhere acknowledges the limitations of its own survey methodology. In particular, it does not state that the application map may be incomplete, that the selected references may be biased toward work visible to the authors, or that the catalog of challenges in Section VII is a synthesis of a non-systematically selected subset of the literature. Given that the paper explicitly builds its contribution on comprehensiveness, the absence of such a limitation statement should be corrected in the revision.","section":"Section IX (conclusion) and the absence of a survey-limitations paragraph"}],"minor_comments":[{"comment":"The epsilon-greedy policy in Eq. (7) is not a valid probability distribution: it assigns probability 1-epsilon to the greedy action and probability epsilon to every non-greedy action, which sums to more than 1 when the action set has more than two actions. The standard formulation is 1-epsilon+epsilon/|A| for the greedy action and epsilon/|A| for each non-greedy action.","section":"Section II-A, Eq. (7)"},{"comment":"The acronym ODE is defined as 'Ordinary Difference Equations' in the text, but the cited models are systems of ordinary differential equations; the definition should read 'Ordinary Differential Equations.'","section":"Section IV-A1 and Table III"},{"comment":"The text refers to 'support vector regression (SVG)' while Table III and the reference [104] describe support vector regression (SVR); the acronym should be corrected consistently to SVR.","section":"Section IV-A1, cancer chemotherapy paragraph"},{"comment":"The SARSA update in Eq. (6) is written as Q_t(s', pi(s')), but SARSA updates use the actually sampled next action a' drawn from the behavior policy; as written the notation conflates the behavior policy with the target policy and could confuse readers.","section":"Section II-A, Eq. (6)"},{"comment":"The text uses '3C (Compartimentalization, Corruption, and Complexity)'; the intended term appears to be 'Compartmentalization,' and the spelling should be corrected.","section":"Section IV-B and Table IV"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent broad survey, but the 'first comprehensive survey' claim is the main obstacle. If the journal is willing to accept a narrative survey rather than a systematic review, the manuscript can be made publishable by removing or rigorously documenting the comprehensiveness claim and by adding a survey-limitations paragraph. I do not see a need for new experiments or a full systematic-review protocol, but the current framing overstates the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is a genuinely broad survey. It organizes the literature into dynamic treatment regimes, automated diagnosis, and other domains, with tables that give a quick read on each study's base method, data source, and stated limits. The challenge list — state/action engineering, reward formulation, off-policy evaluation, model learning, exploration, credit assignment — is sensible and well anchored in examples like sepsis and HIV. For a newcomer or a funder wanting to map the area, this is a decent orientation. The summary of the cited work looks fair at the level I spot-checked, and the authors cite their own work without using it to support the survey's conclusions. That's fine.\n\nThe soft spot is the claim in Section IX that this 'serves as the first comprehensive survey of RL applications in healthcare.' That sentence is load-bearing for the paper's contribution, and it is unsupported. There is no documented search protocol (databases, queries, inclusion criteria, date range), no comparison with earlier surveys of RL in medicine or of DTRs, and no limitation section acknowledging the selection is non-exhaustive. 'Comprehensive' is a factual claim about the citation universe; 'first' is a historical claim. Neither can be checked from the paper. That said, the content is useful even if the framing is overstated. The absence of a protocol is a real flaw for a survey, but it is fixable, and the body of the paper does not become misleading once you soften the framing.\n\nA couple of minor issues: the 'other domains' sections are thin (drug discovery and health management get a few paragraphs each), and some table entries are marked 'Unclear' with no follow-up. These are minor against the survey's scope.\n\nMy take: this deserves a serious referee rather than a desk reject. The fix is straightforward — revise the first-comprehensive claim, add a short methodology paragraph on how papers were found and selected, and soften or defend the 'first' assertion. If the journal is willing to request that revision, the paper is a solid reference contribution.","headline":"Broad, well-organized RL-in-healthcare survey whose usefulness does not depend on the unsupported 'first comprehensive survey' claim.","tokens_in":49574,"tokens_out":2216,"would_cite":true,"duration_ms":21477,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning works across healthcare, but its clinical value hinges on how states, rewards, and policies are formulated.","keywords":["reinforcement learning","healthcare","dynamic treatment regimes","critical care","medical diagnosis","survey"],"falsifier":"A reader could falsify the survey's breadth claim by identifying a substantial published corpus of RL applications in healthcare that this survey omits, or by conducting a systematic literature search that yields a different distribution of application domains and open problems.","tokens_in":48674,"feed_emoji":"🏥","tokens_out":2394,"duration_ms":20717,"temperature":0.7,"pith_summary":"This survey paper tries to establish that reinforcement learning (RL) is a broadly applicable and increasingly successful approach for sequential decision-making in healthcare, and that the field's central obstacles are common across applications rather than domain-specific. The authors claim that RL's ability to learn from evaluative, delayed feedback makes it suited to dynamic treatment regimes, automated diagnosis, and resource-scheduling problems. They argue that the main open issues—state and action engineering, reward formulation, policy evaluation, model learning, exploration, and credit assignment—are what currently limit clinical impact, and they call for future work on interpretability, prior knowledge, small data, ambient intelligence, and in-vivo validation.","feed_headline":"RL in healthcare: promise meets six hard problems","feed_subtitle":"A survey maps where reinforcement learning works in medicine—and which bottlenecks still block clinical use.","key_machinery":"The central framework is the Markov decision process (MDP), defined as a 5-tuple $(S, A, P, R, \\gamma)$, where an agent chooses actions to maximize discounted cumulative reward. The survey organizes RL techniques along two complementary directions: efficient techniques (experience replay and batch RL, model-based RL, transfer RL) and representational techniques (function approximation, deep RL, multi-objective RL, preference-based RL, inverse RL, factored MDPs, hierarchical RL, POMDPs).","core_discovery":"The paper's central claim is that RL has been successfully applied across a broad range of healthcare domains, including dynamic treatment regimes for chronic diseases (cancer, diabetes, anemia, HIV, mental illness) and critical care (sepsis, anesthesia, mechanical ventilation), automated medical diagnosis from both structured and unstructured clinical data, and other domains such as health resource allocation, drug discovery, and health management. It argues that the RL framework—an agent learning optimal policies through trial-and-error interaction with an environment—maps naturally onto medical treatment as a sequential decision process. The authors assert that this survey is the first comprehensive survey of RL applications in healthcare.","pith_inferences":["A testable extension would be a systematic benchmark that evaluates offline RL treatment policies against standardized clinical outcomes across multiple ICU databases, going beyond the selective studies the survey reports.","The survey's emphasis on reward engineering suggests that progress may depend more on clinical expertise in specifying objectives than on advances in RL algorithms themselves.","The paper's catalog of applications implies that interoperable standards for state representation and reward specification could accelerate cross-institution validation of RL-based clinical decision support.","A consequence of the credit assignment discussion is that RL may be most easily validated in domains with short, well-defined decision horizons, such as medication dosing, before moving to longer-horizon interventions."],"forward_implications":["If RL-based treatment policies prove reliable in real clinical settings, they could enable personalized, adaptive treatment regimens that account for individual patient heterogeneity.","Successful integration of RL could reduce reliance on population-averaged treatment protocols by learning from accumulated patient data.","Addressing reward formulation challenges could lead to treatment policies that balance competing objectives such as efficacy and toxicity more explicitly.","Advances in off-policy evaluation and safe exploration would be needed before RL policies could be confidently deployed in clinical practice.","Small-data and transfer learning techniques could extend RL to rare diseases or new patient cohorts where historical data is scarce."],"supporting_citations":[{"why":"Documents recent reinforcement learning achievements in games and other domains, motivating healthcare applicability.","marker":"[10]"},{"why":"Provides the standard framework and notation for MDPs, value functions, and exploration-exploitation trade-offs.","marker":"[14]"},{"why":"Highlights the promise and challenges of applying RL to healthcare and motivates the need for a survey.","marker":"[16]"},{"why":"Covers batch RL, the key technique for learning from retrospective clinical data.","marker":"[39]"},{"why":"Describes Fitted Q Iteration with tree methods, central to many clinical RL applications discussed.","marker":"[41]"},{"why":"Introduces SMART designs that generate data for constructing dynamic treatment regimes.","marker":"[78]"},{"why":"Describes the MIMIC-III ICU database used in numerous sepsis and critical care RL studies.","marker":"[227]"},{"why":"Demonstrates RL for sepsis treatment in the ICU, a canonical application domain.","marker":"[192]"}],"fun_headline_variants":["From sepsis to drug discovery: RL in healthcare","Where RL fits in healthcare: a survey of success and gaps","RL in healthcare: the good, the bad, and the open problems","RL for medicine: a survey of what works and what doesn't","Reinforcement learning maps healthcare's sequential decisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the survey is comprehensive and that its selected literature represents the field relies on an undocumented and potentially non-systematic search and inclusion process, rather than a transparent protocol.","fun_headline_variants_meta":{"raw":{"variants":["From sepsis to drug discovery: RL in healthcare","Where RL fits in healthcare: a survey of success and gaps","RL in healthcare: the good, the bad, and the open problems","RL for medicine: a survey of what works and what doesn't","Reinforcement learning maps healthcare's sequential decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000962,"raw_usage":{"total_tokens":4063,"prompt_tokens":876,"completion_tokens":3187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":3104}},"tokens_in":492,"tokens_out":3187,"duration_ms":23954,"temperature":1.0,"reasoning_tokens":3104,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:44:04.092046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could falsify the survey's breadth claim by identifying a substantial published corpus of RL applications in healthcare that this survey omits, or by conducting a systematic literature search that yields a different distribution of application domains and open problems.","supporting_citations":[],"review_version":1}