{"id":"01facdb6-cec2-4609-83e9-d854efda51df","arxiv_id":"2606.21925","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Introduces a new Q-function definition for continuous-time RL and convergent off-policy algorithms under linear function approximation in model-based and model-free settings.","lead":"The paper defines a new state-action value function for continuous-time reinforcement learning with unknown SDE dynamics and develops off-policy algorithms using only discrete-time trajectory data via the PhiBE approximation. A smart generalist might read it to understand extensions of RL methods to physical systems where data arrives at discrete intervals but control is continuous.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest_assumption correctly isolates the unverifiable core without full text. No load-bearing technical flaw can be identified from the provided abstract alone, so the UNVERDICTED verdict stands.","tokens_in":1753,"tokens_out":224,"duration_ms":19438,"concrete_test":"Retrieve the full manuscript and examine the section defining the new Q-function together with its governing equation derivation; check whether the PhiBE approximation is invoked with any additional regularity conditions not present in the cited prior work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract outlines a new state-action value function definition for CTRL to enable off-policy learning from discrete data, followed by application of the PhiBE approximation and linear-FA convergence results. No full manuscript equations, derivation steps, or proof details are available to inspect for hidden assumptions, inconsistencies in the governing equation, or limitations in how PhiBE transfers to the new Q. The central claim therefore cannot be stress-tested for internal correctness at the level of specific technical steps.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims to introduce a new definition of the state-action value function for continuous-time reinforcement learning (CTRL) governed by unknown SDEs, where the classical Q(s,a) does not exist. It derives the governing equation for this new Q, then applies the PhiBE approximation to develop iterative PhiBE-Q-Learning algorithms for model-based and model-free off-policy approximation of the optimal Q from discrete-time trajectory data only. Under linear function approximation, convergence guarantees and explicit rates are established.","tokens_in":1865,"tokens_out":544,"duration_ms":22954,"significance":"If the new definition is internally consistent and the PhiBE transfer holds with the stated error bounds, the work would provide a concrete bridge between off-policy discrete-time RL methods and continuous-time control, enabling learning from sampled trajectories without requiring the instantaneous advantage or state-value function. The explicit convergence rates under linear FA and the model-free setting are notable strengths if the derivations are rigorous.","major_comments":[{"comment":"The central claim rests on the new state-action value function definition (introduced after the discussion of Baird 1994, Jia and Zhou 2023, and Tallec et al. 2019) and its governing equation. The manuscript must explicitly derive this equation from the SDE and demonstrate that the definition is independent of V(s) while still permitting off-policy updates; without this, the reduction to PhiBE (Zhu 2024; Zhu et al. 2025) risks inheriting the same non-existence issues rather than circumventing them.","section":"Abstract and the section introducing the new Q definition"},{"comment":"§ on convergence analysis: the stated convergence guarantees and rates under linear function approximation are load-bearing for the algorithmic contribution. The proof must be checked for hidden dependence on the specific form of the new Q and for whether the PhiBE approximation error is controlled uniformly across the discrete-time sampling; any unstated assumptions on the SDE coefficients or the function class would undermine the rates.","section":"Convergence analysis section"}],"minor_comments":[{"comment":"The abstract and introduction cite the non-existence results but should include a short self-contained paragraph contrasting the new Q with both the classical Q and the instantaneous advantage q(s,a) to improve readability for readers unfamiliar with the CTRL literature.","section":"Introduction"},{"comment":"Notation for the new Q-function should be introduced with an explicit equation number at first use and kept consistent throughout the model-based and model-free algorithm sections.","section":"Notation and algorithm sections"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a direct extension of the authors' prior PhiBE papers; the journal should assess whether the novelty of the new Q-definition is sufficient relative to the cited 2024/2025 works, especially given the absence of external benchmarks in the abstract."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive and detailed comments. We address each major comment point by point below, indicating where we will revise the manuscript to improve clarity and rigor.","responses":[{"response":"We agree that an explicit derivation from the SDE is necessary to fully substantiate the central claim. In the revised manuscript we will expand the relevant section with a complete, step-by-step derivation of the governing equation directly from the controlled SDE, beginning from the proposed definition of the new Q-function. We will also add a dedicated paragraph demonstrating independence from V(s) and showing how the definition supports off-policy updates. These additions will make clear that the subsequent reduction to PhiBE does not inherit the classical non-existence issues.","revision_made":"yes","referee_comment":"[Abstract and the section introducing the new Q definition] The central claim rests on the new state-action value function definition (introduced after the discussion of Baird 1994, Jia and Zhou 2023, and Tallec et al. 2019) and its governing equation. The manuscript must explicitly derive this equation from the SDE and demonstrate that the definition is independent of V(s) while still permitting off-policy updates; without this, the reduction to PhiBE (Zhu 2024; Zhu et al. 2025) risks inheriting the same non-existence issues rather than circumventing them."},{"response":"We acknowledge the importance of verifying that the convergence rates are free of hidden dependencies and that the approximation error is controlled uniformly. In the revised version we will augment the convergence analysis section and the appendix with additional intermediate steps that explicitly track the dependence on the new Q-form, establish uniform bounds on the PhiBE error with respect to the discrete sampling interval, and list all assumptions on the SDE coefficients and the linear function class. These clarifications will strengthen the presentation without altering the stated rates.","revision_made":"yes","referee_comment":"[Convergence analysis section] § on convergence analysis: the stated convergence guarantees and rates under linear function approximation are load-bearing for the algorithmic contribution. The proof must be checked for hidden dependence on the specific form of the new Q and for whether the PhiBE approximation error is controlled uniformly across the discrete-time sampling; any unstated assumptions on the SDE coefficients or the function class would undermine the rates."}],"tokens_in":1413,"tokens_out":505,"duration_ms":19087,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is a new definition of the state-action value function for continuous-time control that sidesteps the known non-existence of the classical Q(s,a). They derive a governing equation for it and then plug in their earlier PhiBE approximation to get iterative algorithms that work from discrete-time off-policy trajectories, both model-based and model-free. Under linear function approximation they claim convergence with explicit rates.\n\nWhat they do well is identify a practical gap—off-policy methods are useful but classical Q does not carry over—and try to fill it with a tailored definition rather than forcing the discrete-time template. The abstract is clear about building directly on Zhu 2024 and Zhu et al. 2025, so the lineage is transparent.\n\nThe soft spots are the heavy dependence on that prior approximation without new external validation or comparison to other continuous-time approaches, and the fact that the convergence claims sit behind linear FA without the actual derivations or counter-examples visible. The abstract does not show whether the new governing equation introduces hidden regularity conditions or whether the rates degrade under realistic noise. No data or figures are referenced, so it is hard to judge if the method scales beyond the linear case.\n\nThis is for readers already working on continuous-time RL or off-policy control who need a concrete way to handle discrete samples. It is not foundational enough to change the field, but the technical step is specific enough that a serious referee could check the derivations and the transfer of PhiBE. I would send it to review if the proofs hold up internally; otherwise it stays as an incremental note on their own line of work.","headline":"New Q-function definition lets them run off-policy updates on discrete data for continuous-time problems, but the work rests almost entirely on their own prior PhiBE approximation and the proofs are not visible here.","tokens_in":2364,"tokens_out":407,"would_cite":false,"duration_ms":9721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A new state-action value function in continuous-time control enables off-policy Q-learning from discrete data.","keywords":["continuous-time reinforcement learning","off-policy Q-learning","state-action value function","PhiBE approximation","stochastic differential equations","linear function approximation","convergence rates","model-free control"],"falsifier":"Numerical runs on a simple linear SDE where the proposed linear-function-approximation iteration fails to approach the true optimal Q as the number of discrete samples grows.","tokens_in":2638,"feed_emoji":"","tokens_out":619,"duration_ms":13608,"temperature":0.7,"pith_summary":"Classical Q(s,a) does not exist for continuous-time systems governed by stochastic differential equations, blocking standard off-policy methods. The paper defines a replacement state-action value function that does exist and derives the differential equation it must satisfy. This equation is then approximated by the PhiBE method to yield iterative update rules that recover the optimal Q using only discrete-time, off-policy trajectory samples. Both model-based and model-free versions of the iteration are given. Under linear function approximation the iterates converge at an explicit rate that depends on the discount factor and the approximation error.","feed_headline":"New value function enables off-policy learning in continuous-time control","feed_subtitle":"Redefined Q-function yields iterative updates and explicit convergence rates from discrete off-policy data alone.","key_machinery":"New state-action value function whose governing equation is discretized and solved via the PhiBE approximation to produce off-policy Q-updates from discrete samples.","core_discovery":"We introduce a new definition of the state-action value function in CTRL and derive its governing equation. Building on the PhiBE approximation, we propose iterative algorithms to approximate the optimal Q-function in both model-based and model-free settings using only discrete-time off-policy data. Under linear function approximation, we establish convergence guarantees and derive explicit convergence rates for the proposed method.","pith_inferences":["The same construction may allow off-policy updates in other continuous-time settings once an analogous governing equation can be written.","Hybrid discrete-continuous controllers could be trained by switching between the new Q and existing discrete Q-functions on the same trajectory data.","If the linear approximation error can be bounded for wider function classes, the method may extend beyond the linear case without changing the sampling requirement."],"forward_implications":["Model-based and model-free iterative schemes recover the optimal Q from off-policy discrete trajectories.","Explicit convergence rates hold whenever linear function approximation is used.","Off-policy learning becomes possible in CTRL even though the classical Q-function is undefined.","Only discrete-time samples are required; continuous-time derivatives need not be observed."],"fun_headline_variants":["Redefined Q-function bridges off-policy RL and continuous control","Iterative algorithms for optimal Q in model-free continuous-time RL","Explicit rates for PhiBE-Q-Learning convergence from off-policy data","New Q-definition enables discrete off-policy updates in CTRL"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The newly defined state-action value function exists, satisfies a usable governing equation, and can be approximated by PhiBE from discrete off-policy data.","fun_headline_variants_meta":{"raw":{"variants":["Redefined Q-function bridges off-policy RL and continuous control","Iterative algorithms for optimal Q in model-free continuous-time RL","Explicit rates for PhiBE-Q-Learning convergence from off-policy data","New Q-definition enables discrete off-policy updates in CTRL"]},"model":"grok-4.3","cost_usd":0.006284,"raw_usage":{"total_tokens":2943,"prompt_tokens":644,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":62837000,"prompt_tokens_details":{"text_tokens":644,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2233,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":644,"tokens_out":66,"duration_ms":11086,"temperature":1.0,"reasoning_tokens":2233,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T11:57:21.618123+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Numerical runs on a simple linear SDE where the proposed linear-function-approximation iteration fails to approach the true optimal Q as the number of discrete samples grows.","supporting_citations":[],"review_version":1}