{"id":"c79cc9b6-684e-4690-b33c-a7dcb3d4e4ce","arxiv_id":"2508.14093","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Physics-informed reward machines augment reward machines with ODE-driven continuous state, and experiments show that counterfactual experiences and reward shaping built on them speed up reinforcement learning.","lead":"This paper adds continuous physical models, such as coffee temperature dynamics, into reward machines that guide reinforcement learning agents, and reports faster reward acquisition in several control tasks. A generalist reader may care because it offers a way to inject known physics into an agent's reward structure without requiring a full model of the environment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline speedup is supported only by a cross-paper RM comparison; no same-code standard-RM baseline isolates the physics contribution.","rationale":"The reader's stated weakest assumption—perfect labeling and correct ODE—is a legitimate limitation, but it is explicitly acknowledged in the Conclusion and is standard in the RM literature; it does not by itself undermine the ideal-case claim. The most load-bearing uncertainty for the central empirical claim is instead whether the reported speedup is attributable to the physics component at all. Section 5's headline comparison 'under 5000 vs ~20000 steps' is a cross-paper comparison to [IKVM22]; within the paper, Figure 3 only compares pRME/pRME+RS against QL/QL+RS/DDPG/DDPG+RS, none of which use reward-machine structure. Since pRME performs updates for many pRM states per environment step (Algorithms 1 and 2), the environment-step speedup could be due to the increased number of updates or to RM-style counterfactuals, not to the ODE in Eq. (3.3). A same-code standard-RM baseline would settle this. This concern reinforces the reader's CONDITIONAL verdict: the framework is worth publishing with the condition that the physics-specific benefit be demonstrated in a controlled ablation. The Theorem 4.1 convergence proof is also not rigorous (constant kappa, coupling argument obscure), but that is a supporting-theory issue rather than the central empirical claim; it further supports conditionality.","tokens_in":19496,"tokens_out":10657,"duration_ms":113045,"concrete_test":"Implement a standard finite-state RM for the same Office World task in the authors' environment and code: replace the ODE flow in Eq. (3.3) with a discretized temperature counter (e.g., integer bins over psi_T, no continuous dynamics), keep the same task graph and rewards, and run Algorithm 1 with identical pRME sampling and hyperparameters. Measure environment steps to reach maximum reward for (i) RM-only, (ii) RM+pRME, and (iii) pRM+pRME. If RM+pRME reaches maximum reward in under 5000 steps or is statistically indistinguishable from pRM+pRME, the physics component is not responsible for the headline speedup; if pRM+pRME is substantially faster, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5's central comparison—pRM-enhanced RL reaches maximum rewards in under 5000 steps while the fastest RM method from [IKVM22] needs about 20000—is a cross-paper comparison, not a controlled experiment in the authors' code. Figure 3 compares pRME and pRME+RS only against QL, QL+RS, DDPG, and DDPG+RS; none of these baselines use reward-machine structure. Consequently, the observed speedup in environment steps cannot isolate the effect of the physics encoded in Eq. (3.3). The pRME procedure (Algorithm 1 lines 9–11; Algorithm 2 lines 11–13) performs updates for h=|Omega \\ Omega_F| counterfactual pRM states per environment step, whereas plain QL/DDPG perform one update per step. Faster reward acquisition in environment steps could therefore reflect more updates per step or RM-style counterfactual replay rather than the ODE. The only evidence specifically linking the speedup to physics is the citation to [IKVM22]'s 20000-step result, which is uncontrolled for environment version, reward scaling, hyperparameters, and implementation. Granting even perfect labels and a correct ODE, the central claim that physics improves learning efficiency is not yet established by the presented experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces physics-informed reward machines (pRMs), an extension of reward machines in which pRM states carry continuous variables that evolve according to ordinary differential equations. It defines a product construction between an unknown MDP and a pRM, extends counterfactual experience generation and potential-based reward shaping to this setting, and proposes Q-learning and DDPG variants (Algorithms 1 and 2). Experiments in an office gridworld, a two-tank system, a room-temperature model, and a traffic network are reported to support the central claim that pRMs significantly improve learning efficiency. The paper also includes an ablation-style comparison with a PINN-enhanced variant and with results reported for reward machines in prior work.","tokens_in":19802,"tokens_out":10886,"duration_ms":131337,"significance":"The idea of embedding known physics into the reward structure rather than into the policy or value network is clean and potentially useful, and the formal definition of pRMs with ODE-driven continuous components is a natural extension of hybrid-automaton ideas to reward specification. The product construction and the use of potential-based shaping are standard tools, and the paper gives credit to the RM literature it builds on. The main advertised value is the empirical speedup in reward acquisition. That claim is plausible but, as detailed below, it is not yet isolated from algorithmic confounds such as increased update counts and the absence of a controlled reward-machine baseline. The theoretical convergence result is also not proved under the assumptions stated in Algorithm 1.","major_comments":[{"comment":"The claimed asymptotic convergence of Algorithm 1 is not supported by the stated assumptions. In Algorithm 1, line 14, the update uses a fixed learning rate kappa in (0,1), with no decaying schedule, so the classical Q-learning convergence argument (which requires step sizes satisfying the Robbins-Monro conditions) does not apply. The proof in Appendix A introduces a coupling distribution and inequality (A.1), but it does not establish the stochastic-approximation conditions needed for convergence, does not address the asynchronous counterfactual updates in lines 9-18, and therefore does not fill the gap. The theorem should either be proved under an appropriate decaying learning-rate schedule, or replaced by a weaker statement that is actually true under a constant step size.","section":"Theorem 4.1; Algorithm 1; Appendix A"},{"comment":"The headline improvement in learning efficiency is confounded with the number of learning updates per environment step. In Algorithm 1, lines 9-18, each environment step can trigger h = |Omega \\ Omega_F| Q-learning updates, and for the DDPG variant Appendix F states that the mini-batch size and replay buffer are multiplied by h (128h and 50000h) when pRME is used. Faster reward acquisition measured per environment step is therefore expected even if the physics in Eq. (3.3) contributes nothing, simply because the pRME methods perform more updates per step. In addition, the only comparison involving a standard reward machine is the cross-paper reference to [IKVM22] in the Ablation Study, which is uncontrolled for implementation, hyperparameters, reward scale, and environment version. To support the central claim, the authors should include a same-code standard-RM baseline (or an ablation with the ODE removed) and compare methods at matched numbers of updates and matched computation.","section":"Section 5, Figure 3; Appendix F"},{"comment":"The DDPG target computation for counterfactual experiences appears inconsistent. For a replay sample (x, ϱhat, u, rhat, x', ϱhat'), the target Q should be evaluated at the counterfactual next state (x', ϱhat') with an action produced by the target actor at that same state. Line 17 instead uses ρ^μ(x_{k+1}, ϱtilde_{k+1}), i.e., the online actor at the actual next pRM state, rather than a target actor at the counterfactual next pRM state. This mismatches the action and state used in the target value. The pseudocode also never uses the target actor μ' in the update. Please correct the algorithm or clarify the intended target so that the continuous experiments are reproducible.","section":"Algorithm 2 (Appendix B), line 17"}],"minor_comments":[{"comment":"The text says a finite subset of the counterfactual experiences is selected by uniform random sampling, but Algorithms 1 and 2 simply say 'Update E according to (4.1)' and then iterate over E; please make the sampling procedure explicit in the pseudocode, including the number of sampled counterfactual states and how ties and randomness are handled.","section":"Section 4.1, Eq. (4.1) and Algorithm 1"},{"comment":"The sentence 'the potential function takes a negative value' is unclear, since value iteration on a pRM with nonnegative rewards produces nonnegative potentials; please clarify the sign convention and state exactly how the shaping reward is added in the pRME+RS variants, which are not shown in Algorithms 1 and 2.","section":"Section 4.2, around Eq. (4.4)"},{"comment":"The cross-reference 'Figure 1.1' appears to be a typo for Figure 1 or Figure 2, and the text should be checked for similar internal cross-reference errors.","section":"Example 1.1, page 3"},{"comment":"Algorithm 2 uses soft target updates with parameter ι, but Appendix F says the target network is updated every 300 steps; please reconcile the description with the pseudocode.","section":"Appendix B and Appendix F"},{"comment":"The assumption of a perfect labeling function is acknowledged in the Conclusion, but the experimental section makes the learning-efficiency claims without restating it; I recommend stating the oracle-label assumption alongside the experiments and, ideally, adding a sensitivity test under noisy labels or perturbed ODE parameters.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. The core object is a real extension: a reward machine whose states carry a continuous, ODE-driven variable, so known physics (cooling coffee, tank levels) live inside the reward structure. That combination is new relative to the reward-machine literature, and the hybrid-automata connection is honestly drawn. The second thing is that the strongest claim attached to it—that the physics is what accelerates learning—is not established by the experiments as presented, and the convergence theorem in Section 4 has a genuine proof gap.\n\nWhat the paper does well: Definition 3.1 is a clean formalism, the coffee-delivery example is a good illustration, the QL and DDPG variants are clearly specified, and the experiments span finite and continuous domains. The authors also state the perfect-labeling-function assumption in the conclusion; that is a real boundary of the framework, but it is standard in this literature and honestly acknowledged.\n\nSoft spots, in proportion. Theorem 4.1 is stated for Algorithm 1, which uses a constant learning rate κ ∈ (0,1)—the experiments run κ = 0.5. Classical Q-learning convergence requires a decaying step size satisfying the Robbins–Monro conditions. The appendix proof sketches a coupling argument and then says the rest follows [WD92]; that is exactly the step that fails with a constant rate. The result may hold with a decaying schedule, but it is not proven here.\n\nSecond, the motivating ODE in Example 3.2 appears wrong. For mode g0 the temperature derivative is set to T0 (98 °C per second), so the coffee heats up explosively while the agent waits. It should almost certainly be [1; 0]. It is one line in an example, not the framework itself, but it should be fixed.\n\nThird, and this is the load-bearing concern: the experiments do not isolate the physics. Figure 3 compares pRME and pRME+RS only against plain QL and DDPG; there is no same-code standard-RM baseline. The headline 'under 5000 steps versus 20000 steps' is a cross-paper comparison to [IKVM22], uncontrolled for environment version, reward scaling, and hyperparameters. pRME also performs h updates per environment step rather than one, so the speedup could be more updates per step or RM-style counterfactual replay rather than the ODE. The stress-test note is right: the physics-specific contribution is not isolated. I do not think the framework is wrong; the experiments just do not yet support the strongest version of the claim.\n\nMinor: the definition does not say what happens to ψ when the discrete mode switches, and no code or data is shipped, so the experiments cannot be reproduced from the text alone.\n\nWho gets value: people working on reward machines, structured reward RL, or hybrid reward formalisms. The definition and algorithms are worth engaging, and the paper deserves a serious referee. My recommendation: send it out, with reviewers asked to push on the convergence proof and on a controlled ablation with a standard RM baseline under identical code. With those, the paper would be solid.","headline":"A genuine extension of reward machines with ODE-driven continuous state, but the headline speedup claim rests on a cross-paper comparison and the convergence theorem is not established.","tokens_in":20300,"tokens_out":8404,"would_cite":true,"duration_ms":79517,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Physics-informed reward machines encode known dynamics into RL rewards, cutting training steps to under a quarter of the prior fastest method.","keywords":["physics-informed reward machines","reinforcement learning","non-Markovian rewards","counterfactual experiences","reward shaping","hybrid automata","sample efficiency","stochastic control"],"falsifier":"Run the office-grid experiment of Example 1.1 with the same pRM but flip each observed label with probability $p\\in\\{0.05,0.1,0.2\\}$, or perturb the heat-exchange coefficient $\\alpha$ in (3.4) by $\\pm 50\\%$, and record the number of training steps to reach maximum reward. If the sub-5,000-step convergence persists under these perturbations, the claim is robust; if it degrades sharply, the speedup depends on exact labels and an exact ODE.","tokens_in":19297,"feed_emoji":"🤖","tokens_out":6057,"duration_ms":55666,"temperature":0.7,"pith_summary":"Physics-informed reward machines (pRMs) extend reward machines—finite-state automata that specify non-Markovian rewards—by attaching an ordinary differential equation to each machine state. The claim is that when a designer knows the physical dynamics behind a task, such as coffee cooling, tank levels, room temperatures, or traffic flow, writing that dynamics into the reward structure lets a reinforcement-learning agent reach maximum rewards in far fewer training steps than standard reward machines. The paper argues this through a formal pRM definition, a convergence theorem for Q-learning with pRMs, a reward-shaping invariance result, and experiments on finite and continuous control environments. If correct, pRMs give RL a compact, interpretable interface between known physics and reward-driven exploration.","feed_headline":"Physics-informed reward machines hit max rewards 4x faster","feed_subtitle":"Encoding coffee cooling and tank-flow dynamics into reward machines cuts time-to-max-reward to under 5,000 steps.","key_machinery":"The load-bearing object is the pRM tuple $(\\Omega, \\Omega_F, \\Omega_0, \\delta_\\varrho, \\delta_r)$ from Definition 3.1: each machine state is a hybrid pair $(\\varrho_i, \\psi)$ where $\\varrho_i$ is a discrete location and $\\psi$ solves the ODE (3.3). The labeling function $L:X\\to 2^\\Delta$ maps environment transitions to propositional symbols, and the transition and reward functions $\\delta_\\varrho, \\delta_r$ use those symbols together with the ODE flow to decide machine moves and rewards. This construction is what lets known physics shape rewards even when the observable environment state does not change, and it is what makes the counterfactual-experience set (4.1) physically meaningful rather than purely fictitious.","core_discovery":"The central discovery is that the reward mechanism itself can carry a continuous physical model. A pRM augments each discrete automaton state with a vector $\\psi(k)$ evolving under an ODE $d\\psi(k)=g_i(\\psi(k))\\,dk$, so the reward $r_k=\\delta_r(\\tilde\\varrho(k), L(x_{k+1}))$ and the successor machine state depend on both the observed label and the integrated physical quantity. Because the ODE flow is part of the machine, the agent can generate counterfactual experiences for every non-terminal machine state using the same observed transition but different hypothetical machine states, and can apply potential-based reward shaping without changing the set of optimal policies. The paper reports that on an office-grid task the pRM-enhanced methods reach maximum rewards in under 5,000 training steps, whereas the fastest reward-machine method in prior work needs about 20,000.","pith_inferences":["If the speedup is robust, the same pRM construction could be used with any off-the-shelf deep reinforcement-learning algorithm whose replay buffer can accept extra transitions, not only Q-learning and DDPG.","The assumption of a perfect labeling function suggests a natural stress test: corrupting labels with even a small probability should degrade the sub-5,000-step speedup, and the rate of degradation would quantify how much of the gain depends on exact perception.","A testable extension the authors do not run is to replace the ODE with a learned or uncertain differential model and measure whether counterfactual experiences remain more useful than random ones; this would separate the value of physical structure from the value of a correct model.","Because pRMs shape rewards from continuous dynamics even while the observable state stays fixed, they may be particularly suited to partially observable tasks where hidden physical state matters, an application the paper only gestures at through its hybrid-automata inspiration."],"forward_implications":["pRM-enhanced Q-learning and DDPG converge to high rewards with substantially fewer training steps than plain reward machines, reducing sample complexity in structured control tasks.","Combining counterfactual pRM experiences with potential-based reward shaping outperforms either technique alone, and the shaping preserves the optimal-policy set by the paper's Theorem 4.3.","Multiple pRMs can be composed, with averaged rewards, to specify several tasks at once, although the averaging can dilute task-specific shaping signals.","Integrating physics-informed neural networks with pRMs yields further gains, suggesting that the two ways of injecting physics into learning are complementary.","Learned policies under pRMs respect physical constraints such as delivering warm coffee or avoiding prohibited regions, so fewer unsafe behaviors are observed during training."],"supporting_citations":[{"why":"Defines reward machines, the counterfactual-experience idea that pRMs extend, and the 20,000-step Office World baseline that pRMs are compared against.","marker":"[IKVM22]"},{"why":"Introduces learning reward machines for partially observable RL and is cited alongside [IKVM22] as the origin of the reward-machine formalism.","marker":"[TIWK+19]"},{"why":"Supplies the potential-based reward-shaping invariance theorem on which the paper's Theorem 4.3 relies.","marker":"[NHR99]"},{"why":"Establishes Q-learning convergence, which the proof of Theorem 4.1 reduces to after a coupling argument.","marker":"[WD92]"},{"why":"Provides the coupling inequality used in the proof of Theorem 4.1 to bound the difference between the true and counterfactual transition probabilities.","marker":"[DH12]"},{"why":"Introduces DDPG, the continuous-control algorithm that Algorithm 2 extends with pRM counterfactual experiences.","marker":"[LHP+15]"},{"why":"Supplies hybrid automata, the conceptual template for pRMs' discrete-plus-continuous state structure.","marker":"[Hen96]"}],"fun_headline_variants":["Reward machines that encode physics learn 4x faster","Physics in reward machines cuts max-reward steps to 5k","pRMs: physics-aware reward machines reach max reward in 5k steps","Embedding ODEs in reward machines accelerates RL training","Physics-informed reward machines speed up learning 4x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the designer can supply a perfect labeling function and a correct ordinary differential equation for the reward-relevant physical variable; if either is wrong, every counterfactual experience and shaped reward is computed from faulty inputs and the reported speedup loses its foundation.","fun_headline_variants_meta":{"raw":{"variants":["Reward machines that encode physics learn 4x faster","Physics in reward machines cuts max-reward steps to 5k","pRMs: physics-aware reward machines reach max reward in 5k steps","Embedding ODEs in reward machines accelerates RL training","Physics-informed reward machines speed up learning 4x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000765,"raw_usage":{"total_tokens":3366,"prompt_tokens":890,"completion_tokens":2476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2389}},"tokens_in":506,"tokens_out":2476,"duration_ms":18912,"temperature":1.0,"reasoning_tokens":2389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:30:04.831957+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the office-grid experiment of Example 1.1 with the same pRM but flip each observed label with probability $p\\in\\{0.05,0.1,0.2\\}$, or perturb the heat-exchange coefficient $\\alpha$ in (3.4) by $\\pm 50\\%$, and record the number of training steps to reach maximum reward. If the sub-5,000-step convergence persists under these perturbations, the claim is robust; if it degrades sharply, the speedup depends on exact labels and an exact ODE.","supporting_citations":[],"review_version":1}