REVIEW 3 major objections 5 minor 16 references
Physics-Informed Reward Machines
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Physics-informed reward machines encode known dynamics into RL rewards, cutting training steps to under a quarter of the prior fastest method.
desk verdict A genuine extension of reward machines with ODE-driven continuous state, but the headline speedup claim rests on a cross-paper comparison and the convergence theorem is not established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pRM tuple $(\Omega, \Omega_F, \Omega_0, \delta_\varrho, \delta_r)$ from Definition 3.1: each machine state is a hybrid pair $(\varrho_i, \psi)$ where $\varrho_i$ is a discrete location and $\psi$ solves the ODE (3.3). The labeling function $L:X\to 2^\Delta$ maps environment transitions to propositional symbols, and the transition and reward functions $\delta_\varrho, \delta_r$ use those symbols together with the ODE flow to decide machine moves and rewards. This construction is what lets known physics shape rewards even when the observable environment state does not change, and it is what makes the counterfactual-experience set (4.1) physically meaningful rather than purely fictitious.
What would settle it
Run the office-grid experiment of Example 1.1 with the same pRM but flip each observed label with probability $p\in\{0.05,0.1,0.2\}$, or perturb the heat-exchange coefficient $\alpha$ in (3.4) by $\pm 50\%$, and record the number of training steps to reach maximum reward. If the sub-5,000-step convergence persists under these perturbations, the claim is robust; if it degrades sharply, the speedup depends on exact labels and an exact ODE.
Extended reading notes
Core claim
The central discovery is that the reward mechanism itself can carry a continuous physical model. A pRM augments each discrete automaton state with a vector $\psi(k)$ evolving under an ODE $d\psi(k)=g_i(\psi(k))\,dk$, so the reward $r_k=\delta_r(\tilde\varrho(k), L(x_{k+1}))$ and the successor machine state depend on both the observed label and the integrated physical quantity. Because the ODE flow is part of the machine, the agent can generate counterfactual experiences for every non-terminal machine state using the same observed transition but different hypothetical machine states, and can apply potential-based reward shaping without changing the set of optimal policies. The paper reports that on an office-grid task the pRM-enhanced methods reach maximum rewards in under 5,000 training steps, whereas the fastest reward-machine method in prior work needs about 20,000.
Load-bearing premise
The load-bearing premise is that the designer can supply a perfect labeling function and a correct ordinary differential equation for the reward-relevant physical variable; if either is wrong, every counterfactual experience and shaped reward is computed from faulty inputs and the reported speedup loses its foundation.
Editorial extensions
If this is right
- pRM-enhanced Q-learning and DDPG converge to high rewards with substantially fewer training steps than plain reward machines, reducing sample complexity in structured control tasks.
- Combining counterfactual pRM experiences with potential-based reward shaping outperforms either technique alone, and the shaping preserves the optimal-policy set by the paper's Theorem 4.3.
- Multiple pRMs can be composed, with averaged rewards, to specify several tasks at once, although the averaging can dilute task-specific shaping signals.
- Integrating physics-informed neural networks with pRMs yields further gains, suggesting that the two ways of injecting physics into learning are complementary.
- Learned policies under pRMs respect physical constraints such as delivering warm coffee or avoiding prohibited regions, so fewer unsafe behaviors are observed during training.
Reading between the lines
- If the speedup is robust, the same pRM construction could be used with any off-the-shelf deep reinforcement-learning algorithm whose replay buffer can accept extra transitions, not only Q-learning and DDPG.
- The assumption of a perfect labeling function suggests a natural stress test: corrupting labels with even a small probability should degrade the sub-5,000-step speedup, and the rate of degradation would quantify how much of the gain depends on exact perception.
- A testable extension the authors do not run is to replace the ODE with a learned or uncertain differential model and measure whether counterfactual experiences remain more useful than random ones; this would separate the value of physical structure from the value of a correct model.
- Because pRMs shape rewards from continuous dynamics even while the observable state stays fixed, they may be particularly suited to partially observable tasks where hidden physical state matters, an application the paper only gestures at through its hybrid-automata inspiration.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces physics-informed reward machines (pRMs), an extension of reward machines in which pRM states carry continuous variables that evolve according to ordinary differential equations. It defines a product construction between an unknown MDP and a pRM, extends counterfactual experience generation and potential-based reward shaping to this setting, and proposes Q-learning and DDPG variants (Algorithms 1 and 2). Experiments in an office gridworld, a two-tank system, a room-temperature model, and a traffic network are reported to support the central claim that pRMs significantly improve learning efficiency. The paper also includes an ablation-style comparison with a PINN-enhanced variant and with results reported for reward machines in prior work.
Significance. The idea of embedding known physics into the reward structure rather than into the policy or value network is clean and potentially useful, and the formal definition of pRMs with ODE-driven continuous components is a natural extension of hybrid-automaton ideas to reward specification. The product construction and the use of potential-based shaping are standard tools, and the paper gives credit to the RM literature it builds on. The main advertised value is the empirical speedup in reward acquisition. That claim is plausible but, as detailed below, it is not yet isolated from algorithmic confounds such as increased update counts and the absence of a controlled reward-machine baseline. The theoretical convergence result is also not proved under the assumptions stated in Algorithm 1.
major comments (3)
- [Theorem 4.1; Algorithm 1; Appendix A] The claimed asymptotic convergence of Algorithm 1 is not supported by the stated assumptions. In Algorithm 1, line 14, the update uses a fixed learning rate kappa in (0,1), with no decaying schedule, so the classical Q-learning convergence argument (which requires step sizes satisfying the Robbins-Monro conditions) does not apply. The proof in Appendix A introduces a coupling distribution and inequality (A.1), but it does not establish the stochastic-approximation conditions needed for convergence, does not address the asynchronous counterfactual updates in lines 9-18, and therefore does not fill the gap. The theorem should either be proved under an appropriate decaying learning-rate schedule, or replaced by a weaker statement that is actually true under a constant step size.
- [Section 5, Figure 3; Appendix F] The headline improvement in learning efficiency is confounded with the number of learning updates per environment step. In Algorithm 1, lines 9-18, each environment step can trigger h = |Omega \ Omega_F| Q-learning updates, and for the DDPG variant Appendix F states that the mini-batch size and replay buffer are multiplied by h (128h and 50000h) when pRME is used. Faster reward acquisition measured per environment step is therefore expected even if the physics in Eq. (3.3) contributes nothing, simply because the pRME methods perform more updates per step. In addition, the only comparison involving a standard reward machine is the cross-paper reference to [IKVM22] in the Ablation Study, which is uncontrolled for implementation, hyperparameters, reward scale, and environment version. To support the central claim, the authors should include a same-code standard-RM baseline (or an ablation with the ODE removed) and compare methods at matched numbers of updates and matched computation.
- [Algorithm 2 (Appendix B), line 17] The DDPG target computation for counterfactual experiences appears inconsistent. For a replay sample (x, ϱhat, u, rhat, x', ϱhat'), the target Q should be evaluated at the counterfactual next state (x', ϱhat') with an action produced by the target actor at that same state. Line 17 instead uses ρ^μ(x_{k+1}, ϱtilde_{k+1}), i.e., the online actor at the actual next pRM state, rather than a target actor at the counterfactual next pRM state. This mismatches the action and state used in the target value. The pseudocode also never uses the target actor μ' in the update. Please correct the algorithm or clarify the intended target so that the continuous experiments are reproducible.
minor comments (5)
- [Section 4.1, Eq. (4.1) and Algorithm 1] The text says a finite subset of the counterfactual experiences is selected by uniform random sampling, but Algorithms 1 and 2 simply say 'Update E according to (4.1)' and then iterate over E; please make the sampling procedure explicit in the pseudocode, including the number of sampled counterfactual states and how ties and randomness are handled.
- [Section 4.2, around Eq. (4.4)] The sentence 'the potential function takes a negative value' is unclear, since value iteration on a pRM with nonnegative rewards produces nonnegative potentials; please clarify the sign convention and state exactly how the shaping reward is added in the pRME+RS variants, which are not shown in Algorithms 1 and 2.
- [Example 1.1, page 3] The cross-reference 'Figure 1.1' appears to be a typo for Figure 1 or Figure 2, and the text should be checked for similar internal cross-reference errors.
- [Appendix B and Appendix F] Algorithm 2 uses soft target updates with parameter ι, but Appendix F says the target network is updated every 300 steps; please reconcile the description with the pseudocode.
- [Conclusion] The assumption of a perfect labeling function is acknowledged in the Conclusion, but the experimental section makes the learning-efficiency claims without restating it; I recommend stating the oracle-label assumption alongside the experiments and, ideally, adding a sensitivity test under noisy labels or perturbed ODE parameters.
Circularity Check
No circularity found: the pRM reward definitions, counterfactual experience generation, and potential-based shaping are structurally non-circular; the speedup claims are empirical comparisons with some experimental confounds, not derivations from their own inputs.
full rationale
The paper's formal derivation chain is self-contained and does not reduce to its inputs. Definition 3.1 specifies pRM rewards and transitions from a user-supplied labeling function and ODE flows in (3.3); Eq. (4.1) generates counterfactual experiences from those same pRM definitions, and these experiences are used only to create additional Q-value or replay-buffer updates, not to fit or define the measured outcome. Theorem 4.1 is a convergence claim for the resulting Q-learning variant and, whatever its proof-strength issues may be, it does not presuppose the empirical speedup. Theorem 4.3 applies the standard potential-based reward shaping invariance result of Ng et al. (NHR99), so deriving the potential function by value iteration on the pRM is not circular: shaping changes the reward but provably preserves the set of optimal policies. The central empirical claim is an experimental comparison in Section 5. The 5000-step versus 20000-step contrast against the RM method of [IKVM22] is a cross-paper comparison, and pRME performs h counterfactual updates per environment step versus one update for plain QL/DDPG, so the speedup evidence has confounds; however, those are experimental-validity concerns and alternative explanations, not examples of a prediction reducing to a fit or to a self-citation chain. No fitted parameter is renamed as a prediction, no uniqueness or optimality result from the authors' prior work is invoked to forbid alternatives, and the self-citations that appear (e.g., [AZ24b], [LKSZ20], [HPS+19]) are used as environment sources or background context rather than as load-bearing justification of pRM efficacy. The paper explicitly acknowledges the perfect-labeling assumption in the Conclusion; that is a stated applicability limitation, not a circular step. Overall, the derivation is not circular, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (7)
- heat exchange coefficient alpha =
3.3e-4 per second
- ambient temperature T_e =
20 degrees C
- initial coffee temperature T_0 =
98 degrees C
- delivery temperature threshold T_c =
not specified in the text
- time counter bounds N_a, N_b, N_c, N_d =
N_a=N_b=10 for AR1; N_a=N_b=N_c=N_d=30 for AR3
- reward values zeta_i =
zeta_0=zeta_2=0, zeta_1=1 (AR1/AR2); zeta_0=zeta_1=0, zeta_2=1 (AR3)
- sampling interval tau_A =
1 second
assumptions (6)
- domain assumption A perfect labeling function L:X to 2^Delta is available so the pRM receives exact propositional labels for each transition.
- domain assumption The reward-relevant physical quantity evolves according to the known ODE (3.3) with known initial conditions and parameters.
- ad hoc to paper Discretizing the continuous pRM components and uniformly sampling counterfactual experiences does not materially change the learning problem.
- standard math Standard Q-learning convergence results apply to Algorithm 1 when every state-action pair is visited infinitely often.
- standard math Potential-based reward shaping preserves optimal policies.
- domain assumption For the convergence theorem, every state-action pair (x, rhotilde, u) in the finite product MDP is visited infinitely often.
invented entities (1)
-
Physics-informed reward machine (pRM)
Cite this review
Pith. "Pith review of Physics-Informed Reward Machines." pith.science (2026). https://pith.science/paper/R3IMTZTD
@misc{pith2026250814093,
author = {Pith},
title = {Pith review of: Physics-Informed Reward Machines},
year = {2026},
howpublished = {\url{https://pith.science/paper/R3IMTZTD}},
note = {Machine review of arXiv:2508.14093}
}
read the original abstract
Reward machines (RMs) provide a structured way to specify non-Markovian rewards in reinforcement learning (RL), thereby improving both expressiveness and programmability. Viewed more broadly, they separate what is known about the environment, captured by the reward mechanism, from what remains unknown and must be discovered through sampling. This separation supports techniques such as counterfactual experience generation and reward shaping, which reduce sample complexity and speed up learning. We introduce physics-informed reward machines (pRMs), a symbolic machine designed to express complex learning objectives and reward structures for RL agents, thereby enabling more programmable, expressive, and efficient learning. We present RL algorithms capable of exploiting pRMs via counterfactual experiences and reward shaping. Our experimental results show that these techniques accelerate reward acquisition during the training phases of RL. We demonstrate the expressiveness and effectiveness of pRMs through experiments in both finite and continuous physical environments, illustrating that incorporating pRMs significantly improves learning efficiency across several control tasks.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[6]
Exploration-exploitation in constrained MDPs
[EMP20] Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained MDPs. arXiv preprint arXiv:2003.02189,
arXiv 2003
-
[7]
Probably approximately correct MDP learning and control with temporal logic constraints
[FT14] Jie Fu and Ufuk Topcu. Probably approximately correct MDP learning and control with temporal logic constraints. arXiv preprint arXiv:1404.7073 ,
-
[12]
Formal controller syn- thesis for continuous-space MDPs via model-free reinforcement learning
[LSS+20] Abolfazl Lavaei, Fabio Somenzi, Sadegh Soudjani, Ashutosh Trivedi, and Majid Zamani. Formal controller syn- thesis for continuous-space MDPs via model-free reinforcement learning. In 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS) , pages 98–107. IEEE,
work page 2020
-
[14]
[ML23] Amartya Mukherjee and Jun Liu. Bridging physics-informed neural networks with reinforcement learning: Hamilton- Jacobi-Bellman Proximal Policy Optimization (HJBPPO). arXiv preprint arXiv:2302.00237 ,
-
[16]
[XWO+21] Zhe Xu, Bo Wu, Aditya Ojha, Daniel Neider, and Ufuk Topcu. Active finite reward automaton inference and reinforcement learning using queries and counterexamples. In Machine Learning and Knowledge Extraction: 5th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2021, Virtual Event, August 17–20, 2021, Procee...
work page 2021
-
[1996]
Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees
[HKA+19] Mohammadhosein Hasanbeig, Yiannis Kantaros, Alessandro Abate, Daniel Kroening, George J Pappas, and Insup Lee. Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees. In 2019 IEEE 58th conference on decision and control (CDC) , pages 5338–5343. IEEE,
2019
-
[2000]
A survey on physics informed reinforcement learning: Review and open problems
[BNFR23] Chayan Banerjee, Kien Nguyen, Clinton Fookes, and Maziar Raissi. A survey on physics informed reinforcement learning: Review and open problems. arXiv preprint arXiv:2309.01909 ,
-
[2006]
Efficient Reinforcement Learning in Probabilistic Reward Machines
14 DANIEL AJELEYE, ASHUTOSH TRIVEDI, AND MAJID ZAMANI [LZ24] Xiaofeng Lin and Xuezhou Zhang. Efficient reinforcement learning in probabilistic reward machines. arXiv preprint arXiv:2408.10381,
Show all 16 references
-
[2013]
Continuous control with deep reinforcement learning
[LHP+15] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 ,
-
[2014]
PhysQ: a physics informed reinforcement learning framework for building control
[GCD22] Gargya Gokhale, Bert Claessens, and Chris Develder. PhysQ: a physics informed reinforcement learning framework for building control. arXiv preprint arXiv:2211.11830 ,
-
[2019]
Learning non-Markovian reward models in MDPs
[RR20] Gavin Rens and Jean-Fran¸ cois Raskin. Learning non-Markovian reward models in MDPs. arXiv preprint arXiv:2001.09293,
2001 arXiv
-
[2020]
Fast online exact solutions for deterministic MDPs with sparse rewards
[BYW18] Joshua R Bertram, Xuxi Yang, and Peng Wei. Fast online exact solutions for deterministic MDPs with sparse rewards. arXiv preprint arXiv:1805.02785 ,
-
[2021]
Physics informed intrinsic rewards in reinforcement learning
[JFC22] Jiazhou Jiang, Minyue Fu, and Zhiyong Chen. Physics informed intrinsic rewards in reinforcement learning. In 2022 Australian & New Zealand Control Conference (ANZCC) , pages 69–74. IEEE,
2022
-
[2022]
Data-driven construction of finite abstractions for interconnected systems: A compositional approach
[AZ24a] Daniel Ajeleye and Majid Zamani. Data-driven construction of finite abstractions for interconnected systems: A compositional approach. arXiv preprint arXiv:2408.08497 ,
-
[2023]
Control synthesis from linear temporal logic specifications using model-free reinforcement learning
[BWZP20] Alper Kamil Bozkurt, Yu Wang, Michael M Zavlanos, and Miroslav Pajic. Control synthesis from linear temporal logic specifications using model-free reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 10349–10355. IEEE,
2020
-
[2024]
Verification of Markov decision processes using learning algorithms
[BCC+14] Tom´ aˇ s Br´ azdil, Krishnendu Chatterjee, Martin Chmelik, Vojtˇ ech Forejt, Jan Kˇ ret´ ınsk` y, Marta Kwiatkowska, David Parker, and Mateusz Ujma. Verification of Markov decision processes using learning algorithms. In Automated Technology for Verification and Anal...
2014
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.