State Estimation and Control of Dynamic Systems from High-Dimensional Image Data

Ashik E Rasul; Hyung-Jin Yoon

REVIEW 5 major objections 5 minor 21 references

State Estimation and Control of Dynamic Systems from High-Dimensional Image Data

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A CNN-GRU-FCNN state estimator trained from image sequences enables a Deep Q-Network to control CartPole comparably to a policy with full access to the true state.

desk verdict A clean but incremental supervised state-estimation pipeline for CartPole whose 'comparable performance' claim is overstated and whose Markov-sufficiency assumption is unchecked. read the letter →

arxiv 2506.05375 v1 pith:EMSZBRWR submitted 2025-05-30 cs.CV

Ashik E Rasul , Hyung-Jin Yoon This is my paper

classification cs.CV

keywords stateestimationimage-basedcontroldeepreinforcementlearningQ-NetworkgatedrecurrentunitinterpretablestatesCartPoleperception-based

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a deep reinforcement learning agent can control a dynamic system from high-dimensional image observations alone, without ever seeing the true state, by first learning to estimate the state. The proposed estimator, a CNN-GRU-FCNN pipeline trained end-to-end to minimize mean squared error against ground truth, turns a four-frame image sequence plus action history into an interpretable four-dimensional state vector (cart position and velocity, pole angle and angular velocity). A Deep Q-Network trained on these estimated states achieves tracking errors comparable to a DQN trained on the true states on the CartPole benchmark, with pole-angle error actually lower. The paper argues that such learned, physically meaningful state representations can bridge perception and control, offering an interpretable alternative to black-box latent-space policies.

What carries the argument

The carrying mechanism is the perception-to-state estimator defined by Eqs. (1), (2), and (6). Four downsampled RGB frames are passed through a CNN to yield 128-dimensional per-frame features, each concatenated with the corresponding action and fed to a GRU; the GRU's final hidden state $h_4$ encodes the spatiotemporal history, and an FCNN regresses it to the predicted state $\hat{s}_{k+4}$. The whole stack is trained end-to-end by the MSE loss of Eq. (7), and the DQN then operates on this estimated state space via the Bellman optimality equation of Eq. (8), which is valid only if the estimated state is a sufficient statistic for the next transition.

What would settle it

Train an alternative DQN whose observation is the concatenation of the predicted state $\hat{s}_{k+4}$ and the GRU hidden state $h_4$; if this agent achieves substantially lower tracking error than the estimated-state-only agent, then $\hat{s}_{k+4}$ alone is not a sufficient statistic, breaking the assumption behind Eq. (8). A complementary test is to attempt next-frame prediction from $(\hat{s}_k, a_k)$ and compare against prediction from the full four-frame history.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that replacing ground-truth state access with a CNN-GRU-FCNN estimator does not cost much in control performance. The estimator produces state predictions with RMSE below 4% on all CartPole variables except cart position (0.24%), and the resulting DQN policy keeps the pole controlled with MAE of 1.19% on pole angle versus 1.60% for the full-state baseline, while cart-position error rises to 5.30%. The authors present this as evidence that interpretable, low-dimensional state estimates learned from images are sufficient for policy learning in perception-based control, and they provide a two-part evaluation methodology that separates state-estimation accuracy from policy tracking performance.

Load-bearing premise

The load-bearing premise is the Markovian sufficiency stated after Eq. (7): that if the estimated state is close to the true state, the next state depends only on the current estimated state and action; a biased estimator (such as the 5.30% cart-position error in Table 2) can violate this and the paper does not test whether image history still contains predictive information.

Editorial extensions

If this is right

If the claim holds, image-only control with interpretable physical state outputs becomes a practical option for benchmark tasks like CartPole, removing the requirement of ground-truth state feedback.
The separate evaluation of state-estimation RMSE and policy-tracking MAE provides a template for diagnosing whether a perception-based controller fails due to the estimator or the policy.
The framework's state estimates could be reused by any control algorithm that expects physical coordinates, not just DQN, extending the scope beyond the tested benchmark.
The low error on pole angle and angular velocity suggests the estimator captures latent rotational dynamics from image sequences, not just static position.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension not tested in the paper is whether the estimated state is truly a sufficient statistic: if a policy given the GRU hidden state $h_4$ in addition to $\hat{s}_{k+4}$ outperforms the estimated-state policy, then the estimator discards useful history that the Bellman equation (8) would need.
The strong visual distinctiveness of CartPole likely makes state estimation easier than in visually aliased or partially observable domains; the framework would need additional temporal modeling or a latent-dynamics model in such settings.
Because the estimated states are in physical coordinates, the same estimator could be paired with model-predictive control or a linear quadratic regulator, letting classical control theory act on image inputs without retraining the policy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper before reading it. First, it is not the first approach to interpretable state estimation for image-based RL; the authors' own cited literature on latent-state methods undercuts that claim. Second, the central parity claim, that a DQN trained on predicted states performs comparably to one trained on true states, is not actually established by the data they report. The cart position error in Table 2 is ten times larger for the predicted-state agent (5.30% vs. 0.53%). That is a real gap, not a rounding artifact.

What the paper does well: it is a clean, reproducible demonstration. The architecture is standard—CNN encoder, GRU over a 4-frame window with actions, FCNN regressor trained with MSE—and they report RMSE on a validation set. They also ship code, which is more than many papers do. The state estimator clearly learns something useful: the pole angle and angular velocity errors are below 4%, and the RL agent does keep the pole upright. The evaluation methodology, separating state prediction error from control tracking error, is sensible and could be useful as a template.

Now the soft spots, in proportion. The most load-bearing issue is the assumption before Eq. (8): that if the estimated state is close to the true state, the next state depends only on the current estimated state and action. This is not implied by small RMSE. The estimated state is a function of a 4-frame image window plus action history; it could omit task-relevant information. A biased estimate, like the cart velocity error of 3.80%, can leave a non-Markov residual, so the Bellman equation in Eq. (8) may be optimizing the wrong MDP. The paper never tests this, e.g., by checking whether the estimated state is sufficient given the images. This is not fatal—the empirical result on CartPole suggests the estimator captures enough—but the claim needs verification.

There are no error bars or multiple seeds, no comparison to raw-pixel DQN or latent-state baselines, and the only benchmark is CartPole. The paper even admits the evaluation timespan is too short to draw definitive conclusions. The reward shaping weights in Eq. (13) are hand-picked and could affect the comparison.

Overall, this is a legitimate but incremental empirical study. It is not a breakthrough, and the novelty claim should be softened. But the work is honest, the code is available, and the idea is sound. I would send it to peer review for a workshop or short paper, with the expectation that the authors need to add baselines, seeds, and a direct test of the Markov assumption.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes a perception-based control pipeline for CartPole: a CNN-GRU-FCNN architecture maps a four-frame RGB image sequence and action history to an estimate of the four-dimensional physical state, and the estimator is trained end-to-end by MSE against simulator ground truth. A DQN is then trained on these estimated states, and its tracking performance is compared with a DQN trained on true states. The paper reports low RMSE for state estimation (Table 1) and claims comparable RL performance, though the predicted-state agent has a tenfold larger cart-position tracking error (Table 2).

Significance. The framework addresses a relevant problem: replacing ground-truth state access with interpretable, low-dimensional state estimates from raw images. If the central claim were established, the approach would be a useful benchmark for perception-based control. The paper has strengths: the state predictor is trained against an external ground-truth benchmark rather than a fitted quantity; the evaluation methodology separates state-estimation accuracy from control performance; and a code repository is provided. However, the evidence presented does not yet establish the headline claim of comparable performance to full-state DQN, because the control comparison lacks statistical support and the Markov-sufficiency assumption underlying Eq. (8) is unchecked.

major comments (5)

[Reinforcement Learning with DQN, Eq. (8)] The sentence preceding Eq. (8), 'If the estimated state is close to the true state, then the next state depends only on the current estimated state and action,' is not a consequence of small RMSE. Low average prediction error does not guarantee that the estimated state is a sufficient statistic for the next state or reward. This assumption is load-bearing because Eq. (8) solves a Bellman equation on estimated states; if it fails, the DQN optimizes a different MDP than the true system. The paper should test the assumption, for example by comparing the one-step prediction error of a model using only the estimated state and action against a model that also conditions on the raw image window or the GRU hidden state, or by computing Bellman residuals under the learned Q-function.
[Evaluation Results, Table 2] The 'comparable performance' claim is not supported by the reported aggregate numbers alone. The predicted-state agent's cart position MAE is 5.30% versus 0.53% for the full-state agent, an order-of-magnitude gap, while no error bars, number of seeds, or episode counts are reported. Without multiple independent runs and standard deviations it is impossible to tell whether the other three variables' improvements are systematic or noise. Please report mean ± standard deviation over at least 5–10 seeds for both agents, and discuss whether the cart drift is acceptable relative to the control objective.
[Reinforcement Learning with State Prediction] The state estimator is trained on trajectories generated by random actions, but it is deployed inside the loop of a learned policy whose state distribution differs from the training distribution. The paper does not quantify state-estimation accuracy on policy-induced trajectories. This matters because the control comparison in Table 2 can be confounded by estimator degradation under distribution shift. Please report the RMSE of the estimator on replay data collected by the trained predicted-state policy, not only on the random-action validation set.
[Reinforcement Learning with Full State Observation, Eq. (13)] The paper does not specify whether the reward is computed from true environment states or from estimated states when training the predicted-state DQN. If the reward uses true states, then the Bellman equation in Eq. (8) is not written for the actual reward function; if it uses estimated states, the reward itself becomes a random variable driven by estimation error. This should be stated explicitly, and the Bellman recursion should be written with the actual reward argument.
[Experiments and Results] The evaluation does not include a raw-pixel DQN baseline or a latent-state baseline, so the reader cannot tell whether the interpretability of the estimated state comes at a significant performance cost relative to end-to-end image-based RL. At minimum, the paper should cite and compare against a standard image-based DQN result, or include a simple raw-frame CNN-DQN baseline, to contextualize the proposed method's performance and justify its added complexity.

minor comments (5)

[Abstract and Introduction] The text contains a spacing typo, 'UA Vs', which should read 'UAVs'.
[Proposed Framework, Eq. (7)] The loss is written for a single example; the paper should define the mean over the batch or minibatch.
[Reinforcement Learning with Full State Observation] The exploration-rate schedule ('decays by 10% over time') is underspecified; report the exact schedule, such as per episode or per timestep, and any floor value.
[Evaluation Results] The normalization bounds used for RMSE and MAE percentages are not stated; define them explicitly (for example, cart position in [-2.4, 2.4] and pole angle in [-0.21, 0.21]) so the percentages are interpretable.
[Experiments and Results, Figure 6] The text states that the simulation timespan is 'too short to draw definitive conclusions'; please either lengthen the evaluation or remove this caveat, since it weakens the subsequent claims made from the same prediction model.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: state estimates are learned from external simulator ground truth and evaluated independently.

full rationale

The claimed derivation chain is not circular. The state estimator (Eqs. 1-7) is trained end-to-end by MSE against the CartPole simulator's ground-truth state vector, which is an external benchmark not derived from the paper's outputs. The DQN policy is then trained on those estimates using the standard Bellman equation (Eq. 8), and its tracking performance is evaluated with an independent MAE metric (Eq. 10) against the same simulator. The Markov-sufficiency sentence before Eq. (8) is an explicit assumption, not a consequence obtained by substituting the paper's equations, so any failure of that assumption is a correctness/robustness limitation rather than a circular step. Reward-shaping weights in Eq. (13) are hand-selected design choices, not fitted parameters renamed as predictions. The paper's own caveat that the single-episode evaluation in Figure 6 is 'too short to draw definitive conclusions' is an honest limitation, and the subsequent RL comparison addresses it. No load-bearing self-citations appear. Thus no step reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central result rests on the Markovity of the predicted state (which is not verified) and on manually chosen reward weights; no new physical entities are introduced.

free parameters (4)

reward shaping weights lambda1, lambda2, lambda3 = 0.1, 1.0, 0.35
Hand-chosen in Eq. (13); they shape the DQN policy and directly affect the tracking-error comparison in Table 2.
input sequence length = 4
The model uses four image frames; this temporal window is chosen by hand and affects velocity estimation.
CNN and GRU hidden dimensions = not specified
The feature vector dimension is 128, but convolutional channel counts and GRU hidden size are not reported; these are free architectural choices.
state-space discretization bins for data collection = not specified
Data collection discretizes state dimensions into bins (Figure 3) to balance samples, but no bin counts are given.

assumptions (3)

domain assumption Estimated state is a sufficient statistic: if the estimated state is close to the true state, the next state depends only on the current estimated state and action.
Invoked after Eq. (7) to justify Bellman Eq. (8); central to DQN validity.
domain assumption Simulator ground-truth states are correct labels for supervised training.
CartPole environment provides x, x-dot, theta, and theta-dot; Eq. (7) trains to these labels.
standard math Standard neural-network approximation and RL convergence assumptions, including universal approximation and DQN stability with replay and target network.
Background from Hornik, Mnih, and Chung; not proved in this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of State Estimation and Control of Dynamic Systems from High-Dimensional Image Data." pith.science (2026). https://pith.science/paper/EMSZBRWR

@misc{pith2026250605375,
  author       = {Pith},
  title        = {Pith review of: State Estimation and Control of Dynamic Systems from High-Dimensional Image Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMSZBRWR}},
  note         = {Machine review of arXiv:2506.05375}
}

read the original abstract

Accurate state estimation is critical for optimal policy design in dynamic systems. However, obtaining true system states is often impractical or infeasible, complicating the policy learning process. This paper introduces a novel neural architecture that integrates spatial feature extraction using convolutional neural networks (CNNs) and temporal modeling through gated recurrent units (GRUs), enabling effective state representation from sequences of images and corresponding actions. These learned state representations are used to train a reinforcement learning agent with a Deep Q-Network (DQN). Experimental results demonstrate that our proposed approach enables real-time, accurate estimation and control without direct access to ground-truth states. Additionally, we provide a quantitative evaluation methodology for assessing the accuracy of the learned states, highlighting their impact on policy performance and control stability.

Figures

Figures reproduced from arXiv: 2506.05375 by the authors.

**Figure 3.** Dataset distribution across the state space [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Gating mechanism of the GRU Temporal Modeling with GRU We use a Gated Recurrent Unit (GRU) to capture temporal dependencies across the sequence of image frames and corresponding control actions. The GRU receives a sequence of 4 inputs {pk} 4 k=1, where each pk = [fk; ak] is a concatenation of the convolutionally encoded image feature fk ∈ R 128 and the scalar action ak ∈ R at timestep k. Following the formulation … view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: Training and validation loss of the prediction [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Evaluation of the state prediction model on a single [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

**Figure 8.** Figure 8: DQN performance with true state observation [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]

**Figure 9.** Figure 9: DQN performance with predicted state Reinforcement Learning with State Prediction We then train the second DQN agent using the same hyperparameters, but with access only to the predicted state from the learned model. During evaluation, the agent is given access solel…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

K.; and Vikrama, R

Balaji, E.; Brindha, D.; Elumalai, V. K.; and Vikrama, R. 2021. Automatic and non-invasive Parkinson’s disease diagnosis and severity rating using LSTM network. Applied Soft Computing, 108: 107463

work page 2021
[4]

Bengio, Y.; Simard, P.; and Frasconi, P. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2): 157--166

work page 1994
[5]

Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. Openai gym. arXiv preprint arXiv:1606.01540

arXiv 2016
[6]

J.; Preindl, M.; and Emadi, A

Chemali, E.; Kollmeyer, P. J.; Preindl, M.; and Emadi, A. 2018. State-of-charge estimation of Li-ion batteries using deep neural networks: A machine learning approach. Journal of Power Sources, 400: 242--255

work page 2018
[7]

Cho, K.; Van Merri \"e nboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

arXiv 2014
[8]

Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555

arXiv 2014

Show all 21 references

[9]

Grigorescu, S.; Trasnea, B.; Cocias, T.; and Macesanu, G. 2020. A survey of deep learning techniques for autonomous driving. Journal of field robotics, 37(3): 362--386

2020
[10]

Haber, A. 2023. Cart-Pole Control Environment in OpenAI Gym/Gymnasium – Introduction to OpenAI Gym. Blog post. Accessed: Feb 24, 2025

2023
[11]

Hornik, K. 1991. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2): 251--257

1991
[12]

Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 25, 1097--1105

2012
[13]

LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. Nature, 521(7553): 436--444

2015
[14]

LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11): 2278--2324

1998
[15]

X.; Nagabandi, A.; Abbeel, P.; and Levine, S

Lee, A. X.; Nagabandi, A.; Abbeel, P.; and Levine, S. 2020. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33: 741--752

2020
[16]

A.; Veness, J.; Bellemare, M

Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540): 529--533

2015
[17]

Raffin, A.; Hill, A.; Gleave, A.; Kanervisto, A.; Ernestus, M.; and Dormann, N. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268): 1--8

2021
[18]

Siami-Namini, S.; Tavakoli, N.; and Namin, A. S. 2019. The performance of LSTM and BiLSTM in forecasting time series. In 2019 IEEE International conference on big data (Big Data), 3285--3292. IEEE

2019
[19]

Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27

2014
[20]

S.; Barto, A

Sutton, R. S.; Barto, A. G.; et al. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge

1998
[21]

Yang, F.; Li, W.; Li, C.; and Miao, Q. 2019. State-of-charge estimation of lithium-ion batteries based on gated recurrent neural network. Energy, 175: 66--75

2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

K.; and Vikrama, R

Balaji, E.; Brindha, D.; Elumalai, V. K.; and Vikrama, R. 2021. Automatic and non-invasive Parkinson’s disease diagnosis and severity rating using LSTM network. Applied Soft Computing, 108: 107463

work page 2021

[4] [4]

Bengio, Y.; Simard, P.; and Frasconi, P. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2): 157--166

work page 1994

[5] [5]

Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. Openai gym. arXiv preprint arXiv:1606.01540

arXiv 2016

[6] [6]

J.; Preindl, M.; and Emadi, A

Chemali, E.; Kollmeyer, P. J.; Preindl, M.; and Emadi, A. 2018. State-of-charge estimation of Li-ion batteries using deep neural networks: A machine learning approach. Journal of Power Sources, 400: 242--255

work page 2018

[7] [7]

Cho, K.; Van Merri \"e nboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

arXiv 2014

[8] [8]

Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555

arXiv 2014