Pith. sign in

REVIEW 4 major objections 6 minor 36 references

A Convolution and Attention Based Encoder for Reinforcement Learning under Partial Observability

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a lightweight encoder combining depthwise separable convolution and multi-head self-attention lets the TD3 actor-critic outperform recurrent baselines on continuous-control POMDP tasks at comparable parameter counts.

desk verdict Plausible lightweight history-encoder idea, but the paper overclaims on both the theory and the empirical results; worth refereeing with a request for major revision, not a desk reject. read the letter →

arxiv 2505.23857 v2 pith:UQEABDTD submitted 2025-05-29 cs.LG cs.AI

Wuhao Wang , Zhiyong Chen This is my paper
classification cs.LGcs.AI
keywords partiallyobservableMarkovdecisionprocesshistoryencoderdepthwiseseparableconvolutionmulti-headself-attentionactor-criticTD3continuouscontrolrecurrentreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that recurrent networks are not needed for competitive reinforcement learning under partial observability. It reformulates a POMDP as an MDP over a fixed-length window of past observations, then encodes that window with a lightweight temporal encoder built from depthwise separable convolution and multi-head self-attention, integrated into the TD3 actor-critic algorithm. On four continuous-control benchmarks where only position sensors are visible, the resulting method, CAE-TD3, is reported to outperform the LSTM-based baseline in all four environments and the stronger recurrent baseline RMF in three of four, at parameter counts close to or below the recurrent alternatives. A fully observable variant also beats plain TD3 on three of the four tasks. If the comparison is fair, the paper would show that a parallel, parameter-light history encoder can replace recurrent memory in model-free POMDP control.

What carries the argument

The load-bearing object is the Convolution and Attention based Encoder (CAE). It processes a window of past observations and actions by applying depthwise separable convolution separately along the observation axis and the time axis, then multi-head self-attention over the resulting feature grid, and finally average pooling into a fixed-size vector. This vector replaces the recurrent hidden state in both the actor and the critic: convolution captures local correlations across sensors and neighboring time steps, attention assigns adaptive importance to distant positions, and the separate encoders for the two streams keep the learning objectives independent. The accompanying theoretical derivation uses a product-form joint observation function to turn the POMDP into an MDP, which is offered as the reason such an encoder should suffice.

What would settle it

A controlled rerun of the four partial-observability benchmarks in which LSTM-TD3 and RMF receive the same per-environment tuning procedure as CAE-TD3, with a fixed seed count and seed-level reporting; if a tuned recurrent baseline matches or beats CAE-TD3 on Ant-v4, Pendulum-v4, or Hopper-v4, the paper's central superiority claim fails. A separate check is to construct a POMDP whose observation noise is not product-form across the window and show that the claimed POMDP-to-MDP conversion materially changes the value function.

Watch

Extended reading notes

Core claim

The central claim is that a fixed-length observation history is a valid augmented state for a POMDP, and that a history encoder made of depthwise separable convolution followed by multi-head self-attention and average pooling can extract a compact representation from that state more effectively than recurrent encoders at comparable model size. The conversion is carried out by defining a joint observation function over a sliding window of length $N+1$; under the stated observability and product-form assumptions, the windowed observation is treated as fully observable. The authors insert separate encoders into the actor and the critic of TD3 and report results on Ant-v4, Pendulum-v4, Hopper-v4, and Walker-v4 with position-only observations: CAE-TD3 gets the best mean return in three environments and is competitive with RMF in Walker-v4 with lower variance, while clearly beating LSTM-TD3 and fixed-window concatenation everywhere. Under full observability, CAE-TD3-FO outperforms TD3 and LSTM-TD3 on three of four tasks, which the authors take as evidence that structured history also helps when the state is complete.

Load-bearing premise

The load-bearing premise is that the comparison is fair: the recurrent baselines run with their default hyperparameters and the new method's history length is chosen per environment after seeing results, so if the baselines were tuned with the same care, the reported margins could shrink or disappear.

Editorial extensions

If this is right

  • If CAE-TD3 works as reported, recurrent memory is not required for strong control under partial observability; a fixed window plus convolution and attention is enough on these benchmarks.
  • A lightweight temporal encoder can replace recurrent and full-Transformer history models in TD3-style actor-critic methods without a parameter explosion, keeping history-aware RL cheaper than recurrent or Transformer alternatives.
  • The fully observable results imply that feeding a structured observation window can improve stability and sample efficiency even when the current state would in principle be sufficient.
  • The appendix variant comparisons suggest that sharing one encoder between actor and critic, or making the actor generate the whole action sequence, hurts performance; keeping observation and action streams separate appears to matter.
  • The theoretical reduction, if it holds under its assumptions, supports treating fixed-length histories as MDP states in model-free algorithms beyond TD3.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the per-environment history-length ablation implies that one fixed window size is suboptimal across tasks, so an adaptive or learned history length is a natural testable next step.
  • Beyond the paper: the theory assumes a product-form joint observation distribution; running CAE-TD3 on a POMDP with correlated observation noise would separate the encoder's practical value from the theoretical reduction.
  • Beyond the paper: because the encoder is parallel rather than recurrent, it could naturally combine with recurrent or Transformer back-ends or with synchronized multi-agent settings, extensions the paper lists only as future work.
  • Beyond the paper: the reported tables show only mean return over the final 200,000 steps; seed-level reporting with tuned baselines would tell whether the margins on Ant-v4 and Hopper-v4 are stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CAE-TD3, an actor-critic algorithm for POMDPs in which a fixed-length observation history is encoded by a depthwise separable convolution followed by multi-head self-attention and average pooling. The authors argue that fixed-length observation histories turn a POMDP into an 'equivalent MDP' with an augmented state, and that their encoder offers a lightweight alternative to recurrent and full Transformer encoders. They integrate the encoder into TD3 and report experiments on Ant-v4, Pendulum-v4, Hopper-v4, and Walker-v4 under both partial and full observability, comparing with RMF, LSTM-TD3, and a fixed-window TD3 variant. They also include ablations over history length and architectural variants.

Significance. The architecture is plausible and the experimental design covers relevant baselines, including a parameter-count comparison and an ablation of history length. If the empirical claims were supported with proper statistics and a matched evaluation protocol, a lightweight convolution-and-attention history encoder would be a useful addition to POMDP RL. However, the evidence as presented does not establish the headline superiority claim: no seed count is reported, no code is released, the history length is tuned per environment after seeing results, and Table 2 contains instances where a baseline mean exceeds CAE-TD3. The theoretical reformulation in Section II-A does not rigorously produce a policy-independent MDP. The central claims therefore require substantial additional support.

major comments (4)
  1. [§IV-B, Table 2] The central empirical claim of 'superior performance' is not supported by the reported table. Under partial observability on Walker-v4, RMF's mean (1970.68) exceeds CAE-TD3's (1907.39), and the text in Section IV-B acknowledges only 'comparable peak performance'. Under full observability, CAE-TD3-FO is worse than both TD3 and LSTM-TD3 on Hopper-v4 (2302.01 vs 3073.67 and 3136.92) and on Pendulum-v4 (-187.78 vs -154.79 and -146.91), despite Section IV-C claiming it 'frequently outperforms' them. Additionally, the table reports mean±std but never states the number of seeds, so the standard deviations cannot be interpreted as across-seed variation and no significance statement is possible. Please report per-seed results and a statistical test for all headline comparisons.
  2. [§IV-A, §IV-D, Table 3] The comparison protocol gives CAE-TD3 an uncontrolled advantage. The history length N is selected after seeing results (N=3 for Ant-v4, N=5 for Walker-v4, unreported for Pendulum-v4 and Hopper-v4), while RMF and LSTM-TD3 use default hyperparameters from their original papers. This makes the reported margins hard to interpret. Further, Table 3 shows that CAE-TD3 has 200,231 parameters versus 150,658 for LSTM-TD3 on Hopper-v4, so the 'same parameter budget' claim in Contribution 1 is not literally satisfied. Please report N for all environments, justify the choice through validation or a sensitivity analysis, match the tuning budget of the baselines, and either reduce the parameter count or rephrase the claim.
  3. [§II-A] The POMDP-to-MDP equivalence does not follow from the equations as written. First, the joint observation function \bar{O} is defined as the product of marginal factors O_k(o_{t-N+k}|s_{t-N}); this factorization is valid only under conditional independence of observations given the initial hidden state, which is not implied by the HMM structure, because observations separated in time are correlated through the hidden-state chain. Second, the recursive definitions Q_1,...,Q_N propagate the belief through transition probabilities but never condition on the intervening observations o_{t-N+1},...,o_t, so Q_N(s_t|\bar{s}_t) is not the posterior given the history. Third, \bar{P}^\pi and \vartheta depend on the policy \pi being learned, so the 'new MDP' has a policy-dependent transition kernel rather than a fixed one. The section should be reframed as an approximation or heuristic motivation, not an equivalent-MDP theorem.
  4. [Reproducibility (throughout)] The manuscript does not provide code, per-seed learning curves, or a specification of the random seed protocol, and the 'standard deviation' statistic is ambiguous. For a paper whose headline claim is empirical, this is a load-bearing omission. Please release code and data or, at minimum, provide a complete experimental protocol including seeds, environment wrappers, and evaluation intervals.
minor comments (6)
  1. [§III-B, Eq. (4)] The displayed objective has unbalanced parentheses; the target Q-value term appears to extend into the subtracted Q_theta term. Please fix the notation.
  2. [Algorithm 1, line 10] 'wights' should read 'weights'.
  3. [§II-A] The POMDP tuple is written as (S,A,P,R,S_o,O); this duplicates the observation space symbol and omits a distinct observation-space component (typically Ω or O).
  4. [§IV-D, Figure 6] The ablation reports smoothed curves with a moving average but no error bars or seed count; please add per-seed runs or at least state the number of runs used.
  5. [References] References [8] and [21] list the same paper by Hausknecht and Stone under different titles; please deduplicate.
  6. [§IV-E] Since the text notes that parameter counts vary with environment, Table 3 should either report counts for all four environments or explicitly state that only Hopper-v4 was measured.

Circularity Check

1 steps flagged · score 3.0 of 10

The empirical head-to-head against external recurrent baselines (LSTM-TD3, RMF, TD3) is not circular, but the POMDP-to-MDP reformulation in Section II-A is built from the policy being learned: P̄^π is defined via Q_N, which is recursively defined via π, so the theoretical contribution reduces to its own construction, with the dependence disclosed in the paper's own text.

  1. self definitional [Section II-A (Conversion from POMDP to MDP): definitions of P^π, belief recursion Q_0...Q_N, augmented transition P̄^π, 'new MDP' summary, and the stated challenge that follows.]
    "¯Pπ(¯st+1 | ¯st,a t) = ¯Pπ(ot+1 | ¯st,a t) = X st+1,st O(ot+1 |s t+1)P(s t+1 |s t,a t)Q N (st | ¯st). Note thatOandQ N depend onπ. ... In summary, the problem is reformulated as a new MDP ( ¯S,A, ¯Pπ, ¯R). ... The first challenge is that the transition function ¯Pπ depends on the action policy itself."

    The claimed 'equivalent MDP' is not derived from the POMDP environment; it is assembled from the policy being learned. The chain is: P^π(s_{t+1}|s_t) = Σ_{a_t} P(s_{t+1}|s_t,a_t)π(a_t|s_t); the belief states Q_0,...,Q_N are built recursively from P^π; and the new MDP's transition is then defined as P̄^π = Σ O P Q_N. Hence the augmented-state transition reduces by construction to the unknown policy π, and the paper concedes this ('Note thatOandQ N depend onπ' and 'the transition function ¯Pπ depends on the action policy itself').

full rationale

This score reflects one genuine but explicitly acknowledged definitional circularity in the paper's secondary theoretical claim, while the headline empirical claim is self-contained and tested against external baselines. (1) No self-citations appear anywhere in the reference list, so no load-bearing self-citation is present. (2) The empirical comparison is not circular: LSTM-TD3 [26], RMF [28], and TD3 [27] are external methods with their own published hyperparameters, and Table 2 reports measured returns from environment interaction, not quantities forced by construction. The abstract's 'superior performance' is contradicted on Walker-v4 (RMF 1970.68 vs CAE-TD3 1907.39), the seed count is unreported, and Table 3 shows CAE-TD3 (200,231 params) is not literally at the same parameter budget as LSTM-TD3 (150,658); those are empirical-validity issues, not circularity. (3) The history length N (3 for Ant-v4, 5 for Walker-v4) and the architectural variant were selected after seeing results ('The final structure was selected based on empirical results'), which biases the comparison, but the reported numbers are still measured performance, so this is test-set selection, not a fitted parameter renamed as a prediction. (4) In Section II-A, the POMDP-to-MDP reformulation is self-definitional: the transition P̄^π of the 'new MDP' is defined via Q_N, which is recursively defined from P^π, i.e., from the policy π that the learning algorithm is trying to find. The paper discloses this ('Note thatOandQ N depend onπ'; 'the transition function ¯Pπ depends on the action policy itself'), and the product-form joint observation function is an assumed independence rather than the joint of a general HMM. Because the key dependence is stated in the text, this is scored as an acknowledged partial circularity in a non-central contribution, not as a concealed derivation collapse. Verdict: the central encoder claim stands on external evidence; only the theoretical framing reduces to its own construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The empirical central claim depends on tuned hyperparameters (history length, architecture, TD3 settings) that are not fully disclosed. The theoretical conversion depends on an unjustified independence factorization and a policy-dependent transition, so the paper does not provide a self-contained derivation. No new physical or conceptual entities are introduced.

free parameters (3)
  • History length N = N=3 for Ant-v4, N=5 for Walker-v4, unreported for Pendulum-v4 and Hopper-v4
    Selected per environment based on observed performance in the ablation (Section IV-D). The main experiments do not state the history length, so the input window is a tuned hyperparameter that directly affects results.
  • Encoder architecture hyperparameters = Not specified
    Number of attention heads, convolution kernel sizes, channel counts, MLP sizes, and normalization details are not reported; the final architecture was chosen after comparing variants (Appendix VI).
  • TD3 hyperparameters = Not reported
    The paper says it adopts TD3 but does not list learning rates, batch size, discount factor, exploration noise, target update rates, etc., which are required to reproduce the results.
assumptions (4)
  • ad hoc to paper Observations in a fixed-length history are conditionally independent given the initial hidden state s_{t-N}
    Invoked in Section II-A via \bar{O}(\bar{s}_t | s_{t-N}) := \prod_{k=0}^N O_k(o_{t-N+k} | s_{t-N}). This factorization is not valid for a general HMM, where intermediate hidden states create dependencies among observations.
  • domain assumption The observation function is independent of the action
    Stated in Section II-A as a simplified case covering the benchmarks. This restricts the generality of the theoretical conversion.
  • domain assumption The system is observable in the sense that the observation sequence is informative about the latent state
    Assumed in Section II-A to define the posterior Q(s_{t-N} | \bar{s}_t). Without observability the history may not determine the state.
  • ad hoc to paper The transition kernel of the augmented process depends on the learned policy \pi
    Section II-A defines P^\pi and Q_N under \pi, so the 'equivalent MDP' is not a fixed environment; its transition depends on the policy being learned.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Convolution and Attention Based Encoder for Reinforcement Learning under Partial Observability." pith.science (2026). https://pith.science/paper/UQEABDTD

@misc{pith2026250523857,
  author       = {Pith},
  title        = {Pith review of: A Convolution and Attention Based Encoder for Reinforcement Learning under Partial Observability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQEABDTD}},
  note         = {Machine review of arXiv:2505.23857}
}
read the original abstract

Partially Observable Markov Decision Processes (POMDPs) remain a core challenge in reinforcement learning due to incomplete state information. We address this by reformulating POMDPs as fully observable processes with fixed-length observation histories as augmented states. To efficiently encode these histories, we propose a lightweight temporal encoder based on depthwise separable convolution and self-attention, avoiding the overhead of recurrent and Transformer-based models. Integrated into an actor-critic framework, our method achieves superior performance on continuous control benchmarks under partial observability. More broadly, this work shows that lightweight temporal encoding can improve the scalability of AI systems under uncertainty. It advances the development of agents capable of reasoning robustly in real-world environments where information is incomplete or delayed.

Figures

Figures reproduced from arXiv: 2505.23857 by the authors.

Figure 1
Figure 1. Actor–critic architecture with history encoder for POMDP. Past [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Procedure of the history encoder. The white blocks represent observation elements. Depthwise convolution merges information along both the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Actor–critic architecture of CAE-TD3 with full observation (CAE [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Training performance across four partially observable environments comparing CAE-TD3 with baseline algorithms. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Training performance across four fully observable environments comparing CAE-TD3-FO with baseline algorithms. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Training performance with different history lengths in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Second variant of CAE-TD3, where the actor generates the complete [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 7
Figure 7. Figure 7: First variant of CAE-TD3, where actions and observations are jointly [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Comparison of training performance between CAE-TD3 and Variant 1 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 11
Figure 11. Figure 11: Third variant of CAE-TD3, a simplified version of Variant 2, where [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Comparison of training performance between CAE-TD3 and [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 29 canonical work pages

  1. [1]

    Outracing champion gran turismo drivers with deep reinforcement learning,

    P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subra- manian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, L. Gilpin, V . Kompella, P. Khandelwal, H. Lin, P. MacAlpine, D. Oller, C. Sherstan, T. Seno, M. D. Thomure, H. Aghabozorgi, L. Barrett, 9 Fig. 11. Third variant of CAE-TD3, a simplified version of Variant 2, where the actor ...

    work page 2022

  2. [2]

    Perceiving the world: Question-guided reinforcement learning for text-based games,

    Y . Xu, M. Fang, L. Chen, Y . Du, T. Zhou, and C. Zhang, “Perceiving the world: Question-guided reinforcement learning for text-based games,” in Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics, vol. 1, (Dublin, Ireland), pp. 538–560, Association for Computational Linguistics, 2022

    work page 2022

  3. [3]

    Deep reinforcement learning in health- care and bio-medical applications,

    P. Thakur and N. S. Talwandi, “Deep reinforcement learning in health- care and bio-medical applications,” in2024 IEEE International Confer- ence on Computing, Power and Communication Technologies (IC2PCT), vol. 5, pp. 742–747, 2024

    work page 2024

  4. [4]

    Deep reinforcement learning in radiation therapy planning optimization: A comprehensive review,

    C. Li, Y . Guo, X. Lin, X. Feng, D. Xu, and R. Yang, “Deep reinforcement learning in radiation therapy planning optimization: A comprehensive review,”Physica Medica, vol. 125, p. 104498, 2024

    work page 2024

  5. [5]

    Magnetic control of tokamak plasmas through deep reinforcement learning,

    J. Degrave, F. Felici, J. Buchli, M. Neunert, K. Fricke, A. Doerr, A. Huber, M. Riedmiller, D. Hafner, A. Rajeswaran,et al., “Magnetic control of tokamak plasmas through deep reinforcement learning,” Nature, vol. 602, no. 7897, pp. 414–419, 2022

    work page 2022

  6. [6]

    Reinforcement learning for decision-making and control in power systems,

    X. Chen, G. Qu, Y . Tang, S. Low, and N. Li, “Reinforcement learning for decision-making and control in power systems,” inWomen in Power, pp. 265–285, Springer, 2023

    work page 2023

  7. [7]

    Reinforcement learning with long short-term memory,

    B. Bakker, “Reinforcement learning with long short-term memory,” in Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS’01, (Cambridge, MA, USA), p. 1475–1482, MIT Press, 2001

    work page 2001

  8. [8]

    Deep recurrent q-learning for partially observable mdps,

    M. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,” inProceedings of the AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents, (Arlington, Virginia, USA), AAAI Press, 2015

    work page 2015

Show all 36 references
  1. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Łukasz Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 5998–6008, 2017

    2017

  2. [10]

    Decision transformer: Reinforcement learning via sequence modeling,

    L. Chen, K. Lu, A. Rajeswaran, K. Xu, A. Grover, and P. Abbeel, “Decision transformer: Reinforcement learning via sequence modeling,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 15084–15097, 2021

    2021

  3. [11]

    Online decision transformer,

    Q. Zheng, A. Zhang, and A. Grover, “Online decision transformer,” in Proceedings of the 39th International Conference on Machine Learning, vol. 162 ofProceedings of Machine Learning Research, pp. 27042– 27059, PMLR, 17–23 Jul 2022

    2022

  4. [12]

    Trajectory transformer: Model-based reinforcement learning with long-term dependencies,

    M. Janner, Q. Li, S. Levine, and C. Finn, “Trajectory transformer: Model-based reinforcement learning with long-term dependencies,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 11884–11895, 2021

    2021

  5. [13]

    Optimal control of markov processes with incomplete state information,

    K. J. ˚Astr¨om, “Optimal control of markov processes with incomplete state information,”Journal of Mathematical Analysis and Applications, vol. 10, no. 1, pp. 174–205, 1965

    1965

  6. [14]

    Multi- agent rollout and policy iteration for pomdp with application to multi- robot repair problems,

    S. Bhattacharya, S. Kailas, S. Badyal, S. Gil, and D. Bertsekas, “Multi- agent rollout and policy iteration for pomdp with application to multi- robot repair problems,” inProceedings of the 2020 Conference on Robot Learning, pp. 1814–1828, PMLR, 2021

    2020

  7. [15]

    Controlling contact-rich manipulation under partial observability,

    M. R. Dogar and S. S. Srinivasa, “Controlling contact-rich manipulation under partial observability,” inProceedings of Robotics: Science and Systems (RSS), 2020

    2020

  8. [16]

    Magic: Learning macro-actions for online pomdp planning,

    Y . Lee, P. Cai, and D. Hsu, “Magic: Learning macro-actions for online pomdp planning,” inProceedings of Robotics: Science and Systems (RSS), (Virtual), 2021

    2021

  9. [17]

    Optimizing active surveil- lance for prostate cancer using partially observable markov decision processes,

    W. Li, B. T. Denton, and T. M. Morgan, “Optimizing active surveil- lance for prostate cancer using partially observable markov decision processes,”European Journal of Operational Research, vol. 299, no. 1, pp. 273–287, 2022

    2022

  10. [18]

    Diagnostic policies optimization for chronic diseases based on pomdp model,

    Y . Zhang, J. Wang, Z. Li, and Y . Liu, “Diagnostic policies optimization for chronic diseases based on pomdp model,”Healthcare, vol. 10, no. 2, p. 283, 2022

    2022

  11. [19]

    Planning and acting in partially observable stochastic domains,

    L. P. Kaelbling, M. L. Littman, and A. R. Cassandra, “Planning and acting in partially observable stochastic domains,”Artificial Intelligence, vol. 101, no. 1-2, pp. 99–134, 1998

    1998

  12. [20]

    Finding structure in time,

    J. L. Elman, “Finding structure in time,”Cognitive Science, vol. 14, no. 2, pp. 179–211, 1990

    1990

  13. [21]

    Deep recurrent q-learning for partially observable mdps,

    M. J. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,”ArXiv, vol. abs/1507.06527, 2015

    2015 arXiv

  14. [22]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  15. [23]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  16. [24]

    Recurrent determin- istic policy gradient method for bipedal locomotion on rough terrain challenge,

    D. R. Song, C. Yang, C. McGreavy, and Z. Li, “Recurrent determin- istic policy gradient method for bipedal locomotion on rough terrain challenge,”2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), pp. 311–318, 2017

    2018

  17. [25]

    Recurrent soft actor critic reinforcement learning for demand response problems,

    U. Ludolfinger, D. Zinsmeister, V . S. Peri ´c, T. Hamacher, S. Hauke, and M. Martens, “Recurrent soft actor critic reinforcement learning for demand response problems,” in2023 IEEE Belgrade PowerTech, pp. 1– 6, 2023

    2023

  18. [26]

    Memory-based deep reinforcement learning for pomdps,

    L. Meng, R. Gorbet, and D. Kuli ´c, “Memory-based deep reinforcement learning for pomdps,” in2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5619–5626, 2021

    2021

  19. [27]

    Addressing function ap- proximation error in actor-critic methods,

    S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function ap- proximation error in actor-critic methods,” inProceedings of the 35th International Conference on Machine Learning(J. Dy and A. Krause, eds.), vol. 80 ofProceedings of Machine Learning Research, pp. 1587– 1596, P...

    2018

  20. [28]

    Recurrent model-free RL can be a strong baseline for many POMDPs,

    T. Ni, B. Eysenbach, and R. Salakhutdinov, “Recurrent model-free RL can be a strong baseline for many POMDPs,” inProceedings of the 39th International Conference on Machine Learning, vol. 162 ofProceedings of Machine Learning Research, pp. 16691–16723, PMLR, 2022

    2022

  21. [29]

    Ode-based recurrent model-free reinforcement learning for pomdps,

    X. Zhao, D. Zhang, L. Han, T. Zhang, and B. Xu, “Ode-based recurrent model-free reinforcement learning for pomdps,” inProceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, 2023

    2023

  22. [30]

    Efficient recurrent off-policy rl requires a context-encoder-specific learning rate,

    F.-M. Luo, Z. Tu, Z. Huang, and Y . Yu, “Efficient recurrent off-policy rl requires a context-encoder-specific learning rate,” inProceedings of the 38th International Conference on Neural Information Processing Systems, (Vancouver, Canada), 2024

    2024

  23. [31]

    The moving horizon estimation concept,

    R. Bitmead, M. Gevers, and V . Wertz, “The moving horizon estimation concept,” inMoving Horizon Estimation: Theory and Applications, ch. 2, pp. 5–20, London: Springer-Verlag, 1990

    1990

  24. [32]

    R. S. Sutton and A. G. Barto,Reinforcement learning: An introduction. MIT press, 2018

    2018

  25. [33]

    Xception: Deep learning with depthwise separable con- volutions,

    F. Chollet, “Xception: Deep learning with depthwise separable con- volutions,”2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800–1807, 2016. 10

    2017

  26. [34]

    Thin mobilenet: An enhanced mobilenet architecture,

    D. Sinha and M. El-Sharkawy, “Thin mobilenet: An enhanced mobilenet architecture,” in2019 IEEE 10th Annual Ubiquitous Computing, Elec- tronics & Mobile Communication Conference (UEMCON), pp. 0280– 0285, 2019

    2019

  27. [35]

    Gymnasium,

    M. Towers, J. K. Terry, A. Kwiatkowski, J. U. Balis, G. d. Cola, T. Deleu, M. Goul ˜ao, A. Kallinteris, A. KG, M. Krimmel, R. Perez- Vicente, A. Pierr ´e, S. Schulhoff, J. J. Tai, A. T. J. Shen, and O. G. Younis, “Gymnasium,” Mar. 2023

    2023

  28. [36]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inNorth American Chapter of the Association for Computational Linguistics, 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.