Pith. sign in

REVIEW 4 major objections 8 minor 40 references

Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making

T0 review · 4 major / 8 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A double-check step makes offline RL policies hit their requested return targets.

desk verdict A plausible inference-time trick for return alignment in offline RL, with better experiments than theory; the promised bound is missing and the extrapolation claims outrun the Q-learning. read the letter →

arxiv 2508.16420 v3 pith:75CFN56V submitted 2025-08-22 cs.LG

Yue Pei , Hongming Zhang , Chao Gao , Martin Müller , Yingying Zhang , Mengxiao Zhu , Hao Sheng , Ziliang Chen
show 2 more authors
Liang Lin Haogang Zhu
This is my paper
classification cs.LG
keywords offlinereinforcementlearningreturnconditioningtargetalignmenttransformersequencemodelingdouble-checkverificationexpectileQ-learningcontrollablepolicy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to fix a weakness of return-conditioned offline reinforcement learning: a policy told to aim for a particular return often misses, especially when the requested return is rare or absent in the training data. The proposed method, Doctor, trains one Transformer with two objectives at once — reconstructing masked trajectories (supervised) and learning action-values from the same representations (temporal-difference). At inference, it samples several nearby target returns, generates candidate actions for each, and keeps the action whose predicted value is closest to the requested return. The paper argues this double-check keeps alignment error bounded by the coverage of candidate values around the target and the verifier's accuracy. If correct, practitioners get a single offline-trained policy that can be dialed to conservative or aggressive performance levels rather than only maximizing return.

What carries the argument

The double-check selection rule: sample N returns in a δ-window around the target, generate N candidate actions, and pick the action minimizing |Q(s,a)-R_target|. It is carried by a joint training objective that couples masked trajectory reconstruction (action candidates from a bidirectional Transformer) with in-sample expectile value learning (the verifier). The rule's justification is that value-level alignment error is bounded by how well the candidate actions cover the target return and how accurately Q estimates them.

What would settle it

Train Doctor on hopper-medium-replay with 30% of the best trajectories removed, then flip the selection rule to pick the candidate with Q farthest from the target. If actual-return alignment stays nearly as good as the original rule, the action-value check is not the mechanism producing alignment; if alignment collapses, the Q-verifier is doing the claimed work.

Watch

Extended reading notes

Core claim

The central claim is that verification by a learned action-value function, applied to candidates generated under nearby return prompts, turns a return-conditioned sequence model into a precisely controllable policy. Concretely, Doctor optimizes a shared masked-trajectory Transformer with reconstruction loss and an in-sample expectile Q-learning loss; the Q-head gives each generated action a value estimate. Inference draws N returns uniformly within a small window of the requested target, feeds N trajectories through the model, and selects argmin_i |Q(s,a_i)-R_target|. The paper shows that with an optimal Q-function this selection either achieves the target when it is below the attainable opt

Load-bearing premise

The whole selection rule assumes the learned Q-function is accurate for the candidate actions the Transformer proposes, including actions in underrepresented or extrapolated return regions; Section 4.4 states the ideal case Q = Q*.

Editorial extensions

If this is right

  • A single Doctor policy can hit a continuum of target returns, interpolating into underrepresented return regions and extrapolating modestly beyond the maximum return in the dataset.
  • Better target alignment translates into stronger return maximization: Doctor outperforms DT, MTM, IQL, CQL, and TD3+BC on most D4RL locomotion tasks and on Maze2D.
  • On a simulated clinical benchmark (EpiCare), the same policy can modulate treatment aggressiveness, trading higher returns for more adverse events under aggressive targets and safer behavior under moderate targets.
  • Increasing the number of candidate samples N improves alignment, indicating that the target prompt alone is insufficient; the value check is what filters noisy conditioning.
  • Random masking and the double-check both contribute: ablating either raises mean absolute return error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the selection rule is only as good as the Q-estimates, an uncertainty-aware version — weighting candidates by predicted variance — would likely retain alignment where a single-point Q estimate is unreliable, especially far outside data support.
  • The same generate-then-verify pattern could transfer to other conditional generation settings, such as text or image generation with a user-specified attribute, where a learned scorer replaces the Q-function and the candidate pool is sampled from nearby prompts.
  • The paper's bound on alignment error suggests an ordering principle: coverage of the candidate pool around the target matters as much as verifier accuracy. A testable extension is to keep N fixed but spread candidate returns wider than 5% of R_max to map the coverage-alignment trade-off curve.
  • A closed-loop variant could adapt the δ-window online: start wide to find reachable values, then narrow around the best candidate, potentially reducing N without losing alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes Doctor, a bidirectional masked transformer for offline RL that is trained jointly with a trajectory-reconstruction objective and an in-sample expectile Q-learning objective. At inference, Doctor samples multiple returns in a window around the requested target, generates a corresponding set of candidate actions, and selects the action whose predicted Q-value is closest to the target (Eq. 9). The authors evaluate on D4RL locomotion, Maze2D, Adroit, and the EpiCare clinical benchmark, reporting improved target-return alignment, competitive return maximization, and the ability to modulate treatment aggressiveness in one policy. The abstract promises a formal bound on alignment error in terms of candidate-value coverage and verifier accuracy, but Section 4.4 provides only a two-case analysis under the assumption Q=Q*, and no such bound is derived.

Significance. If the empirical alignment results are taken at face value, Doctor is a simple and useful contribution: the double-check mechanism is intuitive, the ablation in Table 2 shows it helps, and the D4RL and EpiCare experiments cover a broader range of settings than many target-conditioning papers. The paper also explicitly notes in Appendix B that the mechanism depends on accurate Q-values within data-supported regions, which is an honest limitation. However, the central theoretical claim is not delivered, and the extrapolation claim rests on unverified Q-accuracy for out-of-support candidate actions. The core method remains defensible as an empirical proposal, but the current text overstates what is shown. The paper does not include machine-checked proofs or a code release, so reproducibility currently depends on the hyperparameter details in Appendix E.

major comments (4)
  1. [Abstract and Section 4.4] The abstract states that the verifier-guided selection rule has "value-level alignment error bounded by candidate-value coverage around the target return and verifier accuracy." No such bound is actually presented. Section 4.4 assumes Q=Q* and gives only Case 1 and Case 2 arguments; it never defines candidate-value coverage, verifier accuracy, or a formal alignment-error quantity. Since Eq. (9) minimizes |q_{t,i}-R_t| by construction, the selection rule trivially aligns the predicted Q with the target; the nontrivial claim is that the realized return aligns. Please either prove a bound under explicit coverage and accuracy assumptions or revise the abstract and Section 4.4 to describe the result as a motivating case analysis rather than a theorem.
  2. [Eq. (5), Appendix B, Figure 4] The Q-function is trained with in-sample expectile regression, and Eq. (1) restricts bootstrapping to dataset actions. Appendix B concedes that the double-check mechanism "is dependent on accurate Q-value predictions within data-supported regions." However, Figure 4 reports alignment for target returns above the maximum dataset return, where candidate actions are not data-supported and the Q-function has no training signal to be accurate. The selection rule can make |q-R| small even when q is completely wrong, so the alignment curves may reflect alignment of predicted Q rather than realized return. To support the extrapolation claim, the paper should provide direct evidence of Q accuracy on the selected candidate actions, e.g., calibration of q against realized returns or a comparison of selected actions with and without value-based selection in the extrapolated regime.
  3. [Eqs. (7)-(9) and Section 4.4, Case 2] Even with a perfect Q*, the mechanism can align the realized return only if one of the N candidate actions has Q*(s,a) close to R_t. The method samples returns in Eq. (7), not actions; the candidate actions are generated by the transformer. Section 4.4 Case 2 only says "There may exist actions a such that Q*(s,a)≈R_t" and does not show that such an action is present in the candidate set, nor that the minimizing candidate has Q close enough. This candidate-coverage condition is exactly what the abstract promises to quantify, but it is never stated or tested. Please state the condition explicitly and report the empirical distribution of min_i |q_{t,i}-R_t| in the experiments, so readers can see whether coverage actually holds in the regimes where alignment is claimed.
  4. [Figure 3 and Appendix D.1] The illustration is described as showing that the agent "achieves an actual return that aligns with the target return," with target returns 10, 40, and 80 and actual returns 37.80, 46.85, and 80.16. For targets 10 and 40, the actual returns are far from the targets (relative errors of 278% and 17%, respectively). This is a concrete overclaim in the central phenomenon under study. If low targets are not well aligned, the text should say so and report the alignment error; alternatively, the demonstration should use a target range where the claim actually holds. The discrepancy should also be reconciled with the strong alignment curves shown later.
minor comments (8)
  1. [Eq. (9)] The notation a_{t,i} = arg min_{a_{t,i}} (|q_{t,i}-R_t|) overloads the variable. It should be written as i* = argmin_i |q_{t,i}-R_t|, then a_t = a_{t,i*}.
  2. [Algorithm 1] The text states that the desired target is updated as R_{t+1} = (R_t - r_t)/gamma, but this update is omitted from Algorithm 1. Please include it so the inference procedure is self-contained.
  3. [Eq. (7)] The notation Unif(delta(R_t), N) is nonstandard. Clarify that N samples are drawn uniformly from the interval [R_t - delta, R_t + delta].
  4. [Figure 4] The alignment curves in Figure 4 have no error bars or seed information. Given that Table 2 reports sizable standard deviations, the curves should be shown with mean and variance over seeds, or at least a statement of how many seeds were used for the alignment evaluation.
  5. [Section 5.3] The sentence "When N=2, the model utilizes an inference strategy akin to the RvS-based method" is confusing because N=2 still uses the double-check mechanism. It would be clearer to say that with small N the double-check has little effect and the behavior resembles plain RvS conditioning.
  6. [Appendix C.1] Typo: "For expample" should be "For example." Also "99% percentile return" in Figures 1 and 4 should be "99th percentile return."
  7. [Tables 2 and 3] Table 2 says results are averaged over three seeds while Table 3 says five random seeds. Please clarify whether the ablation used a different number of seeds and, if so, why.
  8. [General] No code or data link is provided. For a method with several sensitive hyperparameters (N, delta, mask ratios, expectile tau), releasing code would materially improve reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Theoretical justification assumes Q=Q*, making Eq. (9)'s alignment guarantee definitional; empirical evaluation remains independent.

  1. self definitional [Section 4.4 (Justification of Double Check Mechanism), Eq. (9)]
    "Assume our action-value function is optimal, Q=Q*. We select the action a_t that minimizes the absolute difference between the predicted action-value and the desired return according to a_t = arg min_a |Q*(s_t,a)-R_t|. We show that this action selection aligns the expected return with R_t, achieving return alignment."

    The proof of return alignment begins by assuming the learned verifier is the true action-value function. Since Q* is defined as the expected return of an action, Eq. (9) with q=Q* literally selects the action whose expected return is closest to R_t; the conclusion is contained in the premise. The paper does not derive Q≈Q* from the training loss Eq. (5), which is an in-sample expectile regression reliable only inside the data support. Appendix B concedes the mechanism 'is dependent on accurate Q-value predictions within data-supported regions.' The abstract's promised coverage/accuracy bound is not actually given; instead perfect verifier accuracy is assumed, making the alignment guarantee definitional rather than derived.

full rationale

Doctor's core method is tested by executing selected actions in the environment and comparing achieved returns with targets (Figures 3-5), which is an external, falsifiable measure independent of the trained Q values. No parameter is fitted to the evaluation targets and then relabeled as a prediction, and the only self-citations ([19], [22]) are non-load-bearing background on in-sample TD learning. The main circularity is confined to the theoretical justification: Section 4.4 assumes Q=Q*, and under that assumption Eq. (9) trivially minimizes true expected-return distance because Q* is by definition the expected return. The abstract promises an analysis bounding alignment error by candidate-value coverage and verifier accuracy, but the main text instead assumes perfect verifier accuracy outright. Appendix B further concedes the mechanism depends on accurate Q within data-supported regions, which is in tension with the headline extrapolation claims. Thus the analytical support for the extrapolation result reduces to an assumption of the desired property, while the empirical demonstrations remain independent evidence—hence a moderate, not severe, circularity score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central mechanism rests on the validity of the learned Q-function as a verifier for candidate actions, which is an assumption stated nowhere as a proven guarantee. The hyperparameters N, delta, and tau are tuned per benchmark and are free parameters in the evaluation. No new physical entities are introduced.

free parameters (4)
  • expectile tau = 0.7 (Locomotion), 0.9 (Maze2d), 0.8 (Adroit and EpiCare)
    Asymmetric loss weight for Q-learning, tuned per benchmark family; directly controls how the Q-function approximates the best in-sample action value and therefore verifier reliability.
  • N (number of sampled returns) = 300
    Number of candidate trajectories sampled at inference; ablation shows larger N improves alignment, and 300 is used for all main experiments.
  • delta (sampling window) = 5% of R_max in the dataset
    Width of the uniform return sampling interval around the target; tuned per dataset maximum and directly determines candidate-value coverage.
  • mask ratios = [0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 1.0]
    Masking schedule for trajectory reconstruction, inherited from MTM; an architectural hyperparameter affecting learned representations.
assumptions (5)
  • domain assumption The environment is a finite-horizon Markov Decision Process with bounded rewards.
    Standard RL formulation used throughout Section 3.1.
  • domain assumption Expectile regression with tau > 0.5 approximates the maximum over in-sample actions at the next state.
    Borrowed from IQL (ref [18]); used to learn Q without querying out-of-sample actions, as described in Section 4.2 and Appendix B.
  • ad hoc to paper The learned Q-function is sufficiently accurate for the candidate actions generated by the transformer, including underrepresented or extrapolated returns.
    The double-check selection rule assumes verifier reliability; Section 4.4 assumes Q=Q*, but in practice Q is learned from data and may be inaccurate outside data support.
  • ad hoc to paper For target returns below the achievable return, at least one candidate action has a Q-value close to the target.
    Case 2 of Section 4.4 relies on the existence of actions with Q* approximately equal to R_t; candidate-value coverage around the target is assumed rather than proved.
  • domain assumption The EpiCare simulator faithfully represents clinical treatment dynamics.
    Clinical conclusions depend on the simulator; the paper acknowledges in Section 6 that results may not fully reflect real-world healthcare data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making." pith.science (2026). https://pith.science/paper/75CFN56V

@misc{pith2026250816420,
  author       = {Pith},
  title        = {Pith review of: Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/75CFN56V}},
  note         = {Machine review of arXiv:2508.16420}
}
read the original abstract

Target-conditioned sequence models provide a simple interface for controllable offline decision making, but the requested target return can be an unreliable control signal, especially when the target return lies in underrepresented regions of the dataset. This paper proposes Doctor, a hybrid sequence modeling and reinforced verification framework for controllable target-conditioned offline decision making. Doctor trains a shared masked trajectory Transformer with two complementary objectives: masked trajectory reconstruction for candidate generation and in-sample value learning for action-value verification. At inference time, the model samples multiple nearby target returns, generates candidate actions in parallel, and selects the action whose verified value is closest to the requested target return. We analyze this verifier-guided selection rule and show that its value-level alignment error is bounded by candidate-value coverage around the target return and verifier accuracy. Experiments on D4RL and EpiCare show that Doctor improves target-return alignment under reduced high-return coverage, remains competitive on standard offline return-maximization benchmarks, and enables a single policy to modulate between conservative and aggressive operating points in a simulated clinical decision-making task. These results suggest that reinforced verification can improve the controllability of target-conditioned policies.

Figures

Figures reproduced from arXiv: 2508.16420 by the authors.

Figure 1
Figure 1. The actual returns achieved by Doctor and Decision Transformer (DT) con￾ditioned on a wide range of target returns on the Hopper-Medium-Replay dataset. Doctor demonstrates significantly better alignment than DT, not only for well-supported returns within the dataset but also for extrapolated returns beyond the dataset. In recent years, offline reinforce￾ment learning (RL) [1] has made remarkable progress in various … view at source ↗
Figure 2
Figure 2. An overview of Doctor. Left. The training involves reconstructing the original trajectory and estimating the action-value from a partial, randomly masked trajectory. Returns, states and actions are fed into modality-specific embeddings and then processed by the transformers. The value heads estimate the action-value at each timestep. Right. At inference time, Doctor samples actions around the target return and valid… view at source ↗
Figure 4
Figure 4. We evaluate the alignment ability of Doctor on hopper-medium-replay-v2 (top) and walker2d-medium-replay-v2 (bottom) tasks with the top X% returns of trajectories removed. The dashed red line presents the highest return in the dataset. The dashed black lines denote the ideal alignment. Doctor achieves much better alignment across a wide range of target returns compared to DT, MTM [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: Trajectories generated under three different target returns, which are 10, 40 and 80, resulting in actual returns of 37.80, 46.85, and 80.16, respectively, from the left to right. As the target return increases, we observe that the agent is guided closer to the goal. T…
Figure 5
Figure 5. Figure 5: The effect of the sample number N in Doctor. As N increases, Doctor achieves better alignment with the target return. We further analyze the impact of the sampling size N on the align￾ment ability of Doctor in this sec￾tion. To investigate this, we con￾duct an ablation…
Figure 6
Figure 6. Figure 6: Mean returns and adverse-event rates per 10k episodes under moderate (m, 0.4×) and aggressive (a, 0.8×) targets. Num￾bers denote improvement or decrease com￾pared with BC. Results show that (1) DT and MTM exhibit only marginal shifts relative to BC, while Doctor demons…
Figure 7
Figure 7. Figure 7: Mean returns and number of adverse events per 10k episodes (lower is better). [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 32 canonical work pages

  1. [1]

    Levine, A

    S. Levine, A. Kumar, G. Tucker, J. Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01643 (2020)

    arXiv 2005

  2. [2]

    Z. Liu, Y. Zhuang, P. Wu, Y. Liu, Iris: An informa- tion path planning method based on reinforcement learning and information-directed sampling, Pattern Recognition 172 (2026) 112400. doi:https://doi.org/10.1016/j.patcog.2025.112400

    work page arXiv 2026

  3. [3]

    H. Chen, R. Yang, J. Zhang, X. Wen, Y. Chen, D. Yu, C. Bai, Z. Wang, Temporal consistent multi-view perception for robust embodied manipulation, Pattern Recognition 171 (2026) 112177. doi:https://doi.org/10.1016/j.patcog.2025.112177

    arXiv 2026

  4. [4]

    Jiang, S

    B. Jiang, S. Chen, Q. Zhang, W. Liu, X. Wang, Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning, arXiv preprint arXiv:2503.07608 (2025)

    arXiv 2025

  5. [5]

    Hargrave, A

    M. Hargrave, A. Spaeth, L. Grosenick, Epicare: A reinforcement learn- ing benchmark for dynamic treatment regimes, Advances in Neural In- formation Processing Systems 37 (2024) 130536–130568

    work page 2024

  6. [6]

    S. Choi, S. Jang, S. Jung, H. J. Cho, B. Jeon, Deep rein- forcement learning for efficient registration between intraoral-scan meshes and ct images, Pattern Recognition 164 (2025) 111502. doi:https://doi.org/10.1016/j.patcog.2025.111502

    work page arXiv 2025

  7. [7]

    T. Pang, G. Wu, Y. Zhang, B. Wang, Y. Yin, Qfae: Q- function guided action exploration for offline deep rein- forcement learning, Pattern Recognition 158 (2025) 111032. doi:https://doi.org/10.1016/j.patcog.2024.111032

    work page arXiv 2025

  8. [8]

    L. H. Goetz, N. J. Schork, Personalized medicine: motivation, chal- lenges, and progress, Fertility and sterility 109 (6) (2018) 952–963

    work page 2018

Show all 40 references
  1. [9]

    Singla, A

    A. Singla, A. N. Rafferty, G. Radanovic, N. T. Heffernan, Reinforcement learning for education: Opportunities and challenges, arXiv preprint arXiv:2107.08828 (2021). 28

    2021 arXiv

  2. [10]

    H. A. Alawwad, A. Alhothali, U. Naseem, A. Alkhathlan, A. Jamal, En- hancing textual textbook question answering with large language mod- els and retrieval augmented generation, Pattern Recognition 162 (2025) 111332. doi:https://doi.org/10.1016/j.patcog.2024.111332

    2025

  3. [11]

    Jeon, I.-C

    H.-C. Jeon, I.-C. Baek, C.-m. Bae, T. Park, W. You, T. Ha, H. Jung, J. Noh, S. Oh, K.-J. Kim, Raidenv: Exploring new challenges in au- tomated content balancing for boss raid games, IEEE Transactions on Games (2023)

    2023

  4. [12]

    Srinivas, I

    L.Chen, K.Lu, A.Rajeswaran, K.Lee, A.Grover, M.Laskin, P.Abbeel, A. Srinivas, I. Mordatch, Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing sys- tems 34 (2021) 15084–15097

    2021

  5. [13]

    Janner, Q

    M. Janner, Q. Li, S. Levine, Offline reinforcement learning as one big sequence modeling problem, Advances in neural information processing systems 34 (2021) 1273–1286

    2021

  6. [14]

    Emmons, B

    S. Emmons, B. Eysenbach, I. Kostrikov, S. Levine, Rvs: What is essential for offline rl via supervised learning?, arXiv preprint arXiv:2112.10751 (2021)

    2021 arXiv

  7. [15]

    J. Fu, A. Kumar, O. Nachum, G. Tucker, S. Levine, D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.07219 (2020)

    2004 arXiv

  8. [16]

    Kumar, J

    A. Kumar, J. Fu, M. Soh, G. Tucker, S. Levine, Stabilizing off-policy q-learning via bootstrapping error reduction, Advances in Neural Infor- mation Processing Systems 32 (2019)

    2019

  9. [17]

    R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018

    2018

  10. [18]

    Kostrikov, A

    I. Kostrikov, A. Nair, S. Levine, Offline reinforcement learning with implicit q-learning, in: International Conference on Learning Represen- tations, 2022. URLhttps://openreview.net/forum?id=68n2s9ZJWF8 29

    2022

  11. [19]

    Zhang, C

    H. Zhang, C. Xiao, C. Gao, H. Wang, M. Müller, et al., Exploiting the replay memory before exploring the environment: enhancing rein- forcement learning through empirical mdp iteration, Advances in Neural Information Processing Systems 37 (2024) 85658–85692

    2024

  12. [20]

    Fujimoto, S

    S. Fujimoto, S. S. Gu, A minimalist approach to offline reinforcement learning, Advances in neural information processing systems 34 (2021) 20132–20145

    2021

  13. [21]

    J. Wu, H. Wu, Z. Qiu, J. Wang, M. Long, Supported policy optimiza- tion for offline reinforcement learning, Advances in Neural Information Processing Systems 35 (2022) 31278–31291

    2022

  14. [22]

    Zhang, C

    H. Zhang, C. Xiao, H. Wang, J. Jin, bo xu, M. Müller, Replay memory as an empirical MDP: Combining conservative estimation with expe- rience replay, in: The Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=SjzFVSJUt8S

    2023

  15. [23]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, S. Levine, Conservative Q-learning for offline reinforcement learning, Advances in Neural Information Process- ing Systems 33 (2020) 1179–1191

    2020

  16. [24]

    H. Liu, P. Abbeel, Emergent agentic transformer from chain of hindsight experience, in: International Conference on Machine Learning, PMLR, 2023, pp. 21362–21374

    2023

  17. [25]

    Y.-H. Wu, X. Wang, M. Hamaya, Elastic decision transformer, in: Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, 2023, pp. 18532–18550

    2023

  18. [26]

    Chebotar, Q

    Y. Chebotar, Q. Vuong, K. Hausman, F. Xia, Y. Lu, A. Irpan, A. Ku- mar, T. Yu, A. Herzog, K. Pertsch, et al., Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions, in: Conference on Robot Learning, PMLR, 2023, pp. 3909–3928

    2023

  19. [27]

    P. Wu, A. Majumdar, K. Stone, Y. Lin, I. Mordatch, P. Abbeel, A. Ra- jeswaran, Masked trajectory models for prediction, representation, and control, in: International Conference on Machine Learning, PMLR, 2023, pp. 37607–37623. 30

    2023

  20. [28]

    Yamagata, A

    T. Yamagata, A. Khalil, R. Santos-Rodriguez, Q-learning decision trans- former: Leveraging dynamic programming for conditional sequence modelling in offline rl, in: International Conference on Machine Learn- ing, PMLR, 2023, pp. 38989–39007

    2023

  21. [29]

    C.-X. Gao, C. Wu, M. Cao, R. Kong, Z. Zhang, Y. Yu, Act: empow- ering decision transformer with dynamic programming via advantage conditioning, in: Proceedings of the AAAI Conference on Artificial In- telligence, Vol. 38, 2024, pp. 12127–12135

    2024

  22. [30]

    Y. Wang, C. Yang, Y. Wen, Y. Liu, Y. Qiao, Critic-guided decision transformer for offline reinforcement learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, 2024, pp. 15706– 15714

    2024

  23. [31]

    J. Kim, S. Lee, W. Kim, Y. Sung, Adaptiveq-aid for conditional su- pervised learning in offline reinforcement learning, Advances in Neural Information Processing Systems 37 (2024) 87104–87135

    2024

  24. [32]

    Tanaka, K

    T. Tanaka, K. Abe, K. Ariu, T. Morimura, E. Simo-Serra, Return- aligned decision transformer, arXiv preprint arXiv:2402.03923 (2024)

    2024 arXiv

  25. [33]

    K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick, Masked autoen- coders are scalable vision learners, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16000– 16009

    2022

  26. [34]

    X. Li, Y. Zheng, H. Chen, X. Chen, Y. Liang, C. Lai, B. Li, X. Xue, Instruction-guided fusion of multi-layer visual features in large vision-language models, Pattern Recognition 170 (2026) 111932. doi:https://doi.org/10.1016/j.patcog.2025.111932

    2026

  27. [35]

    J. Xia, X. Zhu, B. Jiang, S. Kan, Reasoning elicitation and multi- granularity contrastive learning for text-rich image understanding in large vision-language models, Pattern Recognition 171 (2026) 112278. doi:https://doi.org/10.1016/j.patcog.2025.112278

    2026

  28. [36]

    Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017)

    A. Vaswani, Attention is all you need, Advances in Neural Information Processing Systems (2017). 31

    2017

  29. [37]

    W. K. Newey, J. L. Powell, Asymmetric least squares estimation and testing, Econometrica: Journal of the Econometric Society (1987) 819– 847

    1987

  30. [38]

    Huang, Y

    A. Huang, Y. Wang, R. Liu, H. Zou, X. Zhou, Qvf: Incorpo- rating quantile value function factorization into cooperative multi- agent reinforcement learning, Pattern Recognition 161 (2025) 111323. doi:https://doi.org/10.1016/j.patcog.2024.111323

    2025

  31. [39]

    Zheng, A

    Q. Zheng, A. Zhang, A. Grover, Online decision transformer, in: inter- national conference on machine learning, PMLR, 2022, pp. 27042–27059

    2022

  32. [40]

    Tarasov, A

    D. Tarasov, A. Nikulin, D. Akimov, V. Kurenkov, S. Kolesnikov, Corl: Research-oriented deep offline reinforcement learning library, Advances in Neural Information Processing Systems 36 (2024). 32

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.