Pith. sign in

REVIEW 3 major objections 5 minor 68 references

Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that attention driven only by observation similarity cannot distinguish histories that share observations but differ in action–reward feedback, and introduces a utility-conditioned Transformer that can.

desk verdict A useful architectural idea and a real formalization of feedback-blindness, but the headline universality proof has a concrete gap and the empirics lack significance tests and code. read the letter →

arxiv 2607.18910 v1 pith:HH6FGX25 submitted 2026-07-21 cs.LG

Yuyang Shen , Shan Dai , Daimin Chen This is my paper
classification cs.LG
keywords sequentialdecisionmakingnon-stationaryPOMDPfeedback-blindretrievalutility-conditionedattentionTransformerpoliciesaction-rewardfeedbackregimeshiftbehavioralcloning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that Transformer policies for sequential decisions have a structural blind spot: attention retrieves context by observation similarity only, so two histories with identical observations but different past action–reward outcomes get the same retrieval state. In non-stationary, partially observable environments, that blindness can force suboptimal actions because the regime is often identifiable only through reward feedback. To fix it, the authors introduce the Utility-Augmented Transformer (UAT), which routes a compact utility state—derived from the most recent action–reward pair—into the query, key, and value projections, letting feedback reshape retrieval inside the forward pass. They prove UAT strictly enlarges the observation-only Transformer function class and can uniformly approximate feedback-dependent Lipschitz decision maps, and they show on four benchmarks that it beats observation-only, test-time adaptation, and input-level feedback baselines, with the largest gains in noisier regimes.

What carries the argument

The load-bearing mechanism is the utility-conditioned attention kernel. A small utility encoder maps the most recent action–reward pair to a compact vector u_j; token-level gates g_j and a smoothed regime gate ρ_t (an exponential moving average of the utility states) modulate the query, key, and value projections through diagonal multiplicative terms and additive offsets. This creates a direct, disentangled feedback pathway in the attention logit—four interaction terms (obs–obs, obs–fb, fb–obs, fb–fb)—where feedback can re-rank keys and reshape queries instead of being entangled through a single rank-limited bilinear form as in input-level injection. The zero-gate initialization makes UAT st

What would settle it

On a feedback-informative task (e.g., Domain A with a hidden goal shift), construct two histories that share the observation sequence but differ in the action–reward outcomes, then feed both to a trained UAT and compare the produced action distributions; if they are identical, the forward-pass discrimination that the paper claims is absent. A more direct check of Theorem 1: verify that a utility encoder of the paper's size can actually produce pairwise distinct gating vectors for those two outcomes—if not, the strict-separation proof's load-bearing assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that feedback-blind retrieval—attention whose queries, keys, and values depend only on observations—is a structural bottleneck. On feedback-informative tasks, where observation-equivalent histories with different action–reward outcomes demand different optimal actions, no observation-only causal Transformer can represent the correct policy, regardless of training. UAT removes this bottleneck by conditioning the projections on a utility state u_j = E(f_a(a_{j-1}), r_{j-1}) that is embedded from the shifted action–reward pair and propagated through per-token gates and a smoothed regime gate. Under finite-horizon compactness and Lipschitz assumptions, the paper prov

Load-bearing premise

The strict-separation theorem leans on Assumption 2 (Appendix A.2): for any finite set of distinct action–reward pairs, the utility encoder and gate readouts can be parameterized to make the induced gating vectors pairwise distinct; if the encoder cannot separate those pairs, the proof that UAT strictly enlarges the observation-only class no longer goes through.

Editorial extensions

If this is right

  • On any feedback-informative task, no observation-only causal Transformer can represent the optimal policy, no matter how it is trained, because its frozen attention cannot distinguish histories that differ only in action–reward outcomes.
  • Concatenating action–reward tokens into the input does not fully solve the problem: the attention logit then mixes observation and feedback through one rank-limited bilinear form, so feedback cannot directly re-route retrieval.
  • A UAT with finite depth and width can uniformly approximate any continuous Lipschitz decision map that depends on action–reward feedback (Theorem 2), while a zero-gate configuration recovers the Vanilla Transformer exactly when feedback is uninformative.
  • On the four studied benchmarks, UAT improves expert-action accuracy, Sharpe ratio, and ranking metrics over observation-only, test-time adaptation, and input-level feedback baselines, and the advantage grows in noisier regimes.
  • The reward-inversion experiment suggests UAT uses feedback structure—changes and patterns—rather than absolute reward polarity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial extension: the same feedback-blindness argument applies to any policy whose retrieval kernel is fixed during inference, so utility-conditioned projections could be bolted onto state-space, recurrent, or external-memory agents that face regime shifts.
  • A testable consequence the paper leaves implicit: on tasks where observations fully determine optimal actions, UAT should reduce to vanilla performance; the reported degradation under null and shuffled feedback is consistent with that, and one could formalize it by matching capacity across models.
  • Theorem 1's strict separation rests on Assumption 2 (the encoder can separate finitely many action–reward pairs into distinct gates). If that assumption fails, the hierarchy proof does not go through even though the empirical method might still help—so an independent check of that assumption on the benchmark tasks would clarify how much of the theoretical result transfers.
  • The mechanism plots suggest adaptation speed is gated by the utility memory decay λ; a direct extension would sweep λ against the number of steps needed to recover pre-shift accuracy after a switch, which the sepsis diagnostic already measures at τ≤5 for the default setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies a structural limitation it calls feedback-blind retrieval: Transformer attention kernels driven only by observation features cannot distinguish histories with identical observations but different action–reward outcomes. To address this, it proposes the Utility-Augmented Transformer (UAT), which conditions query/key/value projections on a compact utility state derived from shifted action–reward pairs. The paper claims two theoretical results: a strict representational hierarchy over observation-only Transformers (Theorem 1) and a uniform approximation theorem for feedback-dependent decision maps (Theorem 2), the latter based on an inheritance argument from standard Transformer universality. Empirically, UAT is evaluated on four non-stationary benchmarks—synthetic navigation, sepsis treatment, portfolio allocation, and delayed-feedback recommendation—with ablations, boundary analyses, and mechanism studies, reporting consistent improvements over observation-only, test-time adaptation, and input-level feedback baselines.

Significance. If the theoretical claims were established, the paper would make a useful conceptual and architectural contribution: it isolates a concrete retrieval-level bottleneck in Transformer-based decision models and proposes a simple, computationally cheap modification with a graceful degradation property. The empirical study is broad, with multiple benchmarks, ablations, boundary analyses, and interpretability experiments, and the results are suggestive. However, the central universal-approximation theorem currently rests on an invalid proof step, and the strict-hierarchy theorem depends on a strong, essentially existential assumption. The paper's significance is therefore conditional on substantial repair of the theory; the empirical findings alone are still of interest but do not by themselves support the paper's strongest claims.

major comments (3)
  1. [Appendix A.3, Theorem 2, Step 3; Appendix B, Eq. (23)] The step claiming that UAT with fixed gating is 'exactly equivalent' to a standard Transformer on augmented tokens [x_j; g_j] is false. The UAT attention logit contains the cross term ℓ^(3) = x_t^T (diag(ρ_t) W_Q^u)^T diag(g_j) W_K^u x_j, which is trilinear in (x_t, g_j, x_j). A standard Transformer on tokens [x_t; g_t] and [x_j; g_j] produces only a fixed quadratic form in the coordinates of the two augmented tokens. Even in the scalar case d=dm=1 with W_Q=W_K=0, W_Q^u=W_K^u=1, U=0, the UAT logit is ρ_t g_j x_t x_j, which cannot be matched by any fixed quadratic form over [x_t;g_t] and [x_j;g_j] as the inputs vary. Thus Assumption 3 cannot be inherited in the manner stated, and Theorem 2's uniform approximation claim is unproven. This is load-bearing for the paper's central theoretical contribution.
  2. [Appendix A.3, Theorem 2, Steps 4–5] Even if the fixed-gating equivalence were repaired, the finite-net covering argument does not establish existence of a single finite-depth, finite-width UAT. For each net center, the proof invokes a standard Transformer that approximates F* on that cell; the resulting approximators are in general different networks. UAT has fixed weights and its attention is a smooth function of the gates; the proof does not show how one UAT realizes, or even approximates, all of these cell-dependent transformers simultaneously. The triangle-inequality error accounting therefore does not produce the claimed single UAT network.
  3. [Appendix A.3, Theorem 1, Part 2; Assumption 2] The strict separation construction depends on Assumption 2, which asserts that the utility encoder and gate readouts can make gating vectors pairwise distinct for any finite set of action–reward pairs. This is close to assuming the desired conclusion: it essentially posits that feedback can be distinctly encoded in the gates. The concrete instantiation—a bias-free two-layer MLP with tanh readouts—is not shown to satisfy this assumption for arbitrary finite sets, and the proof also implicitly requires that the observation embeddings x_t and x_{j*+1} are nonzero so that the selected rows of W_Q and W_K^u can yield nonzero products. These conditions are not derived from Assumption 1. The theorem is therefore conditional on assumptions that are neither verified nor derived from the architecture; this weakens the strict-hierarchy claim as stated.
minor comments (5)
  1. [Section 6.2, Table 1] Several entries report zero or near-zero standard deviations (e.g., A-DT 0.866±0.000 in Domain A). With only three seeds, such exact values are unusual and deserve an explanation; they may indicate rounding or a deterministic component in evaluation.
  2. [Section 6.2] The text states UAT achieves 'statistically comparable' or 'best' results, but no significance testing or confidence intervals are reported for the main comparisons. Given the high variance in some metrics (e.g., HSI Sharpe for UAT is 0.554±0.123), the reader cannot assess whether the reported gains are reliable.
  3. [Figure 3] Panel (d) contains garbled axis labels ('VT hT 2', 'UAT T 2'). These appear to denote squared norms ∥h_T∥² and ∥ρ_T∥² but should be typeset correctly for clarity.
  4. [Appendix D.2, Domain D boundary analysis] The description of d=0 as 'immediate access to r_t from the latest completed logged interaction' is slightly confusing in a delayed-feedback setting; clarifying that this is zero delay relative to the logged feedback channel, not access to feedback for the supervised target, would avoid ambiguity.
  5. [Appendix A.1, Proposition 2] The rank-bound statement is correct, but the phrase 'rank-limited, entangled retrieval kernel' and the subsequent claim that depth 'cannot' re-route retrieval directly could be misread as a stronger statement than Proposition 2 proves, since depth can mix channels indirectly. Consider rephrasing to 'no single layer can assign independent pathways.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: flagged Theorem 2 proof gap is a correctness concern, not a circular reduction.

full rationale

After walking the derivation chain, I find no step where a claimed prediction or first-principles result reduces to its own inputs by construction. Theorem 1's strict separation is a constructive existence proof: Assumption 2 explicitly grants that the utility encoder can separate distinct action–reward pairs, and the proof then chooses utility-affine weight rows to convert that gate difference into an attention-logit difference. This uses the assumption as a premise rather than smuggling in the conclusion. Theorem 2 is candidly presented as a 'universality inheritance' result (Section 5: 'Theorem 2 is phrased as universality inheritance result... transparently stating all assumptions borrowed from the underlying Transformer approximation theory'), and its proof relies on Assumption 3, which assumes universality of standard Transformers on augmented tokens [x_j; g_j]. Even if Step 3 of that proof is mathematically questionable — the UAT logit in Appendix B, Eq. (23) includes a feedback-gated observation–observation term that a fixed bilinear form on augmented tokens cannot in general reproduce — this is an invalid or incomplete proof step, not a circular one: the theorem's input is an assumption about standard Transformers, not about UAT, and the claimed reduction is not an identity by definition. The empirical benchmarks are deliberately feedback-informative, but they are held-out test sets with expert labels, and UAT parameters are trained rather than fitted to the reported metrics; no fitted input is renamed as a prediction. I also find no load-bearing self-citation; all references are to external prior work. The closest self-referential element is the benchmark construction, but that is an experimental-design alignment, not a circular derivation. Hence the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central theoretical claims rest on four explicit assumptions. Assumption 2 is the most ad hoc; it grants the encoder enough power to separate arbitrary action–reward pairs, which is close to the capability the paper is trying to establish. Assumption 3 imports known transformer universality but is applied to an augmented token space that may be too small for exact equivalence. No new physical entities are introduced; the 'utility state' is an internal learned vector.

free parameters (2)
  • EMA decay λ = 0.7/0.8/0.9/0.95 per domain
    Sets memory of smoothed utility state; selected by validation (Fig. 5); not used to fit the theorems.
  • Utility dimension d_u = 16/16/32/16 per domain
    Capacity of the feedback state; selected by validation; sensitivity analysis shows modest effect.
assumptions (4)
  • standard math Assumption 1: finite horizon T, compact observation/action/reward domains
    Needed for uniform continuity and approximation arguments in Appendix A.2/A.3.
  • ad hoc to paper Assumption 2: rich utility encoding — encoders exist to make gating vectors pairwise distinct for any finite set of action–reward pairs
    This is an expressiveness assumption tailored to UAT; it is plausible for wide MLPs but not proved and is load-bearing for Theorem 1.
  • standard math Assumption 3: the chosen standard Transformer family is universal on compact finite-length sequences of augmented tokens [x_j; g_j]
    Imported from transformer approximation theory; the proof applies it to UAT, though the exact-equivalence step is questionable.
  • domain assumption Assumption 4: target decision maps are Lipschitz on the history space
    Regularity condition for the uniform approximation statement.
invented entities (1)
  • Utility state u_j and gates g, ρ
    purpose: Modulate Q/K/V projections so action–reward feedback changes attention retrieval
    Internal learned representations; mechanism analyses in Figures 2–3 show correlation with regimes but no external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making." pith.science (2026). https://pith.science/paper/HH6FGX25

@misc{pith2026260718910,
  author       = {Pith},
  title        = {Pith review of: Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HH6FGX25}},
  note         = {Machine review of arXiv:2607.18910}
}
read the original abstract

Sequential decision making in non-stationary and partially observable environments requires rapid adaptation to latent regime changes. However, existing Transformer decision models face a structural bottleneck in the retrieval mechanism: even when reward is used for training or exposed as an input token, attention retrieval remains primarily driven by observation-derived similarity. We formalize this limitation as feedback-blind retrieval, and formally show that, on feedback-informative tasks, observation-equivalent histories with different action-reward outcomes cannot be distinguished by any observation-only attention, resulting in suboptimal choice. To address this mismatch, we propose the Utility-Augmented Transformer (UAT), a new feedback-conditioned retrieval attention architecture in which a compact utility state modulates the query, key, and value projections, allowing action-reward history to directly alter context retrieval during the forward pass. UAT also enjoys an exact zero-gate degradation property that recovers the Vanilla Transformer when feedback is uninformative. Under finite-horizon compactness and Lipschitz assumptions, we prove that UAT strictly enlarges the observation-only Transformer class and can uniformly approximate feedback-dependent decision maps. Across four non-stationary benchmarks: synthetic navigation with hidden goal shifts, non-stationary sepsis treatment, cross-market portfolio allocation, and delayed-feedback recommendation, UAT consistently improves performance over observation-only, test-time adaptation, and input-level feedback baselines, with particularly large gains in noisier regimes that require stronger adaptation.

Figures

Figures reproduced from arXiv: 2607.18910 by the authors.

Figure 1
Figure 1. Boundary analysis. Left: Domain A Gradual under reward noise/corruption; Domain D [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Cyclic-shift attention on DarkRoom (T=150, two shifts; 32-seed mean). (a) VT recalls pre-shift prototypes. (b) UAT refocuses on the active regime. (c) Base attention (lower) with feedback modulation (upper), strongest post-shift. (d) UAT–VT logit change. relative to calm baseline, indicating that the feedback pathway learns interpretable signals [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Regime gate on HSI-45. (a) ∥ρT ∥ tracks regime changes, while VT ∥hT ∥ remains flat. (b) Within-window ∥ρt∥ separates regimes. (c) Gate magnitude rises with volatility quartile. (d) Shock regimes show the largest increase over the calm baseline. 10 5 0 5 10 15 = t tshift 0.0 0.2 0.4 0.6 0.8 1.0 expert-action acc. (a) accuracy around shift Vanilla TF DT-style A-DT UAT (ours) VT DT A-DT UAT 0.0 0.2 0.4 0.6 0.8 1.0 exp… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Adaptation kinetics on Domain B Sepsis ( [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Sensitivity analysis on Domain B using one-at-a-time sweeps over three seeds. Shaded [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Additional diagnostic metrics. Bars sorted ascending; (a) Domain A navigation efficiency [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 1 canonical work pages

  1. [1]

    Mehdi Afsar, Trafford Crump, and Behrouz Far

    M. Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey.ACM Comput. Surv., 55(7), December 2022. ISSN 0360-0300. doi: 10.1145/3543846. URLhttps://doi.org/10.1145/3543846

    doi:10.1145/3543846 2022

  2. [2]

    Aggarwal, Alexander Hinneburg, and Daniel A

    Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. On the surprising behavior of distance metrics in high dimensional space. In Jan Van den Bussche and Victor Vianu, editors, Database Theory — ICDT 2001, pages 420–434, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. ISBN 978-3-540-44503-6

    2001

  3. [3]

    Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine, 34(6):26–38, 2017

    Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine, 34(6):26–38, 2017. doi: 10.1109/MSP.2017.2743240

    arXiv 2017

  4. [4]

    A framework for behavioural cloning

    Michael Bain and Claude Sammut. A framework for behavioural cloning. In K Furukawa, D Michie, and S Muggleton, editors,Machine Intelligence 15: Intelligent Agents. Oxford University Press, 01 2000. ISBN 9780198538677. doi: 10.1093/oso/9780198538677.003.0006. URLhttps://doi.org/10.1093/oso/9780198538677.003.0006

    arXiv

  5. [5]

    Representation learning: A review and new perspectives.IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8): 1798–1828, 2013

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives.IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8): 1798–1828, 2013. doi: 10.1109/TPAMI.2013.50

    doi:10.1109/tpami.2013.50 2013

  6. [6]

    Modeling delayed feedback in display advertising

    Olivier Chapelle. Modeling delayed feedback in display advertising. InProceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, page 1097–1105, New York, NY , USA, 2014. Association for Computing Machinery. ISBN 9781450329569. doi: 10.1145/2623330.2623634. URL https://doi.org/10.1145/ 2623330.2623634

    arXiv 2014

  7. [7]

    Decision transformer: reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: reinforcement learning via sequence modeling. InProceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY , USA, 2021. Curran Associates Inc. IS...

    2021

  8. [8]

    Reinforcement learning for non- stationary markov decision processes: the blessing of (more) optimism

    Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non- stationary markov decision processes: the blessing of (more) optimism. InProceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    2020

Show all 68 references
  1. [9]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas.Entropy, Relative Entropy, and Mutual Information. John Wiley & Sons, Ltd, 2005. ISBN 9780471748823. doi: https://doi.org/10.1002/047174882X.ch2. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/047174882X.ch2

    2005 doi

  2. [10]

    Transformer-XL: Attentive language models beyond a fixed-length context

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdi- nov. Transformer-XL: Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and Lluís Màrquez, editors,Proceedings of the 57th Annual Meet- ing of the Associ...

    2019 doi

  3. [11]

    Donti, Brandon Amos, and J

    Priya L. Donti, Brandon Amos, and J. Zico Kolter. Task-based end-to-end model learning in stochastic optimization. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 5490–5500, Red Hook, NY , USA, 2017. Curran Associates ...

    2017

  4. [12]

    predict, then optimize

    Adam N. Elmachtoub and Paul Grigas. Smart “predict, then optimize”.Management Science, 68(1):9–26, 2022. doi: 10.1287/mnsc.2020.3922. URL https://doi.org/10.1287/mnsc. 2020.3922

    2022

  5. [13]

    Dai, and Quoc V

    David Ha, Andrew M. Dai, and Quoc V . Le. Hypernetworks. InInternational Conference on Learning Representations, 2017. URLhttps://openreview.net/forum?id=rkpACe1lx. 11

    2017

  6. [14]

    Maxwell Harper and Joseph A

    F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context.ACM Trans. Interact. Intell. Syst., 5(4), December 2015. ISSN 2160-6455. doi: 10.1145/2827872. URLhttps://doi.org/10.1145/2827872

    2015 doi

  7. [15]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. doi: 10.1109/CVPR.2016.90

    2016 doi

  8. [16]

    Neural collaborative filtering

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. InProceedings of the 26th International Conference on World Wide Web, WWW ’17, page 173–182, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Co...

    2017

  9. [17]

    Distilling the knowledge in a neural network,

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network,

  10. [18]

    Multilayer feedforward networks are universal approximators.Neural Networks, 2(5):359–366, 1989

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators.Neural Networks, 2(5):359–366, 1989. ISSN 0893-6080. doi: https://doi.org/10.1016/0893-6080(89)90020-8. URL https://www.sciencedirect.com/ science/article/pii/08936...

    1989

  11. [19]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In2017 IEEE International Conference on Computer Vision (ICCV), pages 1510–1519, 2017. doi: 10.1109/ICCV .2017.167

    2017 doi

  12. [20]

    Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Moham- mad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, a freely accessible critical care database.Scientific Data, 3:160035, 2016. doi: 10.1038/sdat...

    2016 doi

  13. [21]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains.Artificial Intelligence, 101(1):99–134,

  14. [22]

    Self-attentive sequential recommendation

    Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In2018 IEEE International Conference on Data Mining (ICDM), pages 197–206, 2018. doi: 10.1109/ ICDM.2018.00035

    2018

  15. [23]

    Yakup Kara, Melek Acar Boyacioglu, and Ömer Kaan Baykan. Predicting direction of stock price index movement using artificial neural networks and support vector machines: The sample of the istanbul stock exchange.Expert Systems with Applications, 38(5):5311–5319,

  16. [24]

    Albert King and Yaojie Zhang. Akshare. https://github.com/akfamily/akshare, 2022

    2022

  17. [25]

    Kolm, Reha Tütüncü, and Frank J

    Petter N. Kolm, Reha Tütüncü, and Frank J. Fabozzi. 60 years of portfolio optimiza- tion: Practical challenges and current trends.European Journal of Operational Research, 234(2):356–371, 2014. ISSN 0377-2217. doi: https://doi.org/10.1016/j.ejor.2013.10.060. URL https://www.sc...

    2014 doi

  18. [26]

    Celi, Omar Badawi, Anthony C

    Matthieu Komorowski, Leo A. Celi, Omar Badawi, Anthony C. Gordon, and A. Aldo Faisal. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care.Nature Medicine, 24(11):1716–1720, 2018. doi: 10.1038/s41591-018-0213-5. URL https://doi...

    2018 doi

  19. [27]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. InInternational Conference on Learning Representations, 2022. URL https: //openreview.net/forum?id=68n2s9ZJWF8. 12

    2022

  20. [28]

    In-context reinforcement learning with algorithm distillation

    Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and V olodymyr Mnih. In-context reinforcement learning with algorithm...

  21. [29]

    LeCun, Léon Bottou, Genevieve B

    Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller.Efficient Back- Prop, pages 9–48. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3- 642-35289-8. doi: 10.1007/978-3-642-35289-8_3. URL https://doi.org/10.1007/ 978-3-642-35289-8_3

    2012 doi

  22. [30]

    Robust performance hypothesis testing with the sharpe ratio

    Oliver Ledoit and Michael Wolf. Robust performance hypothesis testing with the sharpe ratio. Journal of Empirical Finance, 15(5):850–859, 2008. ISSN 0927-5398. doi: https://doi.org/10. 1016/j.jempfin.2008.03.002. URL https://www.sciencedirect.com/science/article/ pii/S0927539808000182

    2008

  23. [31]

    Context-aware dynamics model for generalization in model-based reinforcement learning

    Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. InProceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020

    2020

  24. [32]

    End-to-end training of deep visuomotor policies.J

    Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies.J. Mach. Learn. Res., 17(1):1334–1373, January 2016. ISSN 1532-4435

    2016

  25. [33]

    A survey on transformers in reinforcement learning.Transactions on Machine Learning Research,

    Wenzhe Li, Hao Luo, Zichuan Lin, Chongjie Zhang, Zongqing Lu, and Deheng Ye. A survey on transformers in reinforcement learning.Transactions on Machine Learning Research,

  26. [34]

    Decision-focused learning: Foundations, state of the art, benchmark and future opportunities.J

    Jayanta Mandi, James Kotary, Senne Berden, Maxime Mulamba, Victor Bucarey, Tias Guns, and Ferdinando Fioretto. Decision-focused learning: Foundations, state of the art, benchmark and future opportunities.J. Artif. Int. Res., 80, September 2024. ISSN 1076-9757. doi: 10.1613/jai...

    2024 doi

  27. [35]

    Causal transformer for estimating counterfactual outcomes

    Valentyn Melnychuk, Dennis Frauen, and Stefan Feuerriegel. Causal transformer for estimating counterfactual outcomes. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors,Proceedings of the 39th International Conference on Ma-...

    2022

  28. [36]

    Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. InProceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, page 807–814, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077

    2010

  29. [37]

    Ng, Daishi Harada, and Stuart J

    Andrew Y . Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. InProceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, page 278–287, San Francisco, CA, USA, 1999. M...

    1999

  30. [38]

    URL https://openreview.net/forum?id=r30yuDPvf2

    ISSN 2835-8856. URL https://openreview.net/forum?id=r30yuDPvf2. Survey Certification

  31. [39]

    Francis Song, Jack W

    Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M. Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. InP...

    2020

  32. [40]

    Semantic image synthesis with spatially-adaptive normalization

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2332–2341, 2019. doi: 10.1109/CVPR.2019.00244. 13

    2019

  33. [41]

    Film: visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: visual reasoning with a general conditioning layer. InProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelli...

    2018

  34. [42]

    Differentiation of blackbox combinatorial solvers

    Marin Vlastelica Pogan ˇci´c, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinek. Differentiation of blackbox combinatorial solvers. InInternational Conference on Learning Representations, 2020. URLhttps://openreview.net/forum?id=BkevoJSYPB

    2020

  35. [43]

    The econometrics of financial markets.Journal of Empirical Finance, 3(1): 15–102, 1996

    Adrian Pagan. The econometrics of financial markets.Journal of Empirical Finance, 3(1): 15–102, 1996. ISSN 0927-5398. doi: https://doi.org/10.1016/0927-5398(95)00020-8. URL https://www.sciencedirect.com/science/article/pii/0927539895000208

    1996

  36. [44]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors,Proceedings of the Fourteenth International Conference on Artificial Intell...

    2011

  37. [45]

    McGraw-Hill, New York, 3 edition, 1976

    Walter Rudin.Principles of Mathematical Analysis. McGraw-Hill, New York, 3 edition, 1976

    1976

  38. [46]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors.Nature, 323:533–536, 1986. doi: 10.1038/323533a0. URL https://doi.org/10.1038/323533a0

    1986 doi

  39. [47]

    Retrieval-augmented decision transformer: External memory for in-context rl

    Thomas Schmied, Fabian Paischer, Vihang Patil, Markus Hofmarcher, Razvan Pascanu, and Sepp Hochreiter. Retrieval-augmented decision transformer: External memory for in-context rl. ArXiv, 2024. URLhttps://arxiv.org/abs/2410.07071

    2024 arXiv

  40. [48]

    Pomerleau

    Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. Touretzky, editor,Advances in Neural Information Processing Systems, volume 1. Morgan- Kaufmann, 1988. URL https://proceedings.neurips.cc/paper_files/paper/1988/ file/812b4ba287f5ee0bc9d43bbf5bbe8...

    1988

  41. [49]

    C. E. Shannon. A mathematical theory of communication.The Bell System Technical Journal, 27(3):379–423, 1948. doi: 10.1002/j.1538-7305.1948.tb01338.x

    1948

  42. [50]

    Bert4rec: Se- quential recommendation with bidirectional encoder representations from transformer

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Se- quential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM...

    2019

  43. [51]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249

    2018

  44. [52]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Ho...

    2017

  45. [53]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URLhttps://arxiv.org/abs/1707.06347

    2017 arXiv

  46. [54]

    Continual test-time domain adaptation

    Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7191–7201, 2022. doi: 10.1109/CVPR52688.2022.00706. 14

    2022

  47. [55]

    Zhicheng Wang, Biwei Huang, Shikui Tu, Kun Zhang, and Lei Xu. Deeptrader: A deep reinforcement learning approach for risk-return balanced portfolio management with market conditions embedding.Proceedings of the AAAI Conference on Artificial Intelligence, 35(1): 643–650, May 2021

    2021

  48. [56]

    Melding the data-decisions pipeline: decision- focused learning for combinatorial optimization

    Bryan Wilder, Bistra Dilkina, and Milind Tambe. Melding the data-decisions pipeline: decision- focused learning for combinatorial optimization. InProceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial I...

    2019

  49. [57]

    Elastic decision transformer

    Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 18532–18550. Curran Associates, Inc., 2023. URL htt...

    2023

  50. [58]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum?id=uXl3bZLkr3c

    2021

  51. [59]

    Yunan Ye, Hengzhi Pei, Boxin Wang, Pin-Yu Chen, Yada Zhu, Ju Xiao, and Bo Li. Reinforcement-learning based portfolio management with augmented asset movement pre- diction states.Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):1112– 1119, Apr. 2020. doi: 1...

    2020 doi

  52. [60]

    Are transformers universal approximators of sequence-to-sequence functions? InInternational Conference on Learning Representations, 2020

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? InInternational Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=ByxRM0Ntvr

    2020

  53. [61]

    A survey of autonomous driving from a deep learning perspective.ACM Comput

    Jingyuan Zhao, Yuyan Wu, Rui Deng, Susu Xu, Jinpeng Gao, and Andrew Burke. A survey of autonomous driving from a deep learning perspective.ACM Comput. Surv., 57(10), May 2025. ISSN 0360-0300. doi: 10.1145/3729420. URLhttps://doi.org/10.1145/3729420

    2025 doi

  54. [62]

    Online decision transformer

    Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 ofPro- ceedings o...

    2022

  55. [63]

    Download historical data in Yahoo Finance

    Yahoo Finance. Download historical data in Yahoo Finance. https://help.yahoo.com/kb/ SLN2311.html, 2026. Accessed April 2026

    2026

  56. [68]

    UE & regime lr

    Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: variational bayes-adaptive deep rl via meta-learning.J. Mach. Learn. Res., 22(1), January 2021. ISSN 1532-4435. 15 Appendix A Proof ...

    2021

  57. [1998]

    doi: https://doi.org/10.1016/S0004-3702(98)00023-X

    ISSN 0004-3702. doi: https://doi.org/10.1016/S0004-3702(98)00023-X. URL https://www.sciencedirect.com/science/article/pii/S000437029800023X

    doi

  58. [2011]

    doi: https://doi.org/10.1016/j.eswa.2010.10.027

    ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa.2010.10.027. URL https://www. sciencedirect.com/science/article/pii/S0957417410011711

    2010 doi

  59. [2015]

    URLhttps://arxiv.org/abs/1503.02531

    arXiv

  60. [2023]

    URLhttps://openreview.net/forum?id=hy0a5MMPUv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.