{"id":"3a466c92-aa32-46df-95eb-9afefaa53545","arxiv_id":"2607.18910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A utility-augmented Transformer that conditions attention projections on action–reward history outperforms feedback-blind baselines on non-stationary decision tasks.","lead":"This paper introduces a Transformer variant whose attention queries, keys, and values are modulated by a learned summary of past actions and rewards, so retrieval can respond to feedback within an episode. It reports gains over observation-only and input-level baselines on four non-stationary decision benchmarks, plus theoretical expressiveness results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's proof has an invalid equivalence step: UAT's logits contain feedback-dependent x_t-x_j coupling that a standard Transformer on augmented tokens cannot reproduce.","rationale":"The reader's CONDITIONAL verdict is appropriate, and I agree with their note that the fixed-gating equivalence step in Theorem 2 is problematic. However, I do not think Assumption 2 ('rich utility encoding') is the most load-bearing assumption: Theorem 1 only needs to distinguish one pair of action-reward histories, which a small MLP utility encoder can do. The sharper, more central gap is in Appendix A.3 Step 3, where the proof claims UAT with fixed gating is exactly a standard Transformer on augmented tokens. The UAT attention logit contains higher-order coupling terms between observations and feedback gates (e.g., rho_t g_j x_t x_j in the scalar case) that no fixed quadratic-form attention on [x; g] tokens can express. Since Assumption 3 is then inapplicable, the uniform-approximation theorem is not established by the given proof. This is an internal inconsistency in the proof rather than a disagreement with external consensus. The empirical results are suggestive, and the boundary/ablation analyses are useful, but they do not repair the missing theorem. A revision should either replace Assumption 3 with a universality statement over tokens that include the necessary interaction terms and prove the equivalence, or provide a direct approximation argument for UAT. Because a corrected proof is plausibly within reach, I would keep the reader's CONDITIONAL verdict rather than escalate to REJECT.","tokens_in":26480,"tokens_out":15433,"duration_ms":141990,"concrete_test":"Independently verify Step 3 in the minimal scalar case: set d = dm = 1, one head and one layer, W_Q = W_K = 0, W_Q^u = W_K^u = 1, U = 0, and compare the UAT attention logit rho_t g_j x_t x_j against the most general standard-Transformer logit on augmented tokens [x_t; rho_t] and [x_j; g_j], namely a x_t x_j + b x_t g_j + c rho_t x_j + d rho_t g_j. Show that no (a,b,c,d) matches for all scalar inputs. If the authors instead propose richer augmented tokens (e.g., including pairwise products), check that Assumption 3 is restated with those tokens and that the full multi-layer UAT logit is exactly reproduced before invoking universality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim rests on Theorem 2 (Appendix A.3). Step 3 of its proof asserts that for any fixed gating configuration (g, rho), UAT is 'exactly equivalent' to a standard Transformer on augmented tokens tilde{x}_j = [x_j; g_j], so Assumption 3's universality can be inherited. This equivalence is false. The UAT attention logit (Eq. 23 in Appendix B) includes terms such as x_t^T W_Q^T diag(g_j) W_K^u x_j, and with the bias terms also rho_t^T U_Q^T diag(g_j) W_K^u x_j. Even in the scalar case d = dm = 1, one head/layer, with W_Q = W_K = 0, W_Q^u = W_K^u = 1, U = 0, the UAT logit is rho_t * g_j * x_t * x_j, a degree-4 monomial. A standard Transformer on augmented query [x_t; rho_t] and key [x_j; g_j] has logit a x_t x_j + b x_t g_j + c rho_t x_j + d rho_t g_j, a fixed quadratic form; no choice of constants reproduces rho_t g_j x_t x_j for all inputs. More generally, UAT's coefficient on the observation-observation product x_t^T M x_j is W_Q^T (W_K + diag(g_j) W_K^u) plus rho_t-dependent terms, so it varies with feedback gates, but a fixed bilinear form on augmented tokens cannot vary this way. Therefore Assumption 3 cannot be applied, and Theorem 2's uniform approximation claim is unproven. This is load-bearing because Theorem 2 is the paper's formal basis for claiming UAT 'can uniformly approximate feedback-dependent decision maps.' By contrast, the reader's identified Assumption 2 is not the weakest point: Theorem 1's construction only needs to separate a single pair of histories, which a small MLP utility encoder can do.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a structural limitation it calls feedback-blind retrieval: Transformer attention kernels driven only by observation features cannot distinguish histories with identical observations but different action–reward outcomes. To address this, it proposes the Utility-Augmented Transformer (UAT), which conditions query/key/value projections on a compact utility state derived from shifted action–reward pairs. The paper claims two theoretical results: a strict representational hierarchy over observation-only Transformers (Theorem 1) and a uniform approximation theorem for feedback-dependent decision maps (Theorem 2), the latter based on an inheritance argument from standard Transformer universality. Empirically, UAT is evaluated on four non-stationary benchmarks—synthetic navigation, sepsis treatment, portfolio allocation, and delayed-feedback recommendation—with ablations, boundary analyses, and mechanism studies, reporting consistent improvements over observation-only, test-time adaptation, and input-level feedback baselines.","tokens_in":27018,"tokens_out":11228,"duration_ms":107182,"significance":"If the theoretical claims were established, the paper would make a useful conceptual and architectural contribution: it isolates a concrete retrieval-level bottleneck in Transformer-based decision models and proposes a simple, computationally cheap modification with a graceful degradation property. The empirical study is broad, with multiple benchmarks, ablations, boundary analyses, and interpretability experiments, and the results are suggestive. However, the central universal-approximation theorem currently rests on an invalid proof step, and the strict-hierarchy theorem depends on a strong, essentially existential assumption. The paper's significance is therefore conditional on substantial repair of the theory; the empirical findings alone are still of interest but do not by themselves support the paper's strongest claims.","major_comments":[{"comment":"The step claiming that UAT with fixed gating is 'exactly equivalent' to a standard Transformer on augmented tokens [x_j; g_j] is false. The UAT attention logit contains the cross term ℓ^(3) = x_t^T (diag(ρ_t) W_Q^u)^T diag(g_j) W_K^u x_j, which is trilinear in (x_t, g_j, x_j). A standard Transformer on tokens [x_t; g_t] and [x_j; g_j] produces only a fixed quadratic form in the coordinates of the two augmented tokens. Even in the scalar case d=dm=1 with W_Q=W_K=0, W_Q^u=W_K^u=1, U=0, the UAT logit is ρ_t g_j x_t x_j, which cannot be matched by any fixed quadratic form over [x_t;g_t] and [x_j;g_j] as the inputs vary. Thus Assumption 3 cannot be inherited in the manner stated, and Theorem 2's uniform approximation claim is unproven. This is load-bearing for the paper's central theoretical contribution.","section":"Appendix A.3, Theorem 2, Step 3; Appendix B, Eq. (23)"},{"comment":"Even if the fixed-gating equivalence were repaired, the finite-net covering argument does not establish existence of a single finite-depth, finite-width UAT. For each net center, the proof invokes a standard Transformer that approximates F* on that cell; the resulting approximators are in general different networks. UAT has fixed weights and its attention is a smooth function of the gates; the proof does not show how one UAT realizes, or even approximates, all of these cell-dependent transformers simultaneously. The triangle-inequality error accounting therefore does not produce the claimed single UAT network.","section":"Appendix A.3, Theorem 2, Steps 4–5"},{"comment":"The strict separation construction depends on Assumption 2, which asserts that the utility encoder and gate readouts can make gating vectors pairwise distinct for any finite set of action–reward pairs. This is close to assuming the desired conclusion: it essentially posits that feedback can be distinctly encoded in the gates. The concrete instantiation—a bias-free two-layer MLP with tanh readouts—is not shown to satisfy this assumption for arbitrary finite sets, and the proof also implicitly requires that the observation embeddings x_t and x_{j*+1} are nonzero so that the selected rows of W_Q and W_K^u can yield nonzero products. These conditions are not derived from Assumption 1. The theorem is therefore conditional on assumptions that are neither verified nor derived from the architecture; this weakens the strict-hierarchy claim as stated.","section":"Appendix A.3, Theorem 1, Part 2; Assumption 2"}],"minor_comments":[{"comment":"Several entries report zero or near-zero standard deviations (e.g., A-DT 0.866±0.000 in Domain A). With only three seeds, such exact values are unusual and deserve an explanation; they may indicate rounding or a deterministic component in evaluation.","section":"Section 6.2, Table 1"},{"comment":"The text states UAT achieves 'statistically comparable' or 'best' results, but no significance testing or confidence intervals are reported for the main comparisons. Given the high variance in some metrics (e.g., HSI Sharpe for UAT is 0.554±0.123), the reader cannot assess whether the reported gains are reliable.","section":"Section 6.2"},{"comment":"Panel (d) contains garbled axis labels ('VT hT 2', 'UAT T 2'). These appear to denote squared norms ∥h_T∥² and ∥ρ_T∥² but should be typeset correctly for clarity.","section":"Figure 3"},{"comment":"The description of d=0 as 'immediate access to r_t from the latest completed logged interaction' is slightly confusing in a delayed-feedback setting; clarifying that this is zero delay relative to the logged feedback channel, not access to feedback for the supervised target, would avoid ambiguity.","section":"Appendix D.2, Domain D boundary analysis"},{"comment":"The rank-bound statement is correct, but the phrase 'rank-limited, entangled retrieval kernel' and the subsequent claim that depth 'cannot' re-route retrieval directly could be misread as a stronger statement than Proposition 2 proves, since depth can mix channels indirectly. Consider rephrasing to 'no single layer can assign independent pathways.'","section":"Appendix A.1, Proposition 2"}],"recommendation":"major_revision","confidential_remarks":"This is a paper with a valuable empirical contribution and an interesting conceptual framing, but the theoretical centerpiece is currently unproven. The invalid equivalence in Theorem 2's proof is not a minor typo; it is the mechanism by which universality is inherited. The authors should be asked to either repair the proof with a correct argument, or substantially weaken Theorems 1 and 2 to what can actually be shown (e.g., a representational inclusion under explicit constructive conditions). The empirical study itself is solid enough to survive a major revision, provided the theoretical claims are brought in line with the proofs. I would not support acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper identifies a real structural limitation — attention kernels that ignore action-reward feedback — and proposes a clean fix by conditioning QKV projections on a compact utility state. The empirical work is reasonably credible. But the headline universality theorem is not proven: the key equivalence step in the appendix is wrong.\n\nWhat's genuinely new: the feedback-blind retrieval formalization (Definitions 2-3) and the UAT architecture with zero-gate degradation. Theorem 1's strict hierarchy is plausible and its proof sketch is basically sound, if Assumption 2 is weakened to single-pair separation. The experiments span four quite different benchmarks and include thoughtful diagnostics — ablations, boundary analyses, attention maps. The KV-side modulation ablation is informative; the finding that removing key/value modulation hurts most is a nice piece of evidence.\n\nThe serious problem is Theorem 2. In Appendix A.3 Step 3, the authors claim UAT with fixed gating is exactly equivalent to a standard Transformer on augmented tokens [x; g] and [x; ρ]. It isn't. Even in the scalar case with W_Q = W_K = 0, the logit becomes ρ_t g_j x_t x_j, which no fixed bilinear form on the augmented tokens can produce. So Assumption 3 doesn't apply, and the uniform approximation result is unproven as written. This is load-bearing. The reader's worry about Assumption 2 is less acute: for Theorem 1 you only need to separate one pair of histories, which a small MLP encoder can do; the assumption is over-stated but not the chokepoint. Two smaller issues: no code or data release, and only three seeds with no significance tests on the financial results that have wide error bars. Also, a few baselines (e.g., IQL) are used under a BC protocol that may not match their design, though the main comparisons to DT-style/A-DT are apples-to-apples.\n\nThis paper deserves a serious referee, but the universality claim should be either fixed or downgraded. If the authors can patch Theorem 2 (or replace it with a direct approximation argument) and release the code, the paper would be a solid contribution. For now, treat the theory as provisional; the empirical story is suggestive but not definitive without significance tests and code. I'd bring it to our reading group, cite the architecture and feedback-blindness framing, and would peer review it — but with a heavy-revision request.","headline":"A useful architectural idea and a real formalization of feedback-blindness, but the headline universality proof has a concrete gap and the empirics lack significance tests and code.","tokens_in":27408,"tokens_out":7101,"would_cite":true,"duration_ms":61037,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that attention driven only by observation similarity cannot distinguish histories that share observations but differ in action–reward feedback, and introduces a utility-conditioned Transformer that can.","keywords":["sequential decision making","non-stationary POMDP","feedback-blind retrieval","utility-conditioned attention","Transformer policies","action-reward feedback","regime shift","behavioral cloning"],"falsifier":"On a feedback-informative task (e.g., Domain A with a hidden goal shift), construct two histories that share the observation sequence but differ in the action–reward outcomes, then feed both to a trained UAT and compare the produced action distributions; if they are identical, the forward-pass discrimination that the paper claims is absent. A more direct check of Theorem 1: verify that a utility encoder of the paper's size can actually produce pairwise distinct gating vectors for those two outcomes—if not, the strict-separation proof's load-bearing assumption fails.","tokens_in":26418,"feed_emoji":"🎯","tokens_out":5402,"duration_ms":47853,"temperature":0.7,"pith_summary":"This paper argues that Transformer policies for sequential decisions have a structural blind spot: attention retrieves context by observation similarity only, so two histories with identical observations but different past action–reward outcomes get the same retrieval state. In non-stationary, partially observable environments, that blindness can force suboptimal actions because the regime is often identifiable only through reward feedback. To fix it, the authors introduce the Utility-Augmented Transformer (UAT), which routes a compact utility state—derived from the most recent action–reward pair—into the query, key, and value projections, letting feedback reshape retrieval inside the forward pass. They prove UAT strictly enlarges the observation-only Transformer function class and can uniformly approximate feedback-dependent Lipschitz decision maps, and they show on four benchmarks that it beats observation-only, test-time adaptation, and input-level feedback baselines, with the largest gains in noisier regimes.","feed_headline":"Reward-aware retrieval provably beats feedback-blind Transformers","feed_subtitle":"When identical observations hide different regimes, conditioning attention on action–reward feedback picks the right action.","key_machinery":"The load-bearing mechanism is the utility-conditioned attention kernel. A small utility encoder maps the most recent action–reward pair to a compact vector u_j; token-level gates g_j and a smoothed regime gate ρ_t (an exponential moving average of the utility states) modulate the query, key, and value projections through diagonal multiplicative terms and additive offsets. This creates a direct, disentangled feedback pathway in the attention logit—four interaction terms (obs–obs, obs–fb, fb–obs, fb–fb)—where feedback can re-rank keys and reshape queries instead of being entangled through a single rank-limited bilinear form as in input-level injection. The zero-gate initialization makes UAT st","core_discovery":"The paper's central claim is that feedback-blind retrieval—attention whose queries, keys, and values depend only on observations—is a structural bottleneck. On feedback-informative tasks, where observation-equivalent histories with different action–reward outcomes demand different optimal actions, no observation-only causal Transformer can represent the correct policy, regardless of training. UAT removes this bottleneck by conditioning the projections on a utility state u_j = E(f_a(a_{j-1}), r_{j-1}) that is embedded from the shifted action–reward pair and propagated through per-token gates and a smoothed regime gate. Under finite-horizon compactness and Lipschitz assumptions, the paper prov","pith_inferences":["An editorial extension: the same feedback-blindness argument applies to any policy whose retrieval kernel is fixed during inference, so utility-conditioned projections could be bolted onto state-space, recurrent, or external-memory agents that face regime shifts.","A testable consequence the paper leaves implicit: on tasks where observations fully determine optimal actions, UAT should reduce to vanilla performance; the reported degradation under null and shuffled feedback is consistent with that, and one could formalize it by matching capacity across models.","Theorem 1's strict separation rests on Assumption 2 (the encoder can separate finitely many action–reward pairs into distinct gates). If that assumption fails, the hierarchy proof does not go through even though the empirical method might still help—so an independent check of that assumption on the benchmark tasks would clarify how much of the theoretical result transfers.","The mechanism plots suggest adaptation speed is gated by the utility memory decay λ; a direct extension would sweep λ against the number of steps needed to recover pre-shift accuracy after a switch, which the sepsis diagnostic already measures at τ≤5 for the default setting."],"forward_implications":["On any feedback-informative task, no observation-only causal Transformer can represent the optimal policy, no matter how it is trained, because its frozen attention cannot distinguish histories that differ only in action–reward outcomes.","Concatenating action–reward tokens into the input does not fully solve the problem: the attention logit then mixes observation and feedback through one rank-limited bilinear form, so feedback cannot directly re-route retrieval.","A UAT with finite depth and width can uniformly approximate any continuous Lipschitz decision map that depends on action–reward feedback (Theorem 2), while a zero-gate configuration recovers the Vanilla Transformer exactly when feedback is uninformative.","On the four studied benchmarks, UAT improves expert-action accuracy, Sharpe ratio, and ranking metrics over observation-only, test-time adaptation, and input-level feedback baselines, and the advantage grows in noisier regimes.","The reward-inversion experiment suggests UAT uses feedback structure—changes and patterns—rather than absolute reward polarity."],"fun_headline_variants":["Feedback-blind attention fails; utility-aware retrieval wins","Utility-augmented transformer tackles feedback-blindness","Conditioning attention on rewards fixes feedback-blind retrieval","Reward-driven attention beats feedback-blind transformers","Utility gating on rewards unlocks hidden regimes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The strict-separation theorem leans on Assumption 2 (Appendix A.2): for any finite set of distinct action–reward pairs, the utility encoder and gate readouts can be parameterized to make the induced gating vectors pairwise distinct; if the encoder cannot separate those pairs, the proof that UAT strictly enlarges the observation-only class no longer goes through.","fun_headline_variants_meta":{"raw":{"variants":["Feedback-blind attention fails; utility-aware retrieval wins","Utility-augmented transformer tackles feedback-blindness","Conditioning attention on rewards fixes feedback-blind retrieval","Reward-driven attention beats feedback-blind transformers","Utility gating on rewards unlocks hidden regimes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001496,"raw_usage":{"total_tokens":5858,"prompt_tokens":781,"completion_tokens":5077,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":5018}},"tokens_in":525,"tokens_out":5077,"duration_ms":31258,"temperature":1.0,"reasoning_tokens":5018,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T13:58:27.355685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a feedback-informative task (e.g., Domain A with a hidden goal shift), construct two histories that share the observation sequence but differ in the action–reward outcomes, then feed both to a trained UAT and compare the produced action distributions; if they are identical, the forward-pass discrimination that the paper claims is absent. A more direct check of Theorem 1: verify that a utility encoder of the paper's size can actually produce pairwise distinct gating vectors for those two outcomes—if not, the strict-separation proof's load-bearing assumption fails.","supporting_citations":[],"review_version":1}