{"id":"42bec1d5-0fe9-4951-b6bc-b85555d527a5","arxiv_id":"2509.11259","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A pre-trained tabular regression transformer (TabPFN) can act as a Q-function approximator in fitted Q iteration, matching DQN on CartPole, MountainCar, and Acrobot without gradient updates.","lead":"The paper shows that TabPFN, a transformer trained only on tabular regression tasks, can act as a Q-learning value estimator and solve three classic control games without any gradient updates. A generalist might care because it suggests off-the-shelf regression foundation models could power reinforcement learning, avoiding expensive RL training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward shaping may change the optimal policy, so the claimed match with DQN on the original Gymnasium tasks is unverified.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the hand-crafted shaped rewards are not shown to preserve the original optimal policies. This is more fundamental than the abstract/body PPO/TRPO mismatch or the absence of code and error bars, because it determines whether the empirical comparison is to the claimed benchmark at all. A concrete rerun on the original rewards would settle it. If TabPFN-RL still matches DQN without reward shaping, the concern is resolved and the central claim stands. If not, the claim narrows to shaped MDPs, which is a substantially weaker statement. I keep the reader's CONDITIONAL verdict: the paper should not be rejected outright because the more modest claim (competitive on shaped variants) may be supportable, but the current wording overstates the result until the original-reward evaluation is provided.","tokens_in":12016,"tokens_out":9529,"duration_ms":117516,"concrete_test":"Rerun the complete TabPFN-RL pipeline on the three original unshaped Gymnasium tasks (CartPole-v1: +1 per step until termination; MountainCar-v0 and Acrobot-v1: -1 per step until goal/termination), keeping all other hyperparameters, epsilon schedules, context budgets, and the high-reward gate fixed. If TabPFN-RL no longer matches DQN (or fails to learn), the headline result is conditional on reward shaping. As a secondary analytic check, verify whether each shaped reward is potential-based (F(s,a,s') = gamma*Phi(s') - Phi(s)); for these formulas they are not, so the burden is on the authors to show optimal-policy equivalence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that TabPFN-RL matches or surpasses DQN on CartPole-v1, MountainCar-v0, and Acrobot-v1. But the Appendix states 'we employed different reward shaping methods for each environment' and gives concrete formulas: for Acrobot, r_shaped = -cos(theta1) - cos(theta1+theta2); for MountainCar, r_shaped = (x - x_min)/(-x_min + x_max) + 10|v| - 1; for CartPole, r_shaped = 2 - |x|/x_max - |theta|/theta_max. These are not potential-based reward transformations (there is no Phi such that the shaped reward equals gamma*Phi(s') - Phi(s)), so the optimal policy of the shaped MDP is not guaranteed to coincide with the optimal policy of the original Gymnasium tasks, whose rewards are sparse (-1 per step until goal/termination, +1 per step for CartPole). If the shaped MDP has a different optimal policy, TabPFN-RL is solving a modified task, and the comparison to DQN on the purported benchmark is invalid. The paper's Discussion acknowledges 'the current implementation relies on reward shaping,' but no proof and no experiment establishes policy equivalence. This is load-bearing because the headline inference about TabPFN's RL capability depends on the evaluation task being the standard benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TabPFN-RL, a gradient-free deep RL method that repurposes the TabPFN transformer as a Q-function approximator in a fitted Q iteration (FQI) style. After an initial random transition batch, the algorithm fits Q-values through TabPFN's in-context inference, then performs epsilon-greedy online collection while retaining only high-reward episodes via a percentile gate. The authors report competitive or superior performance relative to DQN on CartPole-v1, MountainCar-v0, and Acrobot-v1, and propose several context-truncation heuristics for continual learning once the fixed context budget is full. The paper also discusses why TabPFN's i.i.d. tabular prior is violated by RL data yet still appears to work.","tokens_in":12374,"tokens_out":3260,"duration_ms":44001,"significance":"If the empirical claim is substantiated, the result is significant: it would demonstrate that a transformer pre-trained on i.i.d. tabular regression tasks can serve as a Q-function approximator for online RL without any gradient updates or task-specific fine-tuning. The framing of FQI as an in-context regression problem is a clean idea, and the explicit discussion of context-budget limits is a useful contribution. The paper is also honest about the prior mismatch between TabPFN's training distribution and RL settings. However, the current evidence is not sufficient to support the headline claims: the quantitative results are only shown qualitatively, the reward shaping changes the tasks in ways that are not justified, and the abstract claims comparisons to PPO/TRPO that do not appear in the body.","major_comments":[{"comment":"The central claim that TabPFN-RL 'matches or surpasses' DQN is not supported by the evidence presented. The text states that 10 seeds were run, but Figure 1 shows only single learning curves with no error bars, confidence intervals, or summary statistics. There is no table of final mean returns, standard deviations, or win rates. Without quantitative comparisons, the reader cannot assess whether the apparent closeness to DQN is meaningful. Moreover, the abstract claims comparisons with PPO and TRPO, but no PPO/TRPO experiments appear anywhere in the paper. This is a load-bearing issue because the paper's main contribution is empirical.","section":"Results, Figure 1"},{"comment":"The shaped reward functions used for each environment are not potential-based, so there is no guarantee that the optimal policy of the shaped MDP coincides with that of the original Gymnasium task. For example, Acrobot's r_shaped = -cos(theta1)-cos(theta1+theta2) and MountainCar's r_shaped = (x-x_min)/(-x_min+x_max)+10|v|-1 change the reward landscape. If DQN is also trained on these shaped rewards (as the appendix says), the comparison is internally fair, but then the paper is not evaluating the standard Gymnasium benchmark, contrary to the title and text. The authors need to either prove policy invariance (e.g., via potential-based shaping) or rerun experiments on the original sparse-reward tasks and report both results.","section":"Appendix: Reward Shaping; Method"},{"comment":"The claim of 'no extensive hyperparameter tuning' is not supported. The algorithm has several free parameters per environment: initial epsilon, epsilon decay, context budget B, initial random transitions N, FQI iterations, reward shaping constants, and the episode-gate percentile. The appendix reports different epsilon and B values for each environment, and the ablation study in Figure 3 selects the best values on MountainCar. This is per-environment tuning, not a fixed, off-the-shelf configuration. The paper should clearly report the selection procedure and either provide a fixed configuration that works across tasks or drop the 'no tuning' claim.","section":"Algorithm 1, Eq. (3); Hyperparameters"},{"comment":"The paper states it 'formalizes the intrinsic context size limit of in-context RL algorithms,' but the formal content is limited to the observation that a fixed-budget context cannot grow beyond B. This is a definitional statement rather than a theorem, and no formal analysis of the consequences for learning guarantees is provided. If the theoretical contribution is intended to be a substantive part of the paper, it needs formal statements with assumptions and proofs; otherwise, the wording should be softened to 'discussion' rather than 'formalization.'","section":"Theoretical Limitations"}],"minor_comments":[{"comment":"The abstract mentions PPO and TRPO, but the experiments only include DQN. Either add the missing baselines or remove the claim.","section":"Abstract vs. Body"},{"comment":"The text says context budgets vary 'from 256 to 8192 in multiples of two,8' and later reports the best size as '8096.' This appears to be a typo; the range should likely be 256 to 8192, and the best value should be checked.","section":"Figure 3 caption / text"},{"comment":"In Algorithm 1, line 11 compares R_E to Quantile_0.95(R_hist), but the initial random transitions are not episodes; the definition of R_hist for the first online episodes should be clarified.","section":"Algorithm 1"},{"comment":"The paper claims computational efficiency but reports no runtime or FLOP measurements. Given that TabPFN's inference cost grows quadratically with context, a runtime comparison with DQN would strengthen the practical claims.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is interesting and potentially publishable, but the current version overclaims on two fronts: the DQN comparison is only shown qualitatively, and the abstract promises PPO/TRPO results that are absent. The reward shaping issue is the most serious concern; if the authors can show that their shaped rewards preserve the optimal policies, or if they clearly reframe the contribution as solving shaped variants, the central claim can be repaired. I recommend major revision rather than rejection because the methodological direction is worth pursuing and the deficiencies are fixable with additional experiments and honest reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper reports a genuinely new empirical claim—that a regression-only transformer (TabPFN) can be dropped into Fitted Q Iteration and produce a working Q-function on three classic control tasks without any gradient updates. That is worth taking seriously. But the paper as written does not yet make the case convincingly: the comparison is against DQN on tasks that have been reward-shaped, the abstract claims PPO/TRPO results that never appear, and the empirical support is thin (no error bars, no code). I'd send it to review, but with a clear request for missing statistics and a proper comparison on the original (unshaped) rewards.\n\nWhat's new: Using TabPFN as an in-context regressor inside FQI is a clean idea, and the episode gate (top-5% episodes) plus the context-truncation heuristics are sensible engineering contributions. I also credit the authors for an unusually candid Discussion: they openly state that TabPFN's i.i.d. assumption is violated by bootstrapped targets and non-stationary visitation, and they acknowledge the reward-shaping dependence. That is more honesty than most RL papers show.\n\nSoft spots, in rough order of severity. First, the reward shaping. The authors say they used different shaped rewards per environment. If DQN is evaluated on the same shaped MDP, the comparison is internally fair, but then the headline \"matches DQN on CartPole-v1\" is misleading because the standard benchmark reward is sparse. The stress-test note is right that the shaped rewards are not potential-based, so the optimal policy could differ; the authors provide no argument that they coincide. This needs to be either proven or the claim reframed as 'on a dense-reward variant.' Second, the abstract promises PPO and TRPO comparisons; the body only shows DQN. That mismatch should be fixed. Third, the empirical section has no error bars or confidence intervals, and the figures are described rather than reproduced in the text. Ten seeds is decent; showing means with variance would help. Fourth, they tuned hyperparameters per environment (different epsilons, context sizes, and an ablation to pick B) while claiming 'no extensive hyperparameter tuning.' That is an overreach.\n\nBottom line: the central observation is plausible and worth chasing, but the current paper overstates its support. A serious referee should ask for the original-reward comparison (or a clear caveat), the missing PPO/TRPO results or a corrected abstract, and error bars. With those, this could be a solid empirical note.","headline":"A genuinely new empirical claim—TabPFN as a regression-only Q-function approximator in FQI—but the evidence is too thin and the reward-shaping comparison unverified; worth a serious referee but not acceptance yet.","tokens_in":12805,"tokens_out":3181,"would_cite":false,"duration_ms":37367,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer pre-trained only on tabular regression can drive RL agents without any gradient updates.","keywords":["in-context learning","TabPFN","fitted Q iteration","gradient-free reinforcement learning","foundation models","classic control","context truncation","reward shaping"],"falsifier":"Run TabPFN-RL on the original unshaped versions of MountainCar-v0 or Acrobot-v1 (sparse reward, no shaping); if the agent fails to reach the goal or match DQN on the unshaped environment, the claim is limited to the shaped MDP. Alternatively, compute the optimal policy of the shaped MDP and check whether it is also optimal in the original MDP; any divergence would invalidate the comparison to DQN trained on the original returns.","tokens_in":11928,"feed_emoji":"🎮","tokens_out":3368,"duration_ms":34599,"temperature":0.7,"pith_summary":"This paper claims that a foundation model pre-trained solely on synthetic tabular regression tasks can be repurposed, with no further training, as a Q-function approximator for reinforcement learning. The method, TabPFN-RL, treats fitted Q iteration as in-context regression: TabPFN predicts Bellman targets from a context of curated transitions in a single forward pass, so no backpropagation is needed at any point. On three Gymnasium classic-control environments, the authors report that TabPFN-RL matches or beats Deep Q-Network. If true, this means the inductive bias learned from i.i.d. tabular data transfers, unexpectedly, to the non-i.i.d., bootstrapped structure of RL targets, opening a gradient-free path for RL with pre-trained models.","feed_headline":"No-gradient RL: a tabular transformer matches DQN on classic control","feed_subtitle":"TabPFN-RL repurposes a regression-only transformer to estimate Q-values, matching a deep Q-network without backpropagation.","key_machinery":"TabPFN, a transformer pre-trained for in-context classification and regression on synthetic tabular data, serves as the Q-function approximator. The method casts Fitted Q Iteration as an in-context regression problem: the context is a set of (state-action, Bellman target) pairs, and Q for a query is obtained by one forward pass with no gradient updates. Context management is carried by a high-reward episode gate that admits only episodes whose return exceeds the 95th percentile, plus truncation heuristics (latest-trajectories, naive de-duplication, embedding de-duplication, reward-variance) for continual learning when the context budget is full.","core_discovery":"The central claim is that a transformer meta-trained for in-context regression on millions of synthetic i.i.d. tabular datasets can estimate Q-values accurately enough to support online RL. The authors implement this by embedding Fitted Q Iteration inside TabPFN's context: each transition (s, a, r, s') is a labeled example, and the Bellman target r + γ max_a' Q(s', a') is the label. A high-reward episode gate keeps only the top 5% of trajectories in the context, refitting the Q-estimate by re-running inference. They argue that even though the RL setting violates the i.i.d. and non-bootstrapped assumptions of TabPFN's prior, the model generalizes well enough to be competitive with DQN.","pith_inferences":["The claim is demonstrated only on three low-dimensional, dense-reward (after shaping) tasks; a natural next test is whether the recipe works on sparse-reward or high-dimensional tasks, where the prior mismatch is larger.","If the hand-crafted reward shaping changes the optimal policy, the comparison to DQN on the original environment is not apples-to-apples; evaluating on the original sparse rewards would clarify the scope.","The quadratic inference cost in context size may limit scaling; compressed-memory architectures (e.g., linear attention or state-space models) could make the approach practical beyond toy domains.","The authors' theoretical discussion suggests bootstrapped targets violate TabPFN's prior; quantifying when this violation is harmless could yield a criterion for when prior-fitted networks are usable as value approximators."],"forward_implications":["TabPFN-RL is competitive with DQN on CartPole-v1, MountainCar-v0, and Acrobot-v1 without any gradient updates.","The approach eliminates backpropagation in both training and inference, sidestepping many hyperparameter sensitivities of gradient-based RL.","Naive de-duplication enables continual learning after the context budget is full, outperforming a stale context and a latest-trajectory baseline.","The success suggests prior-fitted networks can extrapolate beyond their i.i.d. tabular prior to bootstrapped, non-stationary RL targets."],"fun_headline_variants":["RL without gradients: a tabular transformer matches DQN","No training, just in-context regression: TabPFN solves RL","Gradient-free RL: a transformer that regresses to Q-values","A regression transformer that tackles RL without any tuning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The hand-crafted shaped reward functions are assumed to preserve the optimal policy of the original Gymnasium tasks, but the paper provides no proof or potential-based guarantee that they do.","fun_headline_variants_meta":{"raw":{"variants":["RL without gradients: a tabular transformer matches DQN","No training, just in-context regression: TabPFN solves RL","Gradient-free RL: a transformer that regresses to Q-values","A regression transformer that tackles RL without any tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2637,"prompt_tokens":795,"completion_tokens":1842,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1771}},"tokens_in":539,"tokens_out":1842,"duration_ms":17606,"temperature":1.0,"reasoning_tokens":1771,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T16:47:28.647440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TabPFN-RL on the original unshaped versions of MountainCar-v0 or Acrobot-v1 (sparse reward, no shaping); if the agent fails to reach the goal or match DQN on the unshaped environment, the claim is limited to the shaped MDP. Alternatively, compute the optimal policy of the shaped MDP and check whether it is also optimal in the original MDP; any divergence would invalidate the comparison to DQN trained on the original returns.","supporting_citations":[],"review_version":1}