RL as Regressor: A Reinforcement Learning Approach for Function Approximation

Yongchao Huang

REVIEW 3 major objections 3 minor 10 references

RL as Regressor: A Reinforcement Learning Approach for Function Approximation

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that regression can be reformulated as a reinforcement learning problem and demonstrates the reformulation with an actor-critic agent that learns and extrapolates a noisy sine wave.

desk verdict A clear tutorial on a known idea, but the evidence only supports a sine-wave demo—not the paper's 'power and flexibility' claims. read the letter →

arxiv 2508.00174 v1 pith:BWSIXPTC submitted 2025-07-31 cs.LG

Yongchao Huang This is my paper · ORCID

classification cs.LG

keywords reinforcementlearningfunctionapproximationregressionactor-criticprioritizedexperiencereplaypositionalencodingcontextualbandits

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that regression, normally done by minimizing a predefined loss like mean squared error, can instead be framed as a reinforcement learning problem in which the model's prediction is an action and the error signal is a customizable reward. Using a noisy sine wave as a case study, the author builds an actor-critic agent with a Gaussian-kernel reward, then progressively improves it with prioritized experience replay, larger networks, and positional encoding. The central claim is that this RL framework not only fits the function but offers flexibility to encode objectives that are asymmetric, non-differentiable, or otherwise hard to express as a standard loss. A sympathetic reader would take the paper as a demonstration that RL-based regression is worth considering when the objective, not the fitting, is the hard part.

What carries the argument

The central object is the contextual-bandit reformulation of regression: the input features are the state, the model's prediction is the action, and the reward is the Gaussian kernel $R(\hat{y}, y) = \exp\left(-(y-\hat{y})^2/(2\sigma^2)\right)$, with $\sigma=0.2$ in the experiments. The actor is a multi-layer perceptron that outputs the prediction; the critic is another MLP that estimates the Q-value of a state-action pair; the actor is updated to maximize the critic's prediction while the critic is fitted to the observed reward. To make the periodic structure of the sine data explicit, the scalar input $x$ is mapped to a 16-dimensional vector of sine and cosine features, $PE(x)_i = \sin(2^{i/2}x)$ for even $i$ and $PE(x)_i = \cos(2^{(i-1)/2}x)$ for odd $i$, and prioritized experience replay is used to sample high-error transitions more often. These components carry the argument: the reward defines the objective, the actor-critic pair supplies a policy-gradient path to optimizing it, and the feature transform is what finally makes the target function learnable in the reported experiment.

What would settle it

Train the Stage 4 agent on the same noisy sine data with five different random seeds and compare the held-out mean squared error against a standard MLP trained with MSE and the same positional encoding; if the RL regressor loses to the MSE baseline or shows large seed-to-seed variance, the claim that the RL framing is a viable alternative to standard regression is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that function approximation can be turned into a single-step reinforcement learning problem, a contextual bandit, where the state is the input, the action is the predicted value, and the reward is a smooth Gaussian function of prediction error. An actor network proposes the prediction, a critic network estimates the expected reward, and training proceeds by alternating critic updates against observed rewards with actor updates toward higher critic values. The case study shows that this setup first learns part of a sine wave on one period, collapses when the range is widened, recovers with a larger network, and only succeeds when a 16-dimensional sine-cosine positional encoding makes the periodic structure explicit. The author's stated conclusion is that the RL framing is a viable and flexible alternative to standard regression, with the caveat that it is overkill when a simple loss like MSE suffices.

Load-bearing premise

The load-bearing premise is that a single run on one synthetic noisy sine wave, with hand-selected hyperparameters, no repeated-seed variation, and no comparison against a standard regressor, can stand as evidence that the RL framing is generally viable.

Editorial extensions

If this is right

If the central claim holds, any regression task with a complex or non-differentiable objective can in principle be solved by choosing a reward and running a continuous-action RL algorithm, without needing a differentiable surrogate loss.
The case study implies that for periodic data, input representation such as positional encoding can matter more than the learning algorithm, since the same network failed without it and succeeded with it.
The actor-critic regressor inherits RL-style exploration, which may help escape local optima that gradient descent on MSE would not.
The framework is not intended to replace MSE for simple problems; its advantage appears only when the reward captures something MSE cannot.
The same paradigm can be extended from regression to classification, as the conclusion suggests, by defining actions and rewards for class prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the decisive ingredient in the sine experiment may be positional encoding, not RL per se; a standard MLP trained with MSE on the same 16-dimensional features would likely also fit the function, which would show the demonstration supports feature engineering more than the RL framing.
Editorial inference: because the reward is a smooth function of error, the actor-critic regressor is comparable to a stochastic-gradient method on a transformed error; a direct comparison of sample efficiency against MSE-trained networks with identical architecture would separate RL-specific benefits from generic capacity and representation effects.
Editorial inference: the same machinery could be used for quantile or asymmetric regression by replacing the Gaussian reward with a tilted or one-sided reward, a direction the paper mentions conceptually but does not test.
Editorial inference: the promise of handling non-differentiable objectives remains untested because the reported experiments only use a differentiable Gaussian reward; a test with a genuinely non-differentiable or asymmetric reward would be the natural next check.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Rough take: this is a readable tutorial on a well-known idea. If it were presented as a teaching walkthrough, it would be useful. But the abstract and conclusion claim RL gives 'enhanced flexibility' and a 'powerful approach' to function approximation, and the evidence doesn't support that.

What the paper does well: it clearly explains how to frame a regression problem as a one-step RL task, provides a four-stage progressive example, and includes code. The Discussion is honest enough to say RL is overkill for simple regression. The writing is accessible.

The soft spots are substantial. The reward in Eq. (1) is a monotone function of negative squared error, so maximizing it is equivalent to minimizing MSE. The claimed flexibility for asymmetric or non-differentiable costs is never actually tested. There is no supervised baseline—an MLP with the same positional encoding and MSE loss would likely fit the sine wave at least as well. No repeated seeds or error bars, so the 'perfect' fit in Fig. 4 is a single hand-tuned run. The success in Stage 4 is driven by the positional encoding and a larger network; nothing isolates the RL-specific contributions.

The reader's conditional verdict is fair. The framing as a contextual bandit is already in the cited literature (Williams, Langford & Zhang, Lillicrap et al.). The only genuinely new part is the specific progression, which is routine.

I wouldn't send this to a serious research venue as-is. It's a tutorial with overclaimed results. If the authors reframed it as a tutorial and added a baseline and a non-MSE reward example, it could be worth a workshop or teaching venue. For now, it doesn't deserve peer-review time.

Recommendation: don't engage for journal review; if you need a concrete example of RL-as-regression for a class, it's fine to cite.

Referee Report

3 major / 3 minor

Summary. The paper proposes framing standard regression as a reinforcement learning problem, treating the model's prediction as an action and a Gaussian-kernel reward as the training signal. The authors implement an actor-critic agent (with prioritized experience replay) and demonstrate it on a noisy sine-wave regression task through four progressive stages: simple architecture, expanded data range, increased network capacity, and finally a hand-crafted positional encoding. The authors conclude that the RL framework is a viable and powerful approach to regression that offers enhanced flexibility in defining objectives. The manuscript includes code and is written as a tutorial-like case study.

Significance. The paper's core idea—using RL to optimize regression objectives that are non-differentiable or asymmetric—is potentially interesting, and the authors honestly acknowledge that for plain MSE regression the RL framework is overkill. The availability of code and the progressive failure-mode analysis are commendable. However, the empirical demonstration is too narrow to support the abstract's claim of a 'viable and powerful approach' with 'enhanced flexibility': the chosen reward is a monotone transformation of negative MSE, no baseline or ablation is presented, and all conclusions rest on a single visual inspection of one synthetic dataset without repeated seeds or quantitative metrics.

major comments (3)

[Section 3, Eq. (1); Section 5] The reward R(ŷ,y) = exp(−(y−ŷ)²/(2σ²)) is a strictly monotone decreasing function of squared error for a fixed σ, so maximizing the expected reward is equivalent to minimizing expected squared error. Consequently, the experimental demonstration never exercises the claimed ability to handle asymmetric costs or non-differentiable objectives. The Discussion (Section 5) correctly states that for simple MSE regression the RL framework is overkill, but the Abstract and Conclusion nonetheless assert 'enhanced flexibility' without any experiment that uses a non-MSE, non-differentiable, or asymmetric reward. To support the central claim, the authors should include at least one experiment with such a reward and show that the RL agent optimizes it, ideally with a comparison to a supervised baseline.
[Section 4.4, Fig. 4] The claim that the Stage 4 model 'perfectly learns the function' is based solely on visual inspection. No quantitative error metrics (e.g., MSE, MAE, R²) are reported, no train/test split is precisely defined, no repeated seeds or variance bars are provided, and the test-range extrapolation is not evaluated numerically. Given the stochastic nature of RL and the small dataset (1000 samples), this is insufficient evidence for the paper's general claims. The authors should report mean and standard deviation of prediction error over multiple random seeds, with a clear description of training and test ranges.
[Section 4.2–4.4] The progressive study does not isolate the contribution of the RL-specific machinery. Stage 4 changes the input representation (positional encoding) while using the Stage 3 network, but no baseline is provided—e.g., a supervised MLP trained with MSE on the same positional-encoding features, or an actor-critic without PER, or an actor-critic without exploration noise. Without such ablations, the success in Stage 4 could be entirely attributable to the feature engineering and network capacity, which are standard supervised-learning ingredients. The paper would be substantially strengthened by reporting (a) a supervised MLP with the same architecture and positional encoding trained with MSE, and (b) an RL variant with the same features but without PER, to show what the RL formulation adds.

minor comments (3)

[Section 4.1] Please clarify the exact training/test split for each stage. For instance, in Stage 1 the model is trained on x ∈ [−π, π] but 'fails to generalize outside this narrow range'; in Stage 4 the model 'correctly extrapolates to the wider test range'—what is that test range, and how were the test points generated?
[Section 4.2] The text says 'Actor training losses even increase at the end' but no learning-curve plot is referenced; including such a plot or explicitly describing the loss trajectory would help diagnose the failure. Also, the figure axes in Fig. 2–4 appear unlabeled, which makes visual inspection harder.
[Section 4.4, Eq. (2)] The indexing of the positional encoding is not defined for general i; please state whether i starts at 0 and specify how the 16-dimensional vector is ordered (e.g., which i values are used).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RL-for-regression demonstration does not reduce to its inputs; the Gaussian reward is monotone in squared error, but the actor-critic training procedure is a distinct optimization path, and no parameter is fitted and then renamed a prediction.

full rationale

The paper's central content is a progressive empirical case study rather than a derivation chain whose conclusion is encoded in its premises. Equation (1), R(yhat, y) = exp(-(y - yhat)^2 / (2 sigma^2)), is a strictly decreasing transformation of squared error, so in a pure objective sense maximizing R is equivalent to minimizing MSE for fixed sigma. This is a mathematical observation, but it is not circular: the paper explicitly sets this reward as the RL signal and then trains an actor-critic policy to maximize the critic's predicted Q-value, which is a different update mechanism than directly applying MSE gradients to the regressor. The paper does not fit any parameter to a subset of the data and then 'predict' a closely related quantity; all experiments use the same 1000 synthetic samples, and the sine-wave task is illustrative rather than a benchmark whose answer is presupposed. The 'enhanced flexibility' claim is an assertion about the RL framing's capacity to admit arbitrary rewards, not a quantity derived from the demonstration; the absence of any non-MSE, asymmetric, or non-differentiable reward experiment weakens the empirical support for that claim, but unsupported breadth is not circularity. The cited works (Williams, Lillicrap, Schaul, Vaswani, etc.) are standard external algorithm references, and none is a self-citation carrying a load-bearing uniqueness claim. The paper also explicitly acknowledges in Section 5 that for simple regression tasks 'this RL framework would be overkill,' which undercuts any suggestion that the framework's value is being smuggled in as a necessary conclusion. Therefore no circular step meeting the quote-and-reduction standard is present; the main concerns are evidentiary, not circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central demonstration depends on several hand-chosen hyperparameters, especially the reward sigma and the positional encoding dimension, and on the domain assumption that the contextual bandit mapping is appropriate for regression. The Gaussian reward being a monotone transform of squared error is the most important ledger item: it means the claimed flexibility is not actually exercised.

free parameters (4)

Reward length-scale sigma = 0.2
Set by hand in Section 4. Controls the width of the Gaussian reward in Equation 1. Because the reward is a monotone function of squared error, sigma acts as an inverse loss weight, making the effective objective equivalent to MSE.
Exploration noise standard deviation = 0.1
Set by hand in Section 4. Added to the deterministic actor's actions during training. Required for exploration in the DDPG-style setup, and its value affects the exploration-exploitation tradeoff.
Positional encoding dimension = 16
Set by hand in Section 4.4. The dimension of the sine/cosine feature vector in Equation 2. The paper reports that this feature representation is the key ingredient for the successful fit and extrapolation.
Network capacity (hidden layer sizes) = 128/64, later 256/128/64
Chosen iteratively in Sections 4.1 and 4.3. The capacity increase from a 3-layer to a 4-layer network is reported as necessary, though not sufficient, for learning the periodic function.

assumptions (4)

standard math Maximizing the Gaussian reward exp(-(y - y_hat)^2 / (2 sigma^2)) is equivalent to minimizing squared error for fixed y and sigma.
Used implicitly in the reward design in Section 3. The monotone transformation means the RL objective in the demonstration is mathematically the same as MSE regression with a particular weighting.
domain assumption A regression task can be mapped to a contextual bandit where state is the input, action is the prediction, and reward is prediction accuracy.
Stated in Section 3 as the core mapping. This is a known formulation from the cited bandit literature, but the paper treats it as the foundation for the entire approach.
domain assumption A deterministic actor-critic (DDPG-style) update can optimize a continuous-action reward function in this one-step setting.
The paper relies on the convergence and stability properties of DDPG [3] without proving them for this specific regression setup. The Actor loss is the negative Critic value, which assumes the Critic provides a useful gradient signal.
ad hoc to paper The hand-crafted positional encoding in Equation 2 is a generalizable feature representation for periodic regression, not an overfit choice for the sine wave.
The paper concludes that feature representation is more critical than the learning algorithm, but this conclusion rests entirely on a representation engineered specifically for the known periodic structure of the synthetic target.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RL as Regressor: A Reinforcement Learning Approach for Function Approximation." pith.science (2026). https://pith.science/paper/BWSIXPTC

@misc{pith2026250800174,
  author       = {Pith},
  title        = {Pith review of: RL as Regressor: A Reinforcement Learning Approach for Function Approximation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWSIXPTC}},
  note         = {Machine review of arXiv:2508.00174}
}

read the original abstract

Standard regression techniques, while powerful, are often constrained by predefined, differentiable loss functions such as mean squared error. These functions may not fully capture the desired behavior of a system, especially when dealing with asymmetric costs or complex, non-differentiable objectives. In this paper, we explore an alternative paradigm: framing regression as a Reinforcement Learning (RL) problem. We demonstrate this by treating a model's prediction as an action and defining a custom reward signal based on the prediction error, and we can leverage powerful RL algorithms to perform function approximation. Through a progressive case study of learning a noisy sine wave, we illustrate the development of an Actor-Critic agent, iteratively enhancing it with Prioritized Experience Replay, increased network capacity, and positional encoding to enable a capable RL agent for this regression task. Our results show that the RL framework not only successfully solves the regression problem but also offers enhanced flexibility in defining objectives and guiding the learning process.

Figures

Figures reproduced from arXiv: 2508.00174 by the authors.

**Figure 2.** Stage 2 results. With an expanded data range, the simple model fails to learn the periodic [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Stage 3 results. A deeper network begins to learn the pattern but fails to generalize to [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Stage 4 results. With positional encoding, the high-capacity network perfectly learns the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 5 canonical work pages

[1]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January
[2]

The Epoch-Greedy algorithm for contextual multi-armed ban- dits

John Langford and Tong Zhang. The Epoch-Greedy algorithm for contextual multi-armed ban- dits. In Proceedings of the 21st International Conference on Neural Information Processing Systems, NIPS’07, pages 817–824, Red Hook, NY, USA, December 2007. Curran Associates Inc

work page 2007
[3]

Lillicrap, Jonathan J

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, July 2019. arXiv:1509.02971 [cs]. 6

arXiv 2019
[4]

Rusu, Joel Veness, Marc G

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforceme...

work page 2015
[5]

Prioritized Experience Replay, February 2016

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay, February 2016. arXiv:1511.05952 [cs]

arXiv 2016
[6]

Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes, September

Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Mart ´ ın-Mart ´ ın, and Peter Stone. Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes, September

work page
[7]

Attention is All you Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Infor- mation Processing Systems, volume 30. Curran Associates, Inc., 2017

work page 2017
[8]

Williams

Ronald J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Rein- forcement Learning. Mach. Learn., 8(3-4):229–256, May 1992. 7

work page 1992

Show all 10 references

[2017]

arXiv:1412.6980 [cs]

arXiv
[2024]

arXiv:2408.03539 [cs]

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January

[2] [2]

The Epoch-Greedy algorithm for contextual multi-armed ban- dits

John Langford and Tong Zhang. The Epoch-Greedy algorithm for contextual multi-armed ban- dits. In Proceedings of the 21st International Conference on Neural Information Processing Systems, NIPS’07, pages 817–824, Red Hook, NY, USA, December 2007. Curran Associates Inc

work page 2007

[3] [3]

Lillicrap, Jonathan J

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, July 2019. arXiv:1509.02971 [cs]. 6

arXiv 2019

[4] [4]

Rusu, Joel Veness, Marc G

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforceme...

work page 2015

[5] [5]

Prioritized Experience Replay, February 2016

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay, February 2016. arXiv:1511.05952 [cs]

arXiv 2016

[6] [6]

Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes, September

Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Mart ´ ın-Mart ´ ın, and Peter Stone. Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes, September

work page

[7] [7]

Attention is All you Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Infor- mation Processing Systems, volume 30. Curran Associates, Inc., 2017

work page 2017

[8] [8]

Williams

Ronald J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Rein- forcement Learning. Mach. Learn., 8(3-4):229–256, May 1992. 7

work page 1992