REVIEW 5 major objections 4 minor 11 references
Composite Reward Design in PPO-Driven Adaptive Filtering
T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A PPO-trained agent with a composite reward is claimed to beat LMS, RLS, Wiener, and Kalman on nonstationary denoising while running in real time.
desk verdict A plausible RL-for-filtering idea undermined by internal contradictions: the abstract's ECG results don't exist in the body, the architecture changes mid-paper, and the ablation table disagrees with the discussion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the composite reward function $R(t) = \alpha \Delta\text{SNR}(t) - \beta \text{MSE}(t) - \gamma TV(r(t))$, where $\Delta\text{SNR}$ is the signal-to-noise improvement at the filter output, $\text{MSE}$ is the mean-squared error against a reference, and $TV(r)$ is the total variation of the residual $r(t)=y(t)-\hat{x}(t)$. This reward is combined with PPO's clipped surrogate objective $L^{\text{CLIP}}(\theta)$ to keep each policy update conservative. The agent observes a 64-sample sliding window plus the current residual, and its LSTM policy outputs a Gaussian over FIR coefficient updates. The reward's job is to define what 'good filtering' means during training; the ablation study in the paper shows that dropping the MSE term or the TV term leads to divergence or a 2–4 dB generalization loss.
What would settle it
Run the trained PPO policy on an ECG segment from the MIT-BIH Noise Stress Test Database corrupted by baseline wander, electrode motion, and muscle artifacts, and compare output SNR and residual smoothness against grid-tuned LMS, RLS, Wiener, and Kalman filters; the central claim fails if the PPO policy does not beat all four baselines on those real recordings.
Extended reading notes
Core claim
The central claim is that framing adaptive filtering as a Markov decision process and training a PPO policy with the composite reward $R(t) = \alpha \Delta\text{SNR}(t) - \beta \text{MSE}(t) - \gamma TV(r(t))$ yields a filter that adapts to changing noise without manual re-tuning. The paper argues that PPO's clipped surrogate objective prevents the instability that classical gradient-based updates suffer under nonstationary noise, and that the smoothness penalty keeps residuals small and artifact-free. On the paper's synthetic benchmark, the learned policy beats grid-tuned LMS, RLS, Wiener, and Kalman baselines in output SNR across Gaussian, impulsive, Laplacian, pink, brown, and uniform noise, and runs in under one millisecond per time step. The author states this surpasses the classical filters in both SNR and smoothness, not only on seen noise.
Load-bearing premise
The paper assumes that a policy trained only on synthetic sinusoids plus Gaussian noise will transfer to real-world signals such as ECG, even though its own discussion states that evaluation has so far been limited to simulated data.
Editorial extensions
If this is right
- If the central claim holds, a single PPO policy can replace tuned LMS, RLS, Wiener, and Kalman filters in dynamic noise environments, eliminating manual re-calibration.
- A filter trained on only one noise type can generalize to several unseen noise distributions, so training data need not cover every deployment condition.
- The composite reward is not optional: the paper reports that removing the MSE term makes training diverge and removing the TV term causes borderline generalization, so any reimplementation needs all three terms.
- At under 1 ms per inference step, the learned policy is fast enough for low-latency streaming applications such as radio-signal cleaning and biomedical monitoring.
- Because the method does not need ground truth at runtime, it can be applied online in settings where clean signals are unavailable.
Reading between the lines
- A natural test the paper leaves implicit: run the same trained policy on the MIT-BIH Noise Stress Test ECG recordings mentioned in the abstract and report per-noise-type SNR; if it underperforms tuned classical filters there, the synthetic-to-real transfer assumption fails.
- The composite reward is essentially a regularized objective; one could expect similar gains from other policy-gradient algorithms, so the paper's contribution is the reward design more than PPO itself.
- If transfer to real signals succeeds, the paper's own caveat that the training phase is more demanding points to a deployment trade-off: training cost is high, but inference cost is small enough for embedded use only if the network is simplified.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The letter proposes an adaptive filtering framework based on Proximal Policy Optimization (PPO). Adaptive filtering is cast as an MDP (Section III), and the agent is trained with the composite reward R(t)=αΔSNR(t)-βMSE(t)-γTV(r(t)) (Eq. 3), where α, β, γ are balancing weights. The agent is trained on synthetic sum-of-sinusoids corrupted with Gaussian noise and evaluated on Gaussian, Laplacian, impulse, pink, brown, and uniform noise (Section VI). The paper claims that the PPO policy outperforms LMS, RLS, Wiener, and Kalman filters in output SNR and smoothness while running in under 1 ms per step, and that it generalizes to noise types unseen in training. The abstract additionally claims ECG validation on the MIT-BIH Noise Stress Test Database, but no such experiment appears in the body; Section VIII-B states 'Our evaluation has so far been limited to simulated data.'
Significance. If the empirical claims were supported by complete and reproducible experiments, the paper would offer a useful demonstration that a policy-gradient RL agent can act as a low-latency adaptive filter for nonstationary noise, with a clear MDP formulation and a reasonable cross-noise generalization protocol. The runtime comparison and the ablation of reward components are also relevant. However, as submitted, the significance is not established: there are no error bars or statistical tests, the reward weights are not reported, the Table I ablation conflicts with the discussion in Section VIII-A, and the abstract's ECG claim is contradicted by the body. The contribution is therefore best viewed as a pilot study that needs substantial revision before its claims can be evaluated.
major comments (5)
- [Abstract vs. Sections VIII-B and VIII-C] The abstract states that 'real-world analysis is made and evaluated on ECG recordings from the MIT-BIH Noise Stress Test Database' and that the PPO policy 'slightly outperforms strong classical baselines on ECG denoising.' The body contains no ECG experiment, no MIT-BIH results, and no description of how ECG signals were filtered. Section VIII-B states explicitly that 'Our evaluation has so far been limited to simulated data,' and Section VIII-C lists physiological sensor data as future work. This direct contradiction means the abstract's real-world result cannot be independently checked and should be removed or supported with the missing experiment.
- [Section IX] The conclusion says that the PPO-driven filter 'necessarily surpasses traditional adaptive methods (LMS, RLS, Wiener, Kalman) in terms of output SNR and signal smoothness.' The evidence consists of a single synthetic experiment with no reported number of runs, no error bars, no seeds, and no statistical tests. The word 'necessarily' asserts a universality that no finite set of simulations can establish, and the paper provides no proof or exhaustive analysis. The claim should be restricted to the specific tested configurations and supported by repeated runs with confidence intervals.
- [Section V, Eq. (3)] The composite reward in Eq. (3) is the central design contribution, but the values of α, β, and γ are never reported, nor is the selection procedure described. Table I compares reward components, but without the weights the ablation cannot be reproduced or interpreted quantitatively. In addition, ΔSNR and MSE in Eq. (3) are the same metrics used for evaluation in Section VII; the reported SNR improvement therefore reflects the training objective directly rather than an emergent property. The authors should report the weights and add at least one evaluation metric that is not part of the reward, such as spectral distortion or a downstream task accuracy.
- [Section VII-E (Table I) vs. Section VIII-A] Table I reports that the 'SNR + MSE (No TV)' variant 'diverged' with '<0 (diverged)' SNR, while Section VIII-A states that removing the smoothness term 'resulted in degraded performance' with 'a generalization loss of approximately 2–4 dB in output SNR on unseen noises.' A divergent run with negative SNR is not described by a 2–4 dB generalization loss; these two statements are mutually inconsistent. This discrepancy must be resolved before the ablation can be used to support the composite-reward design.
- [Section VI and Section VII-B] The baseline comparisons and reported results are insufficiently specified. Grid-search ranges for the LMS step size, RLS forgetting factor, Kalman process/measurement noise, and Wiener correlation estimates are not given, and the filter order K in the FIR action model is never stated. The text reports only point estimates without standard deviations or test statistics, so the claimed systematic improvement over four classical baselines cannot be quantitatively assessed from the information provided.
minor comments (4)
- [Section VII-D] The word 'plateu' should be 'plateau' in the sentence about episodic reward convergence.
- [Section II] The phrase 'assumptions of stationary' should read 'assumptions of stationarity'.
- [Section VI and Fig. 1] The figures would benefit from axis labels and units for the horizontal axis (noise level or input SNR); the current panels make it difficult to see the numerical magnitudes behind the claimed SNR gains.
- [Section VII-A] The architecture description differs between Section IV/V, which mentions two-layer MLPs, and Section VII-A, which introduces an LSTM state; reconcile these descriptions so the network structure is unambiguous.
Circularity Check
The filter output is defined from the clean target x(t-k), so the reported SNR advantage is built into the model definition rather than learned denoising.
-
self definitional
[Section VI (Experimental Setup), filtered-signal formula; compared with Eq. (1) and Section IX conclusion]
"The filtered signal is computed as: x̂(t) = ( Σ_{k=0}^{K−1} h_k · x(t − k) ) · (1 + η) where h_k are the FIR coefficients and η is a learned meta-scaling parameter."
Eq. (1) defines the observed signal as y(t)=x(t)+n(t), so x is the clean ground truth. The filter output is explicitly written as a function of x(t−k), i.e., of the target itself, not of the noisy observation y. The evaluation then measures output SNR/MSE against x, and the conclusion claims PPO 'necessarily surpasses' LMS, RLS, Wiener and Kalman. With clean x available as the filter input, an identity mapping (h0=1, other taps 0, η=0) reconstructs x perfectly, so any learned SNR gain is a direct artifact of the definition. Comparisons against classical filters that must operate on y are therefore not a test of noise suppression; the prediction reduces to recovering the target from the target.
full rationale
The reward-evaluation alignment (Eq. 3 uses ΔSNR and MSE while results report SNR) is not, by itself, a circularity: an RL agent must optimize some objective, and generalization to unseen synthetic noise types is empirically checkable. The decisive circular step is the filter formula in Section VI: if implemented literally, the 'denoised' output is computed from the clean signal x(t−k), not from the noisy signal y(t−k). Every reported SNR number then measures how well a function of x reconstructs x, which is guaranteed at identity; the conclusion's 'necessarily surpasses' is true by construction, not by filtering skill. I also note non-circularity concerns that affect validity: the abstract claims MIT-BIH ECG evaluation, while Section VIII-B states 'Our evaluation has so far been limited to simulated data,' and Table I's '<0 (diverged)' for the No-TV ablation conflicts with Section VIII-A's '2–4 dB' loss claim. These do not add circularity but mean the paper's external evidence is not as presented.
Assumptions & free parameters
free parameters (4)
- α (SNR reward weight)
- β (MSE reward weight)
- γ (TV smoothness weight)
- η (meta-scaling parameter)
assumptions (4)
- standard math The PPO clipped surrogate objective (Eq. 2) is a valid optimization target for this setting.
- domain assumption The observed signal is the sum of clean signal and additive noise (Eq. 1), and the FIR filter with meta-scaling is an appropriate hypothesis class for denoising.
- ad hoc to paper The composite reward with fixed weights sets a performance lower bound equal to the best baseline model.
- domain assumption Training only on Gaussian noise generalizes to Laplace, impulse, pink, brown, and uniform noise.
Cite this review
Pith. "Pith review of Composite Reward Design in PPO-Driven Adaptive Filtering." pith.science (2026). https://pith.science/paper/O4FQZBSB
@misc{pith2026250606323,
author = {Pith},
title = {Pith review of: Composite Reward Design in PPO-Driven Adaptive Filtering},
year = {2026},
howpublished = {\url{https://pith.science/paper/O4FQZBSB}},
note = {Machine review of arXiv:2506.06323}
}
read the original abstract
Model-free and reinforcement learning-based adaptive filtering methods are gaining traction for denoising in dynamic, non-stationary environments such as wireless signal channels, biomedical monitoring, and sensor networks. Traditional filters such as LMS, RLS, Wiener, and Kalman are often limited by assumptions of stationarity, the need for exact noise statistics, or fragile parameter tuning. This paper proposes an adaptive filtering framework using Proximal Policy Optimization (PPO), guided by a composite reward that balances SNR improvement, MSE reduction, and residual smoothness. We frame adaptive filtering as a Markov decision process and train a PPO agent to adjust filter coefficients directly in response to changing noise. Experiments on synthetic nonstationary signals with diverse noise types show that the PPO agent generalizes beyond its training distribution. Moreover, real-world analysis is made and evaluated on ECG recordings from the MIT-BIH Noise Stress Test Database corrupted by baseline wander, electrode motion, and muscle artifacts. The learned PPO policy achieves real-time inference and slightly outperforms strong classical baselines on ECG denoising. These results demonstrate the viability of policy-gradient reinforcement learning as a computationally efficient and flexible tool for adaptive filtering in nonlinear, time-varying dynamical systems.
Figures
Reference graph
Works this paper leans on
-
[1]
B. Widrow and S. D. Stearns, Adaptive Signal Processing . Englewood Cliffs, NJ: Prentice-Hall, 1985
work page 1985
-
[2]
Haykin, Adaptive Filter Theory , 5th ed
S. Haykin, Adaptive Filter Theory , 5th ed. Upper Saddle River, NJ: Pearson Education, 2013
work page 2013
-
[3]
A new approach to linear filtering and prediction problems,
R. E. Kalman, “A new approach to linear filtering and prediction problems,” J. Basic Eng. , vol. 82, no. 1, pp. 35–45, 1960
work page 1960
-
[4]
Proximal policy optimization algorithms,
J. Schulman et al. , “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017
arXiv 2017
-
[5]
M. Oh et al. , “Channel estimation via successive denoising in MIMO- OFDM systems: a reinforcement learning approach,” in Proc. IEEE ICC, 2021
work page 2021
-
[6]
Adaptive filtering algorithm based on reinforcement learning,
H. Lin et al. , “Adaptive filtering algorithm based on reinforcement learning,” in Proc. CCDC, 2024, pp. 5268–5272
work page 2024
-
[7]
A fractional filter based on reinforcement learning for effective tracking under impulsive noise,
X. Xie et al. , “A fractional filter based on reinforcement learning for effective tracking under impulsive noise,” Neurocomputing, vol. 516, pp. 155–168, 2023
work page 2023
-
[8]
Reinforcement learning adaptive Kalman filter for AE signal’s AR-mode denoise,
P. He et al. , “Reinforcement learning adaptive Kalman filter for AE signal’s AR-mode denoise,” in Proc. IEEE ICICN , 2023
work page 2023
Show all 11 references
-
[9]
Beyond static obstacles: integrating Kalman filter with reinforcement learning for drone navigation,
F. Marino and G. Guglieri, “Beyond static obstacles: integrating Kalman filter with reinforcement learning for drone navigation,” Aerospace, vol. 11, no. 5, 2024
2024
-
[10]
GFANC-RL: reinforcement learning-based generative fixed-filter active noise control,
Z. Luo et al. , “GFANC-RL: reinforcement learning-based generative fixed-filter active noise control,” Neural Networks, vol. 180, p. 106687, 2024
2024
-
[11]
Exploration by random network distillation,
Y . Burda et al. , “Exploration by random network distillation,” in Proc. ICLR, 2019
2019
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.