{"id":"de9fa757-79a0-4d93-a08e-822ab7b513ae","arxiv_id":"2507.02356","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Injecting noise-perturbed actions into offline Q-learning with a distance penalty improves D4RL performance over IQL and TD3 baselines, formalized as Q-learning in a Noisy Action MDP.","lead":"The paper proposes PANI, a simple offline RL trick: add noise to dataset actions during Q-learning and penalize the Q-target by the squared distance from the original action. The authors show this update corresponds to Q-learning in a modified 'noisy action MDP', and report improved D4RL scores on top of IQL, TD3, and QGPO.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem B.3 proves the NAMDP equivalence only for an oracle Qπ target; the reported IQL-AN and TD3-AN algorithms use bootstrapped targets, so the theoretical foundation is not yet connected to the empirical results.","rationale":"The reader identified the same weakest assumption: Theorem B.3 uses an oracle Qπ target while the implementations use bootstrapped targets. I agree this is the most load-bearing concern because the paper's stated novelty includes a theoretical foundation, and the reported algorithms are the entities that need that foundation. The empirical results are substantial and the paper is otherwise clearly written, so a conditional verdict is appropriate rather than rejection; however, the theoretical claim should be either strengthened with a convergence proof for the bootstrapped updates or explicitly narrowed to the oracle-target objective. My proposed check is analytical and tabular, making the gap testable without relying on the authors' code.","tokens_in":37142,"tokens_out":4243,"duration_ms":56196,"concrete_test":"Independently re-derive the fixed point of the coupled updates defined by Eq. 66 for IQL-AN in a small finite MDP: minimize E_{a∼pD, a′∼qσ}[(Q(s,a′) − (r(s,a) − ∥a−a′∥² + γVψ(s′)))²] where Vψ is the IQL expectile of the target Q. Compute the joint fixed point and compare it to the NAMDP Qπ from Definition 5.2. If the fixed point differs, Theorem B.3 does not cover IQL-AN, confirming the gap; if it coincides, the concern is resolved. Repeat the same derivation for the TD3-AN target in Eq. 69 with the target policy action.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is Theorem B.3, which states that minimizing the PANI objective yields the Q-function of the policy π in the NAMDP. The proof, however, assumes an oracle target: ¯y(s,a,a′) contains γQπ(s′,ā) with the true Qπ. The implemented algorithms replace this oracle with bootstrapped targets: IQL-AN uses γVψ(s′) (Eq. 66), where Vψ is trained by expectile regression and is not Qπ, and TD3-AN uses γ mini Qθ′_i(s′, ã) with ã from the target actor plus clipped noise (Eq. 69). No theorem or contraction argument connects these iterative target-network updates to the NAMDP fixed point. Thus the paper shows that a population-level PANI objective with oracle targets solves a modified MDP, but it does not show that the algorithms whose D4RL scores are reported solve that MDP. Since the abstract and introduction explicitly claim a theoretical foundation for the method, this gap is load-bearing: the empirical gains could be driven by the squared-distance penalty as a regularizer rather than by the NAMDP mechanism the paper claims to instantiate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Penalized Action Noise Injection (PANI), a modification of Q-learning for offline RL in which dataset actions are perturbed by noise and the target is penalized by the squared distance between the original and noisy action. The authors introduce a 'Noisy Action MDP' (NAMDP) and claim that minimizing the PANI objective solves this modified MDP, thereby providing a theoretical foundation for the method. They also design a hybrid noise distribution intended to be robust to noise-scale selection. Empirically, the paper applies PANI to TD3, IQL, and QGPO, reporting improved D4RL scores on many Gym-MuJoCo and AntMaze tasks compared with IQL and TD3+BC, together with ablations on noise distributions, noise scales, wall-clock time, and a direct measurement of OOD overestimation.","tokens_in":37448,"tokens_out":6353,"duration_ms":75303,"significance":"If the empirical results hold, PANI is an appealingly simple and computationally cheap alternative to diffusion-based offline RL methods, and the extensive D4RL evaluation plus detailed hyperparameter tables and ablations are valuable. The paper also ships code and gives concrete implementation details, which strengthens reproducibility. However, the theoretical contribution as stated does not currently cover the implemented algorithms: the main theorem assumes an oracle target using the true Q^pi, while the reported IQL-AN and TD3-AN use learned, bootstrapped targets, and no convergence argument bridges this gap. The NAMDP equivalence is also largely a fixed-point identity. The significance of the paper therefore rests mainly on the empirical claims, which are substantial but would need to be presented without the overreach in the theoretical framing.","major_comments":[{"comment":"Theorem B.3 (and its main-text version Theorem 5.3) states that minimizing the PANI objective yields the Q-function of pi in the NAMDP, but its target y-bar contains gamma Q^pi(s', a-bar) with the true Q^pi. The implemented IQL-AN objective in Eq. (66) uses gamma V_psi(s') with V_psi trained by expectile regression, and the TD3-AN objective in Eq. (69) uses gamma min_i Q_{theta'_i}(s', a-tilde) with target networks and clipped policy noise. The paper provides no contraction, convergence, or fixed-point argument that relates these iterative, bootstrapped targets to the NAMDP fixed point of Theorem B.3. Consequently, the claimed theoretical foundation does not apply to the algorithms whose D4RL scores are reported; the empirical gains may be attributable to the squared-distance penalty acting as a plain regularizer. I request either a proof connecting the implemented target updates to the NAMDP solution or an explicit reframing of Theorem B.3 as a population-level fixed-point identity that motivates, rather than establishes, the algorithm.","section":"Theorem B.3 and Appendix C, Eq. (66), (69)"},{"comment":"The theorem is close to a fixed-point identity. Because the target is defined using Q^pi, the derivation shows that any minimizer Q* satisfies a Bellman equation whose solution is already Q^pi; the conclusion then follows from uniqueness of the Bellman equation for a fixed policy. The statement would be more informative if it identified exactly which assumption is being verified, for example that the PANI regression fixed point coincides with the NAMDP Bellman fixed point, rather than presenting the Euler-equation argument as an independent derivation of the NAMDP equivalence.","section":"Theorem B.3 proof, Eq. (8)-(21)"},{"comment":"The proof of Theorem B.11 contains an incorrect expectation. In Eq. (59) the proof bounds max_{a in C(a', p_D(·|s))} Q*_sigma(s,a) below by E_{a ~ p_D(·|s)}[Q*_sigma(s,a)], but Lemma B.7 and the subsequent algebra require the expectation over p_C(·|a',s), the restriction of the dataset action distribution to the closest-point set C. As written, the inequality is false because p_D(·|s) can place mass outside C with larger Q-values. Replacing p_D by p_C in that line repairs the argument, but the theorem's proof currently contains a technical error at a load-bearing step.","section":"Theorem B.11 proof, Eq. (59)"}],"minor_comments":[{"comment":"Definition 5.1 says 'with support supp(q_sigma) such that the action space A is a subset of its support'; since q_sigma is a family of distributions parameterized by a, the support condition should be stated per conditional distribution q_sigma(·|a), and the current phrasing is ambiguous.","section":"Definition 5.1"},{"comment":"The text states that PANI with the hybrid distribution outperforms baselines 'even with all hyperparameters fixed,' but Table 5 reports per-environment optimal log sigma and alpha values selected from grids; please clarify that only the non-noise hyperparameters are fixed, or provide an explicit no-tuning experiment.","section":"Section 7.2"},{"comment":"The OOD overestimation metric P(Q(s,a') > Q(s,a)) with a' drawn uniformly is mechanically lowered by the squared-distance penalty, since distant actions receive penalized targets; please report an additional metric that separates the penalty from genuine value calibration, such as error against true returns in an environment with known dynamics.","section":"Table 6a"},{"comment":"The training curves are labeled with log sigma settings that differ in range between Gaussian/Laplace (0 to -2) and hybrid (-20 to -1); please make the legend keys consistent and legible so that the comparison across noise distributions is not visually misleading.","section":"Figures 7-9"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is solid and the method is simple, well-validated, and potentially useful. The main risk is the theory-practice gap: the central theorem uses an oracle target while the reported algorithms use bootstrapped targets, and the theorem itself is close to a fixed-point identity. If the authors can provide a convergence result or substantially soften the theoretical claims, I would support publication; as it stands, the framing overstates the theoretical foundation for the implemented method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: PANI is a simple, cheap regularizer that empirically lifts offline RL baselines on many D4RL tasks, and the paper's advertised theory does not yet cover the algorithms actually evaluated. The gap is real, but the empirical contribution is substantial enough that the paper deserves a serious referee.\n\nWhat's new: the hybrid noise distribution and the NAMDP formalism. The hybrid distribution, mixing uniform and exponentially scaled Gaussian noise, is a reasonable practical contribution, and the ablations across noise distributions and scales are thorough. The experiments are extensive: 18 Gym-MuJoCo tasks, AntMaze, comparisons against ReBRAC-tuned baselines and a spread of diffusion methods, plus a QGPO variant. Wall-clock overhead is small (3–9%). Code is anonymously linked.\n\nWhere it goes soft: Theorem B.3 proves the NAMDP equivalence only for an oracle target that uses the policy's true Qπ. The implemented IQL-AN and TD3-AN replace it with bootstrapped targets (Vψ or the target critic minima). No contraction or convergence argument connects those updates to the NAMDP fixed point. So the 'theoretical foundation' is, as written, a population-level fixed-point identity rather than a characterization of the algorithm whose scores are reported. The paper should either prove iterative convergence or soften the claim to 'theoretical insight into the regularized objective.' Also, RORL is described in related work as just an ensemble method, which understates that RORL is essentially conservative smoothing with noise-injected actions and penalties—the closest existing relative. The per-environment tuning of log σ (from −1 to −20) is a genuine free parameter, though the hybrid noise distribution demonstrably reduces sensitivity to it.\n\nWho it's for: practitioners who want a drop-in regularizer for offline RL without diffusion inference costs. The empirical gains are real and the method is simple enough to try. The theoretical overclaim is fixable; it should not sink the paper.\n\nRecommendation: send to peer review. The empirical work is solid, the ablation study is careful, and the method is useful even if the theory needs revision. I'd push for conditional acceptance with a required revision on the theory–algorithm connection.","headline":"PANI is a simple, well-tested regularizer that lifts offline RL scores, but the advertised NAMDP theory doesn't yet cover the algorithms that produce those scores.","tokens_in":37924,"tokens_out":3276,"would_cite":true,"duration_ms":39717,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"Penalized action-noise injection turns offline RL into a modified MDP—the Noisy Action MDP—that suppresses out-of-distribution overestimation.","keywords":["offline reinforcement learning","Penalized Action Noise Injection (PANI)","Noisy Action MDP","out-of-distribution overestimation","D4RL benchmark","hybrid noise distribution","Q-learning","action-space coverage"],"falsifier":"Run the reported IQL-AN and TD3-AN critic updates in a small tabular MDP with known dynamics, compute the NAMDP Q-function exactly from the paper's Definition 5.2, and compare the learned Q to it; if the bootstrapped-trained Q does not converge to the NAMDP fixed point, the theoretical foundation does not cover the empirical method.","tokens_in":36974,"feed_emoji":"🎲","tokens_out":8634,"duration_ms":86052,"temperature":0.7,"pith_summary":"PANI claims that a few lines of code—drawing a noisy action from each dataset action and regressing Q against a target reduced by the squared perturbation distance—can give offline RL the same out-of-distribution protection that diffusion-based methods buy with heavy generative models. The paper proves that this recipe is not ad hoc: the PANI objective is exactly Q-learning in a modified process it calls the Noisy Action MDP (NAMDP), whose reward and transition kernels are the dataset dynamics re-weighted by the noise kernel. Because the noise covers the whole action space, Q updates reach low-density regions, and the penalty term pushes the NAMDP's optimal policy back toward dataset actions. On D4RL, adding PANI to TD3 and IQL improves average scores by roughly 11 and 7 points respectively, with only 3–9% extra training time, and it also helps the diffusion-based algorithm QGPO. The broader claim is that sophisticated generative modeling is not necessary for strong offline RL performance.","feed_headline":"A dash of penalized noise lifts offline RL scores by 11 points","feed_subtitle":"PANI perturbs dataset actions, penalizes the distance, and beats tuned baselines without diffusion models.","key_machinery":"The NAMDP is the central object: an MDP $(\\mathcal{S}, \\mathcal{A}, R_\\sigma, P_\\sigma, \\gamma)$ whose reward and transition are the dataset reward and dynamics averaged against the noise-weight function $p_D(a'|s,a,\\sigma)$. The load-bearing derivation uses the Euler equation for functionals to show that the minimizer of the PANI squared-loss objective satisfies the NAMDP Bellman equation, so minimizing the PANI objective is equivalent to learning the NAMDP's Q-function. The practical engine is the hybrid noise distribution—a mixture of a uniform component with exponentially scaled Gaussian scales—which the paper shows is more robust across noise levels than plain Gaussian or Laplace noise.","core_discovery":"The central discovery is that the penalized noise-injection update is not just a trick but defines a new MDP. Given a noise distribution $q_\\sigma$ and a dataset distribution $p_D$, the Noisy Action MDP (NAMDP) has transition kernel $P_\\sigma(s'|s,a') = \\int p_D(s'|s,a) p_D(a'|s,a,\\sigma) da$ and reward $R_\\sigma(s,a') = \\int p_D(a'|s,a,\\sigma)(R(s,a) - \\|a-a'\\|_2^2) da$, where $p_D(a'|s,a,\\sigma)$ is the noise distribution normalized against the dataset action distribution. Theorem 5.3 shows that the minimizer of $\\mathbb{E}_{a\\sim p_D(\\cdot|s), a'\\sim q_\\sigma(\\cdot|a)} \\|Q(s,a') - \\bar y(s,a,a')\\|_2^2$ with target $\\bar y = \\mathbb{E}_{s'\\sim p_D(\\cdot|s,a), \\bar a\\sim\\pi(\\cdot|s')}[R(s,a) - \\|a-a'\\|_2^2 + \\gamma Q^\\pi(s', \\bar a)]$ is exactly the $\\pi$ Q-function of the NAMDP. Around this fixed point, the paper adds an error bound between NAMDP and true MDP returns, and a small-noise analysis showing the NAMDP's optimal policy selects actions within dataset support. Empirically, the method manifests as TD3-AN, IQL-AN, and QGPO-AN, with Table 1 reporting gains over tuned baselines across Gym-MuJoCo and AntMaze.","pith_inferences":["If the NAMDP equivalence is extended from the population fixed point to the actual bootstrapped target networks used in training, PANI would supply a clean theoretical template for why simple penalties can replace generative policies; a tabular MDP experiment comparing learned Q with the exact NAMDP Q would test this directly.","The paper's empirical noise-scale guideline—high action diversity wants low noise, low diversity wants high noise—suggests a practical automatic rule, for instance setting $\\sigma$ from nearest-neighbor action distances or dataset density estimates, which the paper does not develop.","The same noise-injection-plus-penalty idea could be applied to other value-based and model-based offline methods, and to high-dimensional action spaces where Gaussian noise becomes sample-inefficient; the squared-distance penalty might then need normalization to the reward scale.","Because the noise kernel re-weights both reward and transition, the NAMDP view connects PANI to robust MDP and anti-exploration ideas, suggesting that noise scale and penalty weight are dual regularization knobs that could be tuned jointly."],"forward_implications":["PANI can be dropped into any off-policy Q-learning update: only the sampling line and the target arithmetic change, so TD3, IQL, and even the diffusion-based QGPO all improve with minimal code changes.","OOD overestimation falls: the measured probability that a uniformly sampled action outscores a dataset action drops from roughly 0.1–0.5 to below 0.04 on the tested Gym-MuJoCo datasets.","The small-noise analysis implies a safety property: as noise tends to zero, the NAMDP's optimal policy avoids actions outside the dataset support, which is exactly the failure mode offline RL needs to prevent.","A hybrid noise distribution—mixing a uniform component with exponentially scaled Gaussian scales—makes performance stable across noise levels, reducing the need for careful per-task $\\sigma$ tuning.","Diffusion models are not required for the gains: lightweight IQL-AN and TD3-AN match or beat several diffusion-policy baselines on the medium and AntMaze suites (e.g., average medium score 90.2 versus 88.7 for the best diffusion-free baseline shown, and AntMaze average 77.7 versus 78.3 for QGPO)."],"supporting_citations":[{"why":"Supplies the variational Euler-equation tool used to prove Theorem B.3, the central PANI-to-NAMDP equivalence.","marker":"Gelfand et al. [2000]"},{"why":"IQL is the base offline algorithm that PANI is grafted onto, and the primary benchmark comparison for IQL-AN.","marker":"Kostrikov et al. [2021]"},{"why":"TD3 is the base off-policy algorithm that TD3-AN extends by modifying its critic update.","marker":"Fujimoto et al. [2018]"},{"why":"D4RL provides the benchmark datasets and normalized score protocol on which all empirical claims rest.","marker":"Fu et al. [2020]"},{"why":"ReBRAC supplies the extensively tuned baseline results used for fair comparison in Table 1.","marker":"Tarasov et al. [2024]"},{"why":"TD3+BC is the offline TD3 variant whose reported scores serve as a key comparison baseline.","marker":"Fujimoto and Gu [2021]"},{"why":"Supplies the lemma bounding expected-return differences between two MDPs, used in the NAMDP error bound (Theorem B.4).","marker":"Lee et al. [2020]"},{"why":"Diffusion-QL is a diffusion-based method that motivates PANI and serves as a high-performing comparison point.","marker":"Wang et al. [2022]"},{"why":"QGPO is the diffusion-based algorithm to which PANI is additionally applied, producing QGPO-AN.","marker":"Lu et al. [2023]"}],"fun_headline_variants":["Noise injection with penalty beats diffusion in offline RL","Penalized action noise defines a new MDP for offline RL","Simple noise trick improves offline RL without diffusion models","PANI: A lightweight offline RL boost via noise injection","Noisy actions with penalty: offline RL gains without diffusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that PANI solves the noisy action MDP assumes updates are chased toward the true value of the policy, but the implemented algorithms chase their own bootstrapped estimates, and the paper does not prove those two processes converge to the same answer.","fun_headline_variants_meta":{"raw":{"variants":["Noise injection with penalty beats diffusion in offline RL","Penalized action noise defines a new MDP for offline RL","Simple noise trick improves offline RL without diffusion models","PANI: A lightweight offline RL boost via noise injection","Noisy actions with penalty: offline RL gains without diffusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1974,"prompt_tokens":1095,"completion_tokens":879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":711,"completion_tokens_details":{"reasoning_tokens":799}},"tokens_in":711,"tokens_out":879,"duration_ms":8540,"temperature":1.0,"reasoning_tokens":799,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:33:04.679471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the reported IQL-AN and TD3-AN critic updates in a small tabular MDP with known dynamics, compute the NAMDP Q-function exactly from the paper's Definition 5.2, and compare the learned Q to it; if the bootstrapped-trained Q does not converge to the NAMDP fixed point, the theoretical foundation does not cover the empirical method.","supporting_citations":[],"review_version":1}