pith. sign in

arxiv: 2603.19470 · v3 · pith:JNRSKZFDnew · submitted 2026-03-19 · 💻 cs.LG · cs.AI

Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL

Chenlu Ye , Xuanchang Zhang , Yifan Hao , Zhou Yu , Ziji Zhang , Abhinav Gullapalli , Hao Chen , Jing Huang
show 1 more author
Tong Zhang
This is my paper

Pith reviewed 2026-05-21 10:27 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords adaptive layerwise perturbationoff-policy reinforcement learningLLM RLimportance samplingtraining-inference mismatchpolicy stabilityhidden state perturbationsexploration in language models
0
0 comments X

The pith

Injecting learnable perturbations into hidden states at each layer keeps updated LLM policies close enough to the inference policy to tame heavy-tailed importance ratios.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Off-policy reinforcement learning for large language models suffers from growing gaps between the policy used at inference and the one being updated, which produces heavy-tailed importance ratios, inflated gradients, and unstable training. The paper introduces Adaptive Layerwise Perturbation to address this by adding small learnable perturbations to the input hidden states of every transformer layer during the update step. The perturbed policy then serves as the numerator in the importance-sampling ratio while the original inference policy remains the denominator. Because the perturbations enlarge the set of reachable policies, the distribution flattens, the extreme tails of the ratio shrink, and the update stays within a more stable trust region. Experiments on math reasoning and multi-turn tool-use tasks show higher final performance together with the absence of ratio blow-ups and KL spikes that appear in the baseline.

Core claim

By injecting small learnable perturbations into the input hidden states of each layer during updates and using the resulting perturbed policy as the numerator of the importance ratio against the unchanged inference policy in the objective, ALP prevents the updated policy from deviating too sharply from the inference policy, flattens the distribution, reduces the tail of importance ratios, and maintains training stability while boosting exploration.

What carries the argument

Adaptive Layerwise Perturbation (ALP), a mechanism that adds controlled learnable noise to intermediate hidden-state representations at every layer so the effective policy family can cover inference-time mismatch without altering the inference policy itself.

If this is right

  • Final performance rises on both single-turn math problems and multi-turn tool-integrated reasoning tasks.
  • The tail of importance ratios no longer blows up and KL divergence stays flat across iterative updates.
  • Exploration improves because the flattened policy can safely reach a wider set of responses.
  • Perturbing hidden states at every layer outperforms both partial-layer and logits-only versions in ablations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same representation-level noise injection could be tested in other sequence models where training and serving distributions diverge for efficiency reasons.
  • If the perturbations prove robust, practitioners might safely increase update step sizes or reduce reliance on explicit ratio clipping.
  • The approach suggests that off-policy corrections can be moved from the output logits into earlier layers where the policy is still being formed.

Load-bearing premise

The added perturbations will systematically enlarge the reachable policy set in a way that covers the noise from inference-time techniques without creating new optimization instabilities or steering the final policy toward worse behaviors.

What would settle it

On the same math and tool-use tasks, run the identical training loop with and without the layerwise perturbations and measure whether the maximum and variance of importance ratios still spike above the same thresholds and whether KL divergence between consecutive policies still exhibits sudden jumps.

read the original abstract

Off-policy problems such as policy staleness and training--inference mismatch have become a major bottleneck for training stability and further exploration in LLM RL. The distribution gap between the inference and updated policies grows because of the techniques to enhance inference efficiency, leading to heavy-tailed importance ratios. Heavy-tailed ratios arise when the policy is locally sharp, which further inflates gradients and can push updates outside the trust region. To address this, we propose Adaptive Layerwise Perturbation (ALP), which injects small learnable perturbations into the input hidden states of each layer during updates and uses the resulting perturbed policy as the numerator of the importance ratio against the unchanged inference policy in the objective. Intuitively, by adding controlled noise to intermediate representations, ALP prevents the updated policy from deviating too sharply from the inference policy and enlarges the policy family to cover inference-time mismatch noise. Hence, the flattened distribution can naturally tighten the gap between the updated and inference policies and reduce the tail of importance ratios, thus maintaining training stability. This is further validated empirically. Experiments on single-turn math and multi-turn tool-integrated reasoning tasks show that ALP not only improves final performance, but also avoids blow-up in the importance-ratio tail and KL spikes during iterative training, along with boosted exploration. Ablations show that representation-level perturbations across all layers are most effective, substantially outperforming partial-layer and logits-only variants.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript proposes Adaptive Layerwise Perturbation (ALP) as a method to address off-policy problems such as policy staleness and training-inference mismatch in LLM RL. It injects small learnable perturbations into the input hidden states of each layer during updates and uses the resulting perturbed policy as the numerator of the importance ratio against the unchanged inference policy. This is claimed to enlarge the effective policy family, flatten distributions, reduce heavy-tailed importance ratios, prevent sharp deviations, maintain training stability, and boost exploration, with empirical validation on single-turn math and multi-turn tool-integrated reasoning tasks showing improved performance, avoidance of ratio blow-ups and KL spikes, and superior results from full-layer perturbations over partial or logits-only variants.

Significance. If the empirical results hold with proper quantification, ALP could provide a practical, representation-level mechanism for unifying off-policy corrections in LLM RL without altering inference-time behavior, potentially improving stability and exploration in iterative training regimes where distribution gaps arise from efficiency techniques.

major comments (1)
  1. [Abstract and Experiments] Abstract and Experiments: The central empirical claim of improved final performance, reduced importance-ratio tails, and avoided KL spikes rests on qualitative validation only; no effect sizes, statistical significance tests, ablation controls for perturbation optimization, or quantitative comparison metrics are reported, which is load-bearing for assessing whether the gains exceed standard baselines or variance.
minor comments (2)
  1. [Method] Clarify the exact form of the perturbed policy in the importance ratio (e.g., how the layerwise perturbations enter the forward pass and whether they are detached at inference).
  2. [Experiments] Add a brief discussion of computational overhead during training from the learnable perturbations and any hyperparameter sensitivity analysis.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and the recommendation for minor revision. We address the concern regarding the strength of the empirical claims below and will strengthen the quantitative support in the revised manuscript.

read point-by-point responses

  1. Referee: [Abstract and Experiments] Abstract and Experiments: The central empirical claim of improved final performance, reduced importance-ratio tails, and avoided KL spikes rests on qualitative validation only; no effect sizes, statistical significance tests, ablation controls for perturbation optimization, or quantitative comparison metrics are reported, which is load-bearing for assessing whether the gains exceed standard baselines or variance.

    Authors: We appreciate this observation. The current manuscript presents the empirical results primarily through training curves and histograms that visually demonstrate reduced tail mass in importance ratios, absence of KL spikes, and performance gains on math and tool-use tasks, with ablations comparing full-layer, partial-layer, and logits-only perturbations. However, we agree that formal effect sizes, statistical significance testing across random seeds, and explicit quantitative ablation metrics (e.g., mean performance deltas with standard errors) would make the claims more robust. In the revision we will add these: (i) performance tables reporting mean and standard deviation over at least three seeds with paired t-test p-values against baselines, (ii) quantitative summaries of importance-ratio quantiles and KL divergence statistics, and (iii) expanded ablation tables that include controls for whether perturbations are learned versus fixed. These additions will allow readers to assess whether improvements exceed variance. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper proposes ALP by explicitly introducing learnable perturbations to layer input hidden states, then using the resulting perturbed policy only in the importance-sampling numerator while keeping the inference policy fixed. This construction enlarges the effective policy family as a deliberate design choice rather than deriving from or redefining any fitted quantity or prior result. No equations reduce a claimed prediction to a fitted parameter by construction, no uniqueness theorem is imported via self-citation, and no ansatz is smuggled through prior work. The reported benefits (flatter ratios, stability, exploration) are presented as empirical consequences of the new mechanism and measured on external tasks, leaving the central argument self-contained against benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests on the untested premise that representation-level perturbations can be learned to cover inference mismatch without side effects; no free parameters, axioms, or invented entities are explicitly quantified in the abstract.

pith-pipeline@v0.9.0 · 5803 in / 1190 out tokens · 28337 ms · 2026-05-21T10:27:50.027693+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.