REVIEW 5 minor 9 references
Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success
T0 review · 0 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Success conditioning is not a heuristic; it exactly solves a trust-region policy optimization problem, with the improvement, the policy change, and the action-influence equal at every state.
desk verdict Gives success conditioning a precise optimization target with clean proofs; central claims hold, one minor overclaim about deterministic transitions needs correcting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the success-conditioned policy π+(a|s), defined by Bayesian conditioning of the behavior policy on the event that the episode succeeded. The key quantity is action-influence I(s) = (Std_{a∼π0} Q^{π0}(s,a) / E[Q^{π0}(s,a)])², which measures how much random variation in action choice affects success rate. The trust-region objective is the first-order performance-difference term L(π) = Σ_s d^{π0}(s) A^{π0}(s,π), and the trust region is measured by χ² divergence weighted by the success-conditioned occupancy. The proof works by rewriting both objective and constraint as expectations under the same distribution, applying Cauchy-Schwarz, and showing π+ achieves equality.
What would settle it
In a two-armed bandit with uniform behavior policy and arm success probabilities p and q, compute π+(a) = P(success|a)/Σ P(success|a'), then compare ρ(π+)-ρ(π0) with I = ((p-q)/(p+q))². The triple identity predicts they are equal; if they differ for any p,q in (0,1), the central claim is false.
Extended reading notes
Core claim
The central discovery is Proposition 4.4: for the success-conditioned policy π+(a|s)=P^{π0}(A_t=a | S_t=s, R(τ)=1), at every non-terminal state [A^{π0}(s,π+)]/V^{π0}(s) = χ²(π+(·|s) ∥ π0(·|s)) = I_{π0}(s), where I is the squared coefficient of variation of the Q-function under the action draw. This exact triple identity implies that π+ solves the trust-region problem maximizing L_{π0}(π) subject to Σ_s d+_{π0}(s) χ²(π(·|s)∥π0(·|s)) ≤ Σ_s d+_{π0}(s) I_{π0}(s). The proof rewrites both objective and constraint as expectations under a common distribution, applies Cauchy-Schwarz, and shows π+ attains the upper bound. As a corollary, success conditioning never decreases the true success probabilit
Load-bearing premise
The whole construction assumes the observed trajectories were generated by a single time-homogeneous Markov policy whose state includes enough history, and that success has positive probability from every non-terminal state; if V(s)=0 for some reachable state, the ratios defining action-influence and the identity are undefined.
Editorial extensions
If this is right
- Exact success conditioning is guaranteed not to reduce the success probability (Corollary 4.5); the only way it can fail to help is by barely changing the policy when action-influence is tiny.
- At every state, the χ² distance between the learned policy and the behavior policy is exactly equal to the relative improvement, so observable policy movement during training is an exact diagnostic of latent improvement.
- The χ² trust region permits concentrating probability mass on well-supported actions, unlike KL-style updates that forbid dropping actions; instead it penalizes rare-action exploration, with allowed mass scaling as √δ rather than 1/log(1/δ).
- When state transitions are deterministic, the distribution-shift factor M equals 1, so minimizing next-action prediction loss on successful trajectories directly aligns with deployment success.
- Return thresholding acts as a proxy success criterion whose improvement relative to faithful conditioning equals the ratio of proxy to true action-influence times their advantage correlation; it can outperform faithful conditioning but can also degrade performance when the proxy selects for volatile outcomes.
Reading between the lines
- If the identity holds, practitioners can use χ² divergence between the fine-tuned model and the base model as a cheap, training-time proxy for how much the filtered data will improve deployment success—without needing value estimates.
- The theory predicts that success conditioning will be most effective precisely where the base policy's action choices matter most; in domains where diverse actions lead to similar outcomes, the method will stall, suggesting a principled reason to couple it with exploration or action-influence amplification such as action chunking.
- Iterating success conditioning may converge to a policy that assigns all mass to one well-supported action per state; the paper notes progress can stall, so a testable extension is to check whether repeated filtering decreases χ² movement by a predictable factor.
- The proxy-reward formula provides a quantitative criterion for choosing a return threshold: pick the threshold that maximizes the product of the proxy's coefficient of variation and its advantage correlation with the true reward, rather than the highest achievable threshold.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper gives an exact optimization-theoretic characterization of success conditioning. In an episodic MDP with a binary success event, the success-conditioned policy π+(a|s) = P^{π0}(A_t=a | S_t=s, R(τ)=1) is shown to be an optimal solution of the trust-region problem max_π Σ_s d^{π0}(s) A^{π0}(s,π) subject to Σ_s d^+_{π0}(s) χ²(π(·|s)||π0(·|s)) ≤ Σ_s d^+_{π0}(s) I_{π0}(s), where I_{π0}(s) is the squared coefficient of variation of Q^{π0}(s,·). Proposition 4.4 proves the per-state identity A^{π0}(s,π+)/V^{π0}(s) = χ²(π+(·|s)||π0(·|s)) = I_{π0}(s), and Corollary 4.5 derives monotone policy improvement. The paper then discusses finite-data estimation (Prop 5.1), proxy rewards and return thresholding (Prop 6.1), and compares the induced χ² trust region with KL-based TRPO/MDPO. The central proofs are elementary, exact, and I did not find a gap in them.
Significance. The contribution is significant. It provides, to my knowledge, the first precise answer to the question of what optimization problem success conditioning solves, unifying rejection-sampling/SFT, return-conditioned models, and RvS under one trust-region framework. The trust-region radius is not a user-chosen hyperparameter but is determined by the behavior policy's value function, and the action-influence identity turns the observable χ² distance between π+ and π0 into a diagnostic of improvement. The proof is fully checkable and there are no fitted constants or circular arguments. The main caveat is the explicit modeling assumption that success is possible from every non-terminal state, which is required for the ratios A/V and I to be well-defined; this is acknowledged in Section 2 but could be stated more prominently in the abstract and introduction.
minor comments (5)
- [Section 5, after Prop 5.1] The claim 'M=1 whenever state transitions are deterministic' is not correct. Under deterministic transitions but a stochastic initial distribution, d^+_{π0}(s) = (V^{π0}(s)/ρ(π0)) d^{π0}(s) whereas d_{π+}(s) is the occupancy under π+ starting from the original μ; the initial-state posterior reweighting generally makes M > 1. For example, with two deterministic initial states having success probabilities 1 and 0.1 under uniform μ, M ≈ 1.82. The correct statement is 'M=1 when the initial state is deterministic' (or when the initial distribution is also conditioned on success).
- [Appendix B / Section 4.5.2] Numbering typos: Appendix B refers to 'Proposition 3.2' but the relevant identity is Proposition 4.4; Section 4.5.2 refers to 'Lemma 4.3' when comparing χ² and KL budgets, but the result is Lemma 4.6.
- [Figure 1] The bottom panel label '2 Trust Region (Success Conditioning)' is missing the symbol χ; it should read 'χ² Trust Region (Success Conditioning)'.
- [Section 2] The assumption that success is possible from every non-terminal state is load-bearing for the main identities, since V^{π0}(s) appears in denominators. The paper states this assumption, but the abstract and Section 1.1 could more explicitly flag it as a scope condition, especially because many practical offline datasets contain states with zero empirical success probability.
- [Section 6.3] The stylized bandit example is illustrative rather than a proof. It may be worth labeling it as such, since the figure caption and surrounding text could otherwise be read as empirical validation of the theoretical claims.
Circularity Check
No significant circularity: the trust-region and action-influence identities are derived by theorem, not fitted or self-referential.
full rationale
The derivation is self-contained and non-circular. Lemma A.1 is a Bayes-rule identity expressing the success-conditioned policy as a likelihood ratio 1 + A^{pi0}(s,a)/V^{pi0}(s). Proposition 4.1 then proves, via Cauchy-Schwarz, that this policy maximizes the linearized objective L^{pi0}(pi) subject to the chi-squared constraint; the radius sum_s d+_{pi0}(s) I_{pi0}(s) is computed from pi0's Q-function, not chosen by fitting pi+, so the trust-region problem is not constructed to encode the answer. Proposition 4.4 is an algebraic corollary: substituting Lemma A.1 gives E_{pi+}[A/V] = Var_{pi0}(A/V)/V^2 and chi^2(pi+||pi0) = Var_{pi0}(A/V)/V^2, which equals the definition of I_{pi0}(s). The triple equality is therefore a theorem, not an equivalence by definition of the trust-region radius. Corollary 4.5 follows from Bellman monotonicity. No fitted constants are renamed as predictions. The only self-citation, Bhandari & Russo (2024), appears in a survey sentence about convergence theory and is not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. Section 8 openly states the scope limits (finite data, function approximation, iterated success conditioning), which reinforces rather than conceals the theoretical nature of the claim. One non-circular correctness caveat: the statement in Section 5 that 'M=1 whenever state transitions are deterministic' is too strong when the initial-state distribution is stochastic and V^{pi0}(s0) varies across initial states; that is a correctness issue, not a circularity.
Assumptions & free parameters
assumptions (7)
- domain assumption Episodic MDP with a fixed Markovian behavior policy π0; transitions follow a fixed kernel P and initial state distribution μ (Section 2).
- domain assumption Success is possible from every non-terminal state under π0: V^{π0}(s) > 0 (Section 2).
- standard math Performance difference lemma (Kakade & Langford 2002), used in Sections 4.1 and Appendix B.
- standard math Bellman operator monotonicity (Bertsekas 2011), used in the proof of Corollary 4.5.
- standard math Bayes' rule and Cauchy-Schwarz (proofs of Lemma A.1 and Prop 4.1).
- domain assumption Bernoulli labeling reduction: R(τ) ~ Bernoulli(Y(τ)) preserves the expected objective and lets the results apply to dense rewards (Section 6.1).
- domain assumption The main theorems characterize the exact conditioning target π+; finite-data and function-approximation effects are addressed only by Prop 5.1, which requires M = sup_s d+_{π0}(s)/d^{π+}(s) to be finite (Section 8).
Cite this review
Pith. "Pith review of Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success." pith.science (2026). https://pith.science/paper/DLZRE7UO
@misc{pith2026260118175,
author = {Pith},
title = {Pith review of: Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success},
year = {2026},
howpublished = {\url{https://pith.science/paper/DLZRE7UO}},
note = {Machine review of arXiv:2601.18175}
}
abstract
A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories. This principle appears under many names -- rejection sampling with SFT, goal-conditioned RL, Decision Transformers -- yet what optimization problem it solves, if any, has remained unclear. We prove that success conditioning exactly solves a trust-region optimization problem, maximizing policy improvement subject to a $\chi^2$ divergence constraint whose radius is determined automatically by the data. This yields an identity: relative policy improvement, the magnitude of policy change, and a quantity we call action-influence -- measuring how random variation in action choices affects success rates -- are exactly equal at every state. Success conditioning thus emerges as a conservative improvement operator. Exact success conditioning cannot degrade performance or induce dangerous distribution shift, but when it fails, it does so observably, by hardly changing the policy at all. We apply our theory to the common practice of return thresholding, showing this can amplify improvement, but at the cost of potential misalignment with the true objective.
Figures
Reference graph
Works this paper leans on
-
[8]
H. Touvron, L. Martin, K. Stone, P. Albert, A. Alma- hairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,
-
[9]
The first equality is a definition of Lπ0 (π+)
X s d+ π0 (s)Iπ0 (s). The first equality is a definition of Lπ0 (π+). The second is the formula for the success-conditioned occupancy mea- sure in Lemma A.2. Dividing each side by the success- probability under the behavior policy, ρ(π0), yields the result. We’ve focused in the Iπ0 expression, since an identical argument establishes the claim for theχ2 ex...
2011
-
[2011]
Bhandari and D
J. Bhandari and D. Russo. Global optimality guarantees for policy gradient methods.Operations Research, 72 (5):1906–1927,
1906
-
[2015]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[2017]
R. K. Srivastava, P. Shyam, F. Mutz, W. Ja ´skowski, and J. Schmidhuber. Training agents using upside- down reinforcement learning.arXiv preprint arXiv:1912.02877,
arXiv 1912
-
[2019]
S. K. S. Ghasemipour, A. Wahid, J. Tompson, P. Sanketi, and I. Mordatch. Self-improving embodied foundation models.arXiv preprint arXiv:2509.15155,
-
[2022]
S. Levine. Reinforcement learning and control as proba- bilistic inference: Tutorial and review.arXiv preprint arXiv:1805.00909,
-
[2023]
J. Schmidhuber. Reinforcement learning upside down: Don’t predict rewards – just map them to actions.arXiv preprint arXiv:1912.02875,
arXiv 1912
Show all 9 references
-
[2025]
Grattafiori, A
9 A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Ka- dian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.