{"id":"c48a3e7c-0b4a-421f-a89e-2bb23999a486","arxiv_id":"2608.02951","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SP3O is a reward-model-free, critic-free, gradient-based RL algorithm that optimizes policies from segment-level preferences in stochastic MDPs via off-policy importance sampling and PPO-style clipping.","lead":"The paper introduces SP3O, an algorithm that trains policies directly from pairwise preferences over short trajectory segments, without fitting a reward model or a value critic. It targets stochastic reinforcement learning environments where segment feedback is easier for humans to provide than full-trajectory comparisons.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SP3O's Theorem 1 requires pi_ref = pi_theta and discounted state sampling, but the implemented algorithm deliberately violates both; the estimated gradient is then the gradient of a different segment-MDP objective, with no error bound.","rationale":"The reader's weakest assumption was the preference oracle model in Eq. 4 and its implementation via true future reward. That is a real concern, but the more decisive issue is internal: Theorem 1's condition pi_ref = pi_theta is not satisfied by the algorithm's own reference-update schedule, and the implementation explicitly replaces discounted state sampling with uniform sampling (C.1.2) and fixes the reference to the base model in the LLM experiments (C.2.1). So even if a human oracle perfectly matched Eq. 4, the theoretical guarantee would still not cover the executed algorithm. The theorem is plausible at a single point, but no analysis bounds the bias when the policy moves away from the reference. This is consistent with the reader's conditional recommendation, so I would not change the verdict, only sharpen the reason: the missing condition is not mainly about oracle realism but about the mismatch between the theorem's stationary-point identity and the algorithm's multi-step, stale-reference updates. The proposed tabular test can settle whether the mismatch is numerically significant or merely formal. The paper's empirical results and ablations are valuable, and the theoretical idea is interesting, but the theory-practice gap should be closed before the central claim is taken as established.","tokens_in":20609,"tokens_out":17241,"duration_ms":156392,"concrete_test":"Use a small tabular MDP (e.g., 2 states, 2 actions) with known transition and reward. Fix a reference policy pi_ref, pick several current policies pi_theta with increasing KL(pi_theta || pi_ref), and compute exactly (a) the expected gradient of the unclipped SP3O loss in Eq. 6 when D is generated by the oracle model Eq. 4, and (b) (1 - gamma^L) times the true policy gradient grad_theta J_M(pi_theta) from the policy gradient theorem. If the angle or relative error between (a) and (b) grows as pi_theta moves away from pi_ref, the pi_ref = pi_theta assumption is load-bearing; if they remain proportional, the stale-reference violation is not fatal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Theorem 1: when pi_ref = pi_theta, the gradient of the original MDP objective is a scaled version of the gradient of the segment-MDP value difference. The proof in Appendix D.4 hinges on d1_0 not depending on theta, which is valid only if pi_ref is held fixed while differentiating. That is the correct setup for one gradient step starting at theta = theta_ref. But Algorithm 1 refreshes theta_ref only every T inner updates (line 12), so for updates t=2..T the data-generating policy differs from the current policy, and the terminal Q in the segment MDP is Q^{pi_ref}, not Q^{pi_theta}. The loss in Eq. 6 still computes an unbiased gradient of J_M1(pi_theta) - J_M1(pi_{t-1}) for the fixed reference M1, but that quantity is proportional to grad J_M(pi_theta) only at the point pi_ref = pi_theta; no bound quantifies the discrepancy away from that point. The paper itself flags that the practical algorithm deviates: Appendix C.1.2 states discounted state sampling is replaced by uniform state sampling, and Appendix C.2.1 states the LLM experiments keep the base model as the reference despite sampling from the current policy. Thus the theory does not justify the algorithm as implemented, and even a perfect oracle following Eq. 4 would not cure this gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SP3O, a reward-model-free and critic-free policy-gradient method for stochastic MDPs that learns from pairwise preferences over short trajectory segments. The authors introduce a segment MDP whose terminal reward is the Q-function of a reference policy, and prove (Theorem 1) that when the reference policy equals the current policy, the policy gradient of the original MDP objective equals a scaled gradient of the segment-MDP value difference. They also provide a Hoeffding-style error bound (Proposition 1) to motivate a segment-length tradeoff. The algorithm is evaluated against Online DPO, P3O, and ZPG in MuJoCo control tasks and an LLM finetuning task, reporting improved performance especially in long-horizon settings.","tokens_in":20924,"tokens_out":13038,"duration_ms":112076,"significance":"If the theoretical claims held for the algorithm as implemented, this would be a meaningful contribution: it would give the first gradient-based, first-order PbRL method for general stochastic MDPs with segment feedback that avoids both reward models and critics. The segment-MDP construction with a terminal Q-reward is an elegant idea that directly addresses the known limitation of partial-return preference models. The paper is clearly written, provides full proofs in the appendix, and its experiments are extensive (including a large-seed ablation study). The main weakness is a substantial gap between the theorem's assumptions and the implemented algorithm, which is partially acknowledged in the appendix but not resolved or bounded.","major_comments":[{"comment":"","section":"Section 3 (Algorithm 1), Section 4.1 (Theorem 1), Appendix C.1.2, C.2.1"},{"comment":"","section":"Section 4.2 (Proposition 1) and Eq. (6)"},{"comment":"","section":"Section 5.2 and Appendix C.2.1"}],"minor_comments":[{"comment":"","section":"Section 4.2 and Appendix D.5"},{"comment":"","section":"Lemma 1 (Section 3)"},{"comment":"","section":"Theorem 1 (Section 4.1)"},{"comment":"","section":"Section 3 and Lemma 1"},{"comment":"","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The core theorem appears correct under its stated assumptions, and the segment-MDP idea is clever. The main concern is that the implemented algorithm deliberately violates two key assumptions of Theorem 1 (lagging reference policy and non-discounted state sampling), and no approximation bound is supplied. This is a load-bearing gap for the paper's central claim that the theory supports the algorithm. In revision, the authors should either extend the analysis to bound the lagging-reference and sampling-distribution bias, or explicitly reframe the theory as a motivation for the loss design rather than as a guarantee for the implemented method. The LLM experiments should also be aligned with the proposed preference model or clearly reported as evaluating a variant."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [name],\n\nShort version: SP3O is a real contribution—the first reward-model-free, critic-free, first-order PbRL method for stochastic MDPs with segment preferences from arbitrary start states. But the central theorem is proven exactly at pi_ref = pi_theta, and the algorithm deliberately leaves that point, so the theory as written doesn't cover the implementation. Also, the simulated \"human\" oracle in the robot experiments gets the true future return to compute the terminal Q estimate. Both are fixable, and the paper is honest about the first gap, so I'd send it to review.\n\nWhat's new and good: The segment-MDP equivalence in Theorem 1 is neat, and Lemmas 2–3 are clean. Combining P3O clipping with off-policy IS on that segment MDP is a sensible and previously missing combination. The segment-length tradeoff (Prop 1) gives a real design insight, and the 4000-seed ablation is consistent with it. Experiments are thorough for the subfield: 100 seeds per point, GPT-J LoRA finetuning, comparisons against Online DPO, P3O, and ZPG. Credit where due: the authors disclose in the appendix that they use uniform state sampling (C.1.2) and a fixed reference in the LLM experiments (C.2.1).\n\nSoft spots: (1) The stress-test concern holds. Theorem 1 requires pi_ref = pi_theta; Algorithm 1 updates theta T times before refreshing theta_ref, so for t=2..T, the reference is lagging and there is no bound on the resulting bias in the estimated gradient. The proof's interchange of derivative and initial distribution is only valid at the exact point. This is a real theory-practice gap. It's not fatal—small learning rates may make it benign—but the paper needs either a bound or a revised theorem. (2) The preference oracle in Eq. (4) is strong: it assumes the human can evaluate Q_pi_ref at the segment endpoint. In the control experiments, they implement pQ with the true discounted future return, which no human would have. So the empirical results likely overstate what you'd get from real segment feedback. This is partially discussed, but the paper still claims \"human evaluators\" benefits. (3) No code/data uploaded despite a \"code appendix\" mention; that's a release issue, not a scientific one.\n\nRecommendation: I'd accept it for peer review with high confidence. The core idea is sound, the empirical work is solid, and the gaps are clearly identifiable and addressable. I'd push the authors to close the lagging-reference gap, revisit the oracle assumptions in the narrative, and release the code.","headline":"SP3O is a genuinely new first-order, reward-model-free PbRL method from segment preferences, but the central theorem is proven only at pi_ref = pi_theta and the simulated oracle is too well-informed; both gaps are fixable.","tokens_in":21424,"tokens_out":4862,"would_cite":true,"duration_ms":43027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that policy gradients in a stochastic MDP can be recovered, up to a constant, from pairwise preferences between short trajectory segments, and builds SP3O, a reward-model-free and critic-free algorithm on that identity.","keywords":["preference-based reinforcement learning","segment preferences","reward-model-free","policy gradient","PPO","stochastic MDP","reinforcement learning from human feedback","off-policy importance sampling"],"falsifier":"Run SP3O in a stochastic MDP with a known true reward, but replace the preference oracle with one that follows the standard partial-return model, where preferences depend only on the reward sum inside the segment and ignore the quality of the final state. If the policy-gradient estimate is biased and learning degrades in an environment where final-state quality matters, such as goal reaching, that would confirm the method's validity depends on the Eq-4 oracle model rather than on segment feedback alone.","tokens_in":20383,"feed_emoji":"🤖","tokens_out":7621,"duration_ms":58687,"temperature":0.7,"pith_summary":"The paper introduces SP3O, a preference-based reinforcement learning algorithm that trains a policy directly from pairwise comparisons of short trajectory segments, in general stochastic environments, without ever learning a reward model or a value critic. Its central assertion is that segment-level preferences contain enough information to compute first-order policy updates: a policy gradient in the original MDP equals, up to a constant, the gradient of a value difference in an auxiliary segment MDP whose terminal reward is the Q-function of the reference policy. If correct, this closes a gap: prior reward-model-free methods either assumed deterministic dynamics or used gradient-free optimization, and trajectory-level feedback becomes impractical for long horizons. The paper backs the claim with a theorem, an error bound on segment length, and experiments in robotic control and LLM fine-tuning showing SP3O improves as horizons grow.","feed_headline":"No reward model needed: policy gradients from segment preferences","feed_subtitle":"Short segment comparisons replace reward engineering, and they improve as horizons lengthen.","key_machinery":"The load-bearing object is the segment MDP $M_1$: a discounted finite-horizon MDP of length $L$ whose per-step reward is the original reward for the first $L-1$ steps and $r'_L(s,a) = Q^{\\pi_{\\text{ref}}}_M(s,a)$ at the terminal step, with initial distribution the discounted occupancy measure of the reference policy, $d'_0(s) = (1-\\gamma^L)\\sum_{k\\ge 0} \\gamma^{kL} \\Pr[s_{kL+1}=s \\mid \\pi_{\\text{ref}}]$. Lemmas 2 and 3 establish that under $\\pi_{\\text{ref}}$ the Q-function and discounted state occupancy match those of the original MDP; Theorem 1 then converts the policy gradient of $J_M$ into a scaled gradient of a value difference $J_{M_1}(\\pi_\\theta) - J_{M_1}(\\pi_1)$. This is what lets segment comparisons, passed through the inverse-logistic estimator $D$ and a clipped importance-weighted PPO-style loss, act as a first-order policy gradient signal in a general stochastic MDP.","core_discovery":"The paper's core discovery is an identity (Theorem 1): when the reference policy used to collect segments is the current policy, the gradient of the expected return in the original infinite-horizon discounted MDP is a scaled gradient of a value difference in a constructed finite-horizon segment MDP, $\\nabla_\\theta J_M(\\pi_\\theta) = \\frac{1}{1-\\gamma^L} \\nabla_\\theta (J_{M_1}(\\pi_\\theta) - J_{M_1}(\\pi_1))$ for any policy $\\pi_1$. The segment MDP grants rewards $r(s,a)$ at its first $L-1$ steps and ends with a terminal reward equal to $Q^{\\pi_{\\text{ref}}}_M(s_L,a_L)$, the reference policy's Q-function at the segment's final state-action pair; its initial state distribution is the discounted occupancy of the reference policy. The proof shows that the Q-function and discounted occupancy of the reference policy coincide in the two MDPs (Lemmas 2 and 3), so the policy gradient theorem applies. This identity lets SP3O estimate the gradient from segment preference feedback: preferences are assumed to follow a logistic model on segment return plus the noisy terminal Q-value, the inverse logistic function yields an estimate $D$ of the return difference, and a PPO-style clipped, off-policy importance-weighted loss converts $D$ into a policy update. The same machinery yields a segment-length tradeoff: too short a segment inherits the oracle's bounded terminal-Q noise, too long a segment inflates variance for a fixed feedback budget.","pith_inferences":["A natural testable extension would replace the terminal Q-term with a learned or estimated value bootstrap when the Q-function of the reference policy is unknown; the theory suggests the estimator remains valid to the extent that bootstrap tracks $Q^{\\pi_{\\text{ref}}}_M(s_L,a_L)$.","The segment-MDP identity suggests a bridge to off-policy evaluation: any algorithm that can estimate $J_{M_1}$ differences from segment data inherits a first-order estimate of the original policy gradient, so segment-level reward-model-free critic architectures could be built on the same construction.","If real human preferences follow the partial-return model rather than the Eq-4 model, SP3O's $D$ is biased; an informative comparison would measure preference models on human data, for instance whether final-state quality dominates choice in goal-reaching tasks as the paper argues.","The horizon-dependent expertise factor used in the control experiments (0.1 for SP3O versus $10/H$ for baselines) is a practical calibration choice; an ablation varying it could separate algorithmic advantage from oracle-calibration effects."],"forward_implications":["Segment-level preference feedback is sufficient for first-order policy optimization in stochastic MDPs; the preference query budget no longer needs to scale with trajectory length.","The gradient estimator is unbiased when the reference policy is the current policy and the oracle follows the paper's Eq-4 preference model, with PPO-style clipping providing stability in off-policy updates.","There is a provable tradeoff in segment length: the optimal $L$ balances oracle terminal-Q noise $\\gamma^{L-1}\\nu$ against variance $\\frac{1}{1-\\gamma}\\sqrt{L\\log(2/\\delta)/N}$, and experiments show the best $L$ grows with the oracle budget $N$.","Because $\\pi_1$ in Theorem 1 is arbitrary, the update can compare the current policy against a slightly older policy or any similar reference, which the loss function exploits by using $\\pi_{\\theta_{t-1}}$ as the comparison policy.","Longer horizons help rather than hurt SP3O, since segments extract more preference signal per evaluation; the control and LLM experiments show this advantage growing with the horizon."],"supporting_citations":[{"why":"Supplies the policy gradient theorem and the value and occupancy definitions that Theorem 1 builds on.","marker":"Sutton et al., 1998"},{"why":"P3O, the pairwise proximal policy optimization loss that SP3O generalizes from contextual bandits to stochastic MDPs with segments.","marker":"Wu et al., 2024"},{"why":"Provides the motivation and the regret-based preference model that accounts for final-state quality, which the paper's Eq-4 model adapts.","marker":"Knox et al., 2022"},{"why":"Baseline reward-model PbRL with segment feedback; the setting SP3O removes the reward model from.","marker":"Christiano et al., 2017"},{"why":"Online DPO baseline used in the control and LLM experiments to benchmark SP3O.","marker":"Guo et al., 2024"},{"why":"ZPG baseline, the zeroth-order gradient-free approach SP3O is compared against as a slower-converging alternative.","marker":"Zhang and Ying, 2025a"},{"why":"PPO clipping mechanism that the SP3O loss function adapts for stability.","marker":"Schulman et al., 2017"}],"fun_headline_variants":["SP3O: Direct policy gradients from segment preferences","No reward model, no critic: SP3O learns from segments","Segment preferences to policy gradients, reward-free","RL from segment feedback: gradient-based, no reward model","SP3O: Use short segments, skip reward modeling entirely"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole construction assumes that when people compare two short segments, they judge them by the rewards collected inside the segment plus how good the final situation is under the current policy, and that this judgment is only mildly noisy; if real preferences work differently, the estimated reward differences and the gradients built from them are biased.","fun_headline_variants_meta":{"raw":{"variants":["SP3O: Direct policy gradients from segment preferences","No reward model, no critic: SP3O learns from segments","Segment preferences to policy gradients, reward-free","RL from segment feedback: gradient-based, no reward model","SP3O: Use short segments, skip reward modeling entirely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000504,"raw_usage":{"total_tokens":2534,"prompt_tokens":1093,"completion_tokens":1441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":1361}},"tokens_in":709,"tokens_out":1441,"duration_ms":9831,"temperature":1.0,"reasoning_tokens":1361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:54:36.672207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SP3O in a stochastic MDP with a known true reward, but replace the preference oracle with one that follows the standard partial-return model, where preferences depend only on the reward sum inside the segment and ignore the quality of the final state. If the policy-gradient estimate is biased and learning degrades in an environment where final-state quality matters, such as goal reaching, that would confirm the method's validity depends on the Eq-4 oracle model rather than on segment feedback alone.","supporting_citations":[],"review_version":1}