{"id":"6aabf238-071b-409b-870e-a256b51c5b73","arxiv_id":"2607.04728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SIS converts many off-policy tokens to unit-weight on-policy tokens via rejection sampling, tightening the token-vs-sequence IS gap and improving LLM RL post-training as a low-overhead plug-in.","lead":"The paper introduces Selective Importance Sampling (SIS), a plug-in that uses token-level rejection sampling to treat many off-policy tokens as on-policy during LLM RL post-training. It reports consistent gains and better stability across GRPO, DAPO, and GSPO on dense and MoE models for math and agent tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Top-K envelope (Prop. 2) leaves residual TV error ξ_K, so unit weights on accepted tokens are only approximately unbiased, undermining the exact on-policy certificate of Prop. 1 that the central claim relies on.","rationale":"The reader correctly isolates the top-K certificate as the weakest link supporting the “on-policy with unit weight” half of the strongest claim. Prop. 3’s reduction of D holds regardless of approximation quality, yet the paper’s narrative that this conversion improves the off-policy estimator (and therefore alignment) inherits the exactness assumption of Prop. 1. Empirical robustness to K (Table 3) and the small residuals plotted in Fig. 2(B) supply evidence that the concern may not materialize in the reported regimes, which is why the verdict remains CONDITIONAL rather than being lowered; a direct measurement of ξ_K under the training distribution would decide the issue. No deeper internal contradiction was found—the theory is narrowly and honestly scoped to the token–sequence gap, the plug-in nature is clear, and the stress-test gains (staleness, MoE+R3, clip-free) stand independently.","tokens_in":24374,"tokens_out":674,"duration_ms":42590,"concrete_test":"From a mid-training checkpoint of the Qwen3-8B GRPO+SIS math run, evaluate mean and 95th-percentile ξ_K = ∑_{v ∉ V_K} π_θ(v | x, y_<t) over every token of a large validation batch of rollouts, using the exact K=10 of the main experiments. If mean ξ_K > 0.02 or the 95th percentile > 0.10, residual mass is large enough that the unit-weight “zero-bias” justification fails for the estimator actually optimized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim’s theoretical half asserts that accepted tokens may safely receive unit importance weight (zero bias) because they are on-policy. This holds exactly only under Prop. 1. The implemented estimator uses the top-K envelope of Prop. 2 / Alg. 1, under which accepted tokens follow the conditional π_θ(· | v ∈ V_K) and D_TV(· ‖ π_θ) = ξ_K exactly. Consequently e_w_t = 1 is unbiased solely for the conditional, not the true target; an additional bias of order ξ_K (plus the fact that E[e_w_t | y_t] ≠ w_t even for exact rejection) enters the token-level gradient. Prop. 3 still tightens the D-bound pathwise, but the interpretive claim that SIS “turns off-policy tokens on-policy” and thereby improves the estimator relative to true ∇J rests on ξ_K being negligible. The paper only shows this for a few snapshots in Fig. 2(B) and does not propagate the residual into the gradient-gap bound or the policy-gradient bias under the long-horizon, high-drift regimes SIS is intended to handle.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Selective Importance Sampling (SIS), a plug-in for off-policy RL post-training of LLMs. Viewing the behavior policy as a proposal, SIS runs a token-level acceptance–rejection test: accepted tokens are treated as on-policy and receive unit importance weight, while rejected tokens keep the usual IS ratio. An efficient top-K envelope approximates the rejection constant. The authors prove that the pathwise total log-importance deviation D that bounds the sequence-level vs. token-level gradient gap is reduced (Thm. 1, Prop. 3), and show consistent accuracy and stability gains when SIS is plugged into GRPO, DAPO, and GSPO on dense and MoE Qwen3 models (and Llama-3.2) across math and agentic-search benchmarks, including under rollout staleness and MoE train–inference mismatch.","tokens_in":24716,"tokens_out":1609,"duration_ms":19970,"significance":"If the claims hold, SIS is a useful, low-overhead, algorithm-agnostic stabilization tool for the dominant “rollout then update” LLM RL stack. Strengths include: (i) a clean classical derivation from rejection sampling and IS factorization rather than ad-hoc clipping; (ii) a plug-in interface that only rewrites the token ratio and composes with existing objectives; (iii) elementary but relevant theory linking reduced D to a tighter sequence–token gap; (iv) broad empirical coverage (dense/MoE, math/agent, multiple base algorithms, staleness and MoE routing stress tests) with reported ~1% wall-clock overhead. The work is of practical interest to the RLHF/post-training community even if the exact on-policy certificate is only approximate under the implemented top-K estimator.","major_comments":[{"comment":"§3.1–3.2, Prop. 1 vs. Prop. 2 / Alg. 1: The central interpretive claim that accepted tokens “are already on-policy” and that unit weight “introduces zero bias” is exact only under Prop. 1 (full-vocabulary Mt). The implemented estimator uses the top-K envelope of Prop. 2, under which accepted tokens follow the conditional πθ(· | v ∈ VK) with D_TV = ξ_K exactly. Unit weight is therefore unbiased only for that conditional, not for πθ. The paper shows ξ_K is small in a few snapshots (Fig. 2B) but does not propagate residual mass into a bias bound for the token-level gradient (6) or for long-horizon / high-drift regimes that motivate SIS. Please either (a) extend Thm. 1 / Prop. 3 with an explicit O(ξ_K) (or similar) remainder for Alg. 1, or (b) restate all “zero bias / on-policy certificate” claims as holding only for the exact procedure and quantify approximation error for the implemented me","section":"§3.1–3.2, Prop. 1–2, Alg. 1"},{"comment":"§3.1, Eq. (8)–(10) and the claim after Prop. 1 that plugging ewt into (6) “results in identical gradient estimation”: even under exact rejection, E[ewt | yt] = (wt/Mt)·1 + (1−wt/Mt)·wt ≠ wt in general. The estimator is therefore not the same as standard token-level IS in expectation; it is a stochastic mixture of unit-weight and IS terms. The manuscript should state precisely what quantity is unbiased (or approximately unbiased): the true policy gradient ∇J, the sequence-level IS gradient, or only the token-level surrogate under a conditional law. Without this, the link from “D_SIS ≤ D” (Prop. 3) to improved approximation of ∇J remains incomplete, because Prop. 3 only tightens the sequence–token gap for a fixed realization of ratios, not the bias of the SIS mixture relative to on-policy sampling.","section":"§3.1, Eqs. (8)–(10); §3.3, Prop. 3"},{"comment":"§4.2–4.3 and Tables 1–2 / Fig. 3–4: Empirical gains are consistent and the staleness/MoE stress tests are valuable, but several load-bearing comparisons lack uncertainty quantification (no multi-seed means/std, no statistical tests). For claims of “consistent improvements” and “substantially stronger robustness,” especially the large MoE math lifts (e.g., GRPO +6.37 Avg) and the collapse-vs-recovery narrative in Fig. 4, please report at least 2–3 seeds or error bars, and clarify whether evaluation (Avg@1 / Avg@32) uses a fixed decoding seed. Without this, it is hard to separate method effect from run-to-run variance in LLM RL.","section":"§4.2–4.3, Tables 1–2, Fig. 3–4"}],"minor_comments":[{"comment":"Abstract and §1: several grammatical issues (“a efficient paradigm”, “can be combine with a vast vary of”, “while the token-level ratios compound… variance exploded”). A careful language pass would help.","section":"Abstract, §1"},{"comment":"Fig. 2 caption and panels: “Accpet Rate” typo; panel labels A/B/C are useful but the residual-mass plot should state the model and step at which ξ_K is measured.","section":"Fig. 2"},{"comment":"Table 4: SIS top-K differs by algorithm (10/30/50); §4.4 claims robustness to K but main results use different K per baseline. State the selection rule or unify K for fairer plug-in comparison.","section":"Table 4, §4.4"},{"comment":"Eq. (15) and Appendix E.2: GSPO+SIS sequence ratio (40) uses a stop-gradient on πθ for accepted tokens; a one-sentence justification would clarify why that form preserves the intended sequence-level semantics.","section":"Eq. (15), App. E.2"},{"comment":"Related work §5: briefly contrast SIS with per-token masking / IcePop-style filters, since those also zero or downweight off-policy tokens rather than converting them.","section":"§5"},{"comment":"Notation: w(θ), wt(θ), ewt(θ), and wi,t(θ) are clear, but Mt vs. cMt and z_t vs. A in the appendix proof of Prop. 2 should be unified.","section":"§3, App. A.3"}],"recommendation":"major_revision","confidential_remarks":"The idea is incremental but practically relevant; the main risk is overclaiming an exact on-policy certificate for an approximate top-K estimator. If the authors tighten the theory–implementation gap and add minimal multi-seed reporting, this is a solid contribution for a methods-oriented venue. Scope fits cs.CL / LLM post-training well. No concerns about circular evaluation: benchmarks are external."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit here is not a new theory of off-policy RL. It is a cheap, composable trick: run a token-level accept/reject test against the current policy, set accepted ratios to 1, leave rejected ones alone, and drop that into GRPO/DAPO/GSPO. That packaging, plus the staleness and MoE stress tests, is what is actually new relative to clipping, soft clipping, and R3.\n\nWhat they do well: the method is genuinely plug-in (only the ratio changes), overhead is claimed ~1% with no extra forward, and the experiments are broader than the usual single-backbone math run—dense and MoE, math and agent, three base objectives, plus N-staleness, clip-free, and SIS+R3. Table 2’s “trick-free” framing is honest: SIS with plain w already beats several stabilizers, and pairing with DAPO is strongest. Prop. 1 is textbook rejection sampling; Thm. 1 / Prop. 3 correctly show that zeroing accepted log-ratios shrinks the D that bounds the token-vs-sequence estimator gap. That bound is elementary but directionally right for the problem they state.\n\nSoft spots, in proportion. The stress-test lands: the implemented top-K envelope (Prop. 2) only certifies the conditional πθ(· | v ∈ VK), with TV error exactly ξK. Unit weight is exact for Prop. 1, approximate for Alg. 1. They show ξK small in Fig. 2B and keep Prop. 3 pathwise, but they never fold residual mass or accept/reject noise into the gradient-bias story under long drift. That is a real interpretive gap, not a collapse of the method—empirics still move in the claimed direction. Also missing: multi-seed error bars, released code, and a clean link from smaller D to final accuracy. Hyperparameter K looks robust in their sweep; free parameters are mostly inherited from the base algorithms.\n\nMath and citations look solid for this venue class—classical IS/rejection plus the right recent LLM-RL papers, no circular self-definition of the metrics. Who it is for: people shipping GRPO-style post-training who care about stale rollouts and MoE train–inference mismatch. Worth a serious referee; I would bring it to reading group if we are deep in LLM RL this month, and I would cite the plug-in + stress-test results if I am writing in that lane.","headline":"Practical plug-in that often works; theory is elementary and the top-K certificate is only approximate, but the empirics and stress tests are the real reason to care.","tokens_in":25352,"tokens_out":622,"would_cite":true,"duration_ms":9815,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Selective Importance Sampling turns off-policy tokens on-policy so LLM RL can reuse stale rollouts without exploding importance ratios.","keywords":["selective importance sampling","off-policy RL","LLM alignment","rejection sampling","token-level importance ratios","GRPO","policy gradient","mixture-of-experts"],"falsifier":"On the same math or agent runs, measure residual mass outside the top-K set and retrain with a deliberately large residual (or with exact full-vocabulary Mt when feasible); if accuracy and the claimed reduction in log-importance deviation both vanish once residual mass is no longer tiny, the unit-weight certificate fails for the implemented estimator.","tokens_in":25269,"feed_emoji":"🎯","tokens_out":1011,"duration_ms":10339,"temperature":0.7,"pith_summary":"Post-training large language models with reinforcement learning almost always updates on data rolled out from an older policy, so the training distribution is off-policy. Standard importance sampling corrects the mismatch, but token-level ratios multiply across long sequences and the variance explodes. This paper proposes Selective Importance Sampling (SIS): treat the behavior model as a proposal, run a cheap token-level accept/reject test, give accepted tokens unit weight as if they were on-policy, and leave rejected tokens with the usual ratio. The authors prove that this strictly shrinks the log-importance gap that bounds how far the practical token-level gradient sits from the true sequence-level gradient, and they show the method plugs into common algorithms with almost no extra compute. Across dense and mixture-of-experts models on math and agent tasks, SIS raises accuracy and keeps training stable even when rollouts are reused for many updates or when train and inference routing disagree.","feed_headline":"Off-policy tokens become on-policy with a reject test","feed_subtitle":"A one-line ratio swap lifts math and agent RL scores and stabilizes stale rollouts","key_machinery":"Selective Importance Sampling (SIS): at each token, the behavior policy is treated as a proposal; an acceptance probability equal to the token importance ratio divided by a top-K envelope of that ratio certifies the token as on-policy (weight 1) or leaves it off-policy (standard ratio). The resulting modified ratios are substituted into any existing policy loss.","core_discovery":"The paper claims that a token-level rejection test, with accepted tokens reweighted to 1 and rejected tokens kept under ordinary importance sampling, reduces the cumulative log-importance deviation that controls the error between sequence-level and token-level off-policy policy-gradient estimators, and that this single change is enough to improve multiple RL post-training objectives on real LLM alignment benchmarks while remaining a drop-in ratio swap.","pith_inferences":["If accept rates stay high under longer tool-use trajectories, SIS could become a default stabilizer for multi-turn agent RL without redesigning the reward or the rollout system.","The same accept/reject idea might be applied to other multiplicative importance corrections (e.g., preference or distillation losses) wherever long products of ratios appear.","Token-level accept/reject clouds suggest SIS may act as an automatic filter that prefers reasoning tokens over formatting noise; that filtering effect could be measured as an auxiliary curriculum signal.","Because overhead is only about one percent of step time, SIS is a candidate default in production RL stacks that already cache old and new logits."],"forward_implications":["Any off-policy token-level RL objective (GRPO, DAPO, GSPO and related variants) can replace its importance ratio with the SIS ratio and expect higher average accuracy with no change to the rest of the loss.","Reusing the same rollouts for many mini-batch updates becomes less harmful because a large share of tokens continue to receive unit weight even as the policy drifts.","Mixture-of-experts train–inference routing mismatch can be mitigated by SIS alone or further by stacking SIS with routing-replay methods.","Clipping can be removed or relaxed while still keeping training stable, because SIS itself suppresses the most severe off-policy ratios.","The theoretical bound on sequence-versus-token gradient error tightens whenever the accept rate is positive, giving a direct diagnostic (log-importance deviation) for off-policy health."],"fun_headline_variants":["Reject test flips off-policy tokens to on-policy in LLM RL","SIS: token reject accepts units, keeps IS on rejects","Plug-in ratio swap cuts token-vs-sequence off-policy gap","Selective IS turns stale rollouts on-policy via reject test","One-line reject test stabilizes off-policy LLM alignment"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The practical top-K envelope is close enough to exact rejection sampling that accepted tokens can safely be treated as fully on-policy, which requires that almost all of the target policy’s probability mass sits inside those few top tokens.","fun_headline_variants_meta":{"raw":{"variants":["Reject test flips off-policy tokens to on-policy in LLM RL","SIS: token reject accepts units, keeps IS on rejects","Plug-in ratio swap cuts token-vs-sequence off-policy gap","Selective IS turns stale rollouts on-policy via reject test","One-line reject test stabilizes off-policy LLM alignment"]},"model":"grok-4.5","effort":"low","cost_usd":0.005298,"raw_usage":{"total_tokens":1477,"prompt_tokens":795,"num_sources_used":0,"completion_tokens":93,"cost_in_usd_ticks":52980000,"prompt_tokens_details":{"text_tokens":795,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":589,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":795,"tokens_out":93,"duration_ms":4706,"temperature":1.0,"reasoning_tokens":589,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T14:28:44.667287+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same math or agent runs, measure residual mass outside the top-K set and retrain with a deliberately large residual (or with exact full-vocabulary Mt when feasible); if accuracy and the claimed reduction in log-importance deviation both vanish once residual mass is no longer tiny, the unit-weight certificate fails for the implemented estimator.","supporting_citations":[],"review_version":1}