{"id":"e96bd52f-17d4-41e5-a595-00d426dfd170","arxiv_id":"2508.01174","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"RSPO trains LLMs directly against Pass@k and Max@k using unbiased gradient estimators derived from the probability that a response is the maximum among k samples.","lead":"RSPO is a training method that directly optimizes Pass@k and Max@k metrics in large language models, instead of maximizing average reward. A generalist reader may care because it targets the mismatch between how LLMs are trained and how they are evaluated, potentially improving performance when several responses are sampled and the best one is used.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unbiasedness of the RSPO gradient hinges on an exact, tractable derivative of the policy-dependent CDF F_theta(r); the unreadable submission provides no evidence this exists, and any batch/empirical approximation biases the Pass@k/Max@k estimator.","rationale":"The reader's weakest assumption points at the closed-form maximum probability and the unbiasedness of the nested gradient, and I agree that this is the load-bearing condition. I sharpen it: even when the probability itself is computable, the unbiased gradient of Max@k must differentiate through the cumulative distribution F_theta(r), because the max statistic has CDF F_theta(r)^k. That derivative is a policy-dependent sum over all lower-reward responses and is not obviously available in closed form for LLM sequence spaces. A batch-empirical estimate of F_theta would introduce bias. Since the provided text is unreadable, the proof cannot be checked; the responsible verdict remains UNVERDICTED, with the concrete simulation above providing a decisive check once a clean manuscript is available. I do not claim the paper is wrong; I claim its central estimator is not yet supported by the submitted evidence.","tokens_in":15212,"tokens_out":10326,"duration_ms":130391,"concrete_test":"On a finite action space with, say, 10 responses and fixed rewards, enumerate all k=3 tuples to compute the exact gradient of E[max_i r_i] and of Pass@k under a softmax policy. Implement the RSPO gradient twice: once with F_theta computed exactly by enumeration, and once with F_theta estimated from the k in-batch samples. If the exact-F_theta gradient does not match the enumerative gradient to numerical precision, the nested-gradient derivation is incorrect; if only the empirical version is what the paper prescribes, its unbiasedness claim is refuted.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is that RSPO yields unbiased gradient estimators for Pass@k and Max@k by using the closed-form probability that a response is the maximum among k samples. For Max@k, the exact gradient must differentiate through the distribution of the maximum, whose CDF is F_theta(r)^k with F_theta(r) = sum_{y: r(y) <= r} p_theta(y). The score-function expansion therefore contains a product-rule term proportional to (k-1) F_theta(r)^{k-2} grad_theta F_theta(r), in addition to the simpler term r F_theta(r)^{k-1} grad log p_theta(x). The grad F_theta term is a policy-dependent sum over the entire response space; for LLM outputs it is not closed-form in any usable sense and must be recomputed at every gradient step. If RSPO computes F_theta only from the k in-batch responses, or replaces it with a softmax or temperature surrogate, the estimator is biased and the claimed equivalence to directly optimizing Pass@k and Max@k fails. The submitted full text is illegible, so the promised proof of unbiasedness cannot be checked, and no code or formal verification accompanies the claim. This is a concrete correctness risk rather than a mere exposition gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Risk-Seeking Policy Optimization (RSPO), a post-training objective for large language models that directly optimizes Pass@k and Max@k evaluation metrics. The abstract claims that RSPO addresses the 'hitchhiking' problem—where low-reward responses are reinforced if they co-occur with high-reward responses in a batch of k samples—by leveraging the closed-form probability that a given response is the maximum among k samplings, and that this yields efficient, unbiased gradient estimators for both metrics. The abstract also promises rigorous theoretical analysis and comprehensive experimental results, including a derivation and empirical tables. The body of the submitted manuscript, however, is severely corrupted and unreadable: the text is garbled, equations are absent or illegible, and tables contain no interpretable numeric entries. Consequently, the theoretical derivation, the unbiasedness proof, and the experimental support for the central claims cannot be checked from the submitted version.","tokens_in":15488,"tokens_out":4028,"duration_ms":49143,"significance":"If the method worked as claimed, it would be a valuable contribution: directly optimizing the evaluation metrics Pass@k and Max@k rather than a risk-neutral proxy is a meaningful goal, and explicitly addressing the hitchhiking problem is a relevant practical concern for LLM post-training. The significance is, however, conditional on the correctness of the unbiased gradient estimators and on the experimental evidence, neither of which is verifiable in the current submission. No code, formal proof, or reproducible experimental protocol is provided.","major_comments":[{"comment":"The claim of unbiased gradient estimation for Max@k requires differentiating the CDF F_theta(r)^k, whose gradient contains a term proportional to (k-1) F_theta(r)^{k-2} ∇_theta F_theta(r), with ∇_theta F_theta(r) = Σ_{y: r(y) ≤ r} p_theta(y) ∇_theta log p_theta(y) summed over the entire response space. For autoregressive LLMs this sum is not closed-form in any computationally usable sense. The submitted text does not show how this term is handled; if it is replaced by a finite-sample or softmax approximation, the estimator is biased and the central claim fails. Please provide the exact expression and an unbiasedness proof, or explicitly state the approximation and its bias.","section":"Abstract"},{"comment":"The body of the manuscript is unreadable: the text is garbled and corrupted, equations are missing or illegible, and the experimental tables contain no interpretable values. The abstract's promises of 'rigorous theoretical analysis' and 'comprehensive experimental results' therefore cannot be verified. This is a load-bearing verifiability failure: a reader cannot check the derivation, the unbiasedness proof, or the empirical gains without a readable manuscript.","section":"Full text (all sections)"}],"minor_comments":[{"comment":"The body includes the unrelated line 'arXiv:2508.01178v1 [cs.SD] 2 Aug 2025'; this appears to be a formatting artifact and should be removed in any resubmission.","section":"Full text (header/footer)"},{"comment":"The phrase 'closed-form probability that a given response is the maximum among k samplings' is potentially misleading: the probability itself may be expressible in closed form, but its gradient with respect to policy parameters involves a sum over the response space and is not closed-form in a practically tractable sense. The wording should be clarified to distinguish the probability from its gradient.","section":"Abstract"}],"recommendation":"reject","confidential_remarks":"The submission appears to have suffered severe text corruption, making it impossible to evaluate the technical content. If the authors were to resubmit a clean, readable version, the paper might be worth reconsidering, but the specific concern about the ∇_theta F_theta(r) term in the Max@k gradient must be addressed explicitly: without a tractable exact expression, the unbiasedness claim is not credible. As submitted, the paper does not meet the standard for publication because no claim can be checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing to know: this submission is unreadable. The full text came through as garbled mojibake, so the equations, the experiments, and the related-work discussion cannot be checked. That forces an unverifiable verdict, not a verdict on the science. The abstract is legible, and it makes a claim that deserves a serious look.\n\nWhat the paper has going for it: the problem is real. LLM post-training mostly optimizes expected reward while evaluation uses pass@k and best-of-k, and that risk-preference mismatch is a known practical annoyance. The hitchhiking problem (low-reward responses getting reinforced because they co-occur with a high-reward response in a batch of k) is a genuine failure mode, and naming it cleanly is a useful contribution. If RSPO really delivers unbiased, low-variance gradients for these metrics, it would be a meaningful addition to the post-training toolbox.\n\nNow the soft spots. The central theoretical claim (unbiased gradient estimators via the closed-form probability that a response is the maximum among k samples) has a load-bearing subtlety. That probability is proportional to p_theta(y) times F_theta(r(y))^(k-1), where F_theta is the reward CDF under the policy. For an autoregressive LLM, F_theta(r) is a sum over the entire response space; it is not closed-form in any usable sense, and differentiating it requires differentiating that whole sum. If the paper instead estimates F_theta from the k in-batch responses, the estimator is biased and the claimed equivalence to directly optimizing Pass@k and Max@k collapses. The abstract does not say which route is taken, and the corrupted text prevents checking. This is a concrete correctness risk, not an exposition nit.\n\nAlso: no code or data accompany the submission, so the comprehensive experimental results cannot be verified either, and without a legible related-work section I cannot tell whether the formulation is new relative to best-of-k optimization or risk-seeking RL.\n\nWho this is for: people doing LLM post-training who care about aligning training objectives with evaluation metrics. The motivation is sound; the method's key property is unverified in this file. My recommendation: do not desk-reject it. Ask the authors for a readable PDF plus the proofs, and ideally code or a small worked example establishing the estimator's unbiasedness and variance. Then send it to a referee who knows RL and can check the derivation. The idea deserves referee time; this particular file does not yet provide the evidence to judge it.","headline":"Real problem, plausible direction, but the submission is unreadable and the unbiasedness claim carries a tractability burden the abstract never addresses.","tokens_in":15994,"tokens_out":6328,"would_cite":false,"duration_ms":70824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that directly optimizing risk-seeking evaluation metrics like Pass@k and Max@k with unbiased gradient estimators improves LLM post-training over maximizing expected reward.","keywords":["Risk-seeking policy optimization","Pass@k","Max@k","risk-seeking metrics","LLM post-training","unbiased gradient estimator","hitchhiking problem","policy optimization"],"falsifier":"On a small bandit-style task with known rewards, compute RSPO's gradient estimate for a fixed policy and compare it with an exhaustive Monte Carlo estimate of the true Pass@k gradient. If the difference exceeds the estimator's standard error across many seeds, the unbiasedness claim is false. Similarly, with a reward function corrupted by increasing noise, RSPO should lose its advantage if the max-probability computation is the reason it works.","tokens_in":15062,"feed_emoji":"🎯","tokens_out":3487,"duration_ms":42730,"temperature":0.7,"pith_summary":"Standard post-training for large language models maximizes expected reward, but evaluation measures risk-seeking metrics such as Pass@k (whether any of k sampled responses succeeds) and Max@k (the best reward among k samples). The paper argues that this mismatch leaves performance on the table and proposes RSPO, which optimizes the evaluation metric itself. RSPO uses the closed-form probability that a given response is the maximum among k samples to build gradient estimators that are unbiased and avoid reinforcing low-reward responses that happen to ride along with a good one, a failure the paper calls hitchhiking. The claim is that optimizing this metric directly yields better post-training performance than risk-neutral objectives.","feed_headline":"RSPO trains LLMs on Pass@k instead of expected reward","feed_subtitle":"Closed-form max probabilities give unbiased gradients, so post-training matches how models are actually evaluated.","key_machinery":"The load-bearing object is the closed-form probability that a response is the maximum among k samplings. Given a reward function $r$, this probability acts as an assignment weight in a policy-gradient update: each sampled response contributes to the gradient in proportion to the chance that it is the best of its batch, not in proportion to raw reward. The mechanism converts a non-differentiable max or success indicator into an expectation over the policy that can be differentiated, yielding unbiased nested gradient estimators for both Pass@k and Max@k.","core_discovery":"The central claim is that Pass@k and Max@k can be treated as differentiable training objectives rather than evaluation-only statistics. For a set of k responses sampled from the policy, the paper writes the probability that a specific response attains the maximum reward, and uses this probability to weight the gradient. This weighting makes the gradient estimator unbiased while suppressing the hitchhiking effect, in which poor responses receive reward because they co-occur with a high-reward response. The paper supports the construction with theoretical analysis and experiments.","pith_inferences":["Beyond the paper, the same max-probability weighting could be applied to any sample-based metric that is a function of the best outcome, such as worst-case reward or quantile rewards, by swapping the event being conditioned on.","The benefit of RSPO should depend on reward-noise levels: noisier reward signals blur which response is truly maximal, so one testable prediction is that RSPO's advantage over expected-reward training shrinks as reward noise grows.","The authors leave implicit that the choice of k at training time creates a new hyperparameter; a natural follow-up is scheduling k from small to large during training."],"forward_implications":["Training and evaluation become aligned: a model tuned at a given k is optimized for exactly the Pass@k or Max@k score that benchmarks report.","The hitchhiking problem disappears from credit assignment, so optimization effort shifts toward responses that actually win batches.","Because the gradient estimators are unbiased, RSPO can in principle be dropped into existing policy-gradient post-training pipelines without changing the update rule's target.","Matching train-time k to evaluation-time k should matter most on tasks with high response variance, such as code generation and open-ended reasoning."],"supporting_citations":[],"fun_headline_variants":["Train LLMs for Pass@k, not expected reward","Risk-seeking RL: optimizing LLMs for Max@k directly","Unbiased gradients kill hitchhiking in Pass@k training","Direct optimization of Pass@k for LLM post-training","RSPO: train LLMs on the metrics they're actually tested on"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the probability that a given response is the best of k samples can be computed from the policy in closed form, differentiated, and estimated without bias; if nested sampling dependencies or reward noise break that estimate, the unbiasedness claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Train LLMs for Pass@k, not expected reward","Risk-seeking RL: optimizing LLMs for Max@k directly","Unbiased gradients kill hitchhiking in Pass@k training","Direct optimization of Pass@k for LLM post-training","RSPO: train LLMs on the metrics they're actually tested on"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000518,"raw_usage":{"total_tokens":2449,"prompt_tokens":823,"completion_tokens":1626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1539}},"tokens_in":439,"tokens_out":1626,"duration_ms":13297,"temperature":1.0,"reasoning_tokens":1539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:46:47.917295+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small bandit-style task with known rewards, compute RSPO's gradient estimate for a fixed policy and compare it with an exhaustive Monte Carlo estimate of the true Pass@k gradient. If the difference exceeds the estimator's standard error across many seeds, the unbiasedness claim is false. Similarly, with a reward function corrupted by increasing noise, RSPO should lose its advantage if the max-probability computation is the reason it works.","supporting_citations":[],"review_version":1}