{"id":"a93b95e4-b6ad-4929-95d3-7564c1b0745b","arxiv_id":"2607.03528","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"RLSR aligns LLMs via a lifted AURC reward and batch ranking inside GRPO, producing better risk-coverage curves than accuracy- or calibration-based RL on in- and out-of-domain tasks.","lead":"This paper trains large language models to abstain on uncertain inputs by optimizing selective-prediction metrics instead of raw accuracy or calibration. The resulting models answer fewer questions but make fewer mistakes on the ones they do answer, which matters for high-stakes human-AI handoff.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified; the batch-ranking surrogate is the softest link but is already stress-tested by the paper.","rationale":"The Reader correctly isolates the batch-ranking approximation as the weakest assumption and still assigns ACCEPT with high confidence because (i) the paper already reports a graceful-degradation ablation, (ii) the empirical gains are consistent across two model families, two training domains, six OOD sets, and a risk-controlled MedQA deployment, and (iii) no critical red flags (circular math, missing baselines, or contradictory figures) appear. My stress-test reaches the same conclusion: the surrogate is imperfect but not shown to be fatal, and the concrete larger-batch re-run would settle residual doubt without altering the present evidence base. Therefore the verdict remains ACCEPT; no adjustment is warranted.","tokens_in":29552,"tokens_out":588,"duration_ms":5441,"concrete_test":"Re-train the Qwen2.5-7B HotPotQA run once with a much larger effective batch (B\times G ≥ 4096, e.g., B=128, G=32) while keeping all other hyperparameters identical, then recompute Table 1 ID/OOD AURC and Acc@10/25/50. If the gains over RLCR shrink by more than ~0.03 AURC or the Acc@k gaps close, the batch-surrogate fidelity is load-bearing; if the numbers stay within the reported seed variance (Table 5), the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (RLSR yields substantially better risk-coverage trade-offs than RLVR/RLCR/BASE on ID and OOD tasks) rests on the claim that the lifted AURC reward computed from ranks inside each B\times G pooled mini-batch (Alg. 1, Sec. 2.2) is a faithful enough surrogate for the population AURC that the GRPO updates still optimize the intended selective-prediction objective. The paper itself flags this as Challenge (2) and supplies only an empirical ablation (B=48/32/16, G=32 fixed; HotPotQA ID AURC stays 0.44/0.44/0.45 while OOD degrades from 0.41 to 0.48). That ablation shows graceful degradation rather than a proof that the stochastic gradient is unbiased for the population ranking, so a residual risk remains that the observed gains partly reflect a batch-level ranking artifact rather than true population SP improvement. No other internal inconsistency or circular derivation appears; the CC-vs-SP distinction, the lifted reward, and the multi-model multi-domain results are coherent.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Reinforcement Learning for Selection Reward (RLSR), an LLM post-training alignment method that directly optimizes selective-prediction (SP) performance via a lifted, batch-approximated AURC reward inside the GRPO framework. It argues that confidence calibration is neither necessary nor sufficient for SP, replaces the usual correctness or Brier-style rewards with a signed, rank-weighted reward R_RLSR = ±α̂_i derived from the weighted form of AURC, and shows that the resulting models achieve lower AURC and higher Acc@10/25/50 than BASE, RLVR, and RLCR on HotPotQA- and BigMath-aligned Qwen2.5-7B and Llama-3.1-8B models, both in-domain and out-of-domain, with an additional risk-controlled MedQA deployment study.","tokens_in":29945,"tokens_out":1100,"duration_ms":8454,"significance":"If the empirical gains hold, the work supplies a practical, first-of-its-kind alignment objective that targets the risk-coverage trade-off rather than accuracy or ECE alone. The multi-model, multi-domain evaluation (including a high-stakes MedQA threshold-selection experiment), the explicit CC-vs-SP distinction, the lifted reward that supplies two-sided signals and margin enforcement, and the ablations on batch size, confidence scorers, and SFT baselines constitute a solid empirical package. The method is immediately usable with existing GRPO pipelines and verbalized or logit-based confidence, so the contribution is both conceptual and deployable.","major_comments":[{"comment":"Sec. 2.2 and Alg. 1: the central technical claim is that ranking the B×G pooled rollouts inside each mini-batch yields a sufficiently faithful surrogate for the population ranking that defines AURC_w. The only supporting evidence is the brief ablation (B=48/32/16, G=32 fixed) showing ID AURC stable at 0.44/0.44/0.45 while OOD AURC degrades from 0.41 to 0.48. That shows graceful degradation, not that the stochastic gradient remains unbiased for the population objective. A short theoretical argument (e.g., concentration of batch ranks, or a controlled experiment that freezes ranks to a large fixed pool) would strengthen the claim that the observed SP gains are not partly a batch-ranking artifact.","section":null},{"comment":"Sec. 5 Limitation and Sec. 4.2: the paper itself notes that holistic AURC optimization is less directly useful than risk-constrained coverage maximization for deployment. The MedQA experiment (target 75 % accuracy) is the most practically relevant result, yet the training objective never sees a risk constraint. Either a risk-constrained variant of the reward or a clearer statement of how practitioners should choose the operating point after RLSR training would make the deployment claim more complete.","section":null}],"minor_comments":[{"comment":"Fig. 1 caption and surrounding text: the left panel is helpful, but the claim that perfect CC can still violate perfect SP ordering would be clearer with an explicit numerical example of two samples whose calibrated confidences reverse the desired ranking.","section":null},{"comment":"Eq. (2.14)–(2.15): the equivalence AURC_w_lift = 2 AURC_w − 1 is stated; a one-line remark that the constant shift does not affect the GRPO advantage (zero-mean within group) would remove any residual doubt about the sign flip.","section":null},{"comment":"Table 1 vs. Tables 8–11: the main table reports averages; the per-dataset tables reveal that on a few OOD sets (e.g., CommonsenseQA, GPQA under HotPotQA training) RLSR is not uniformly best. A short discussion of when the ranking signal fails would be useful.","section":null},{"comment":"Typo: “Abalation study” (Sec. 4.1) should be “Ablation study”; “vise versa” appears twice and should be “vice versa”.","section":null},{"comment":"Sec. C.4.2: LoRA rank r=1 is unusually low; a one-sentence justification or pointer to the cited “LoRA Without Regret” note would help reproducibility.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The batch-ranking surrogate is the softest technical link, but the paper already flags it and supplies a usable ablation; I do not view it as grounds for major revision. The work is a clean, well-executed first step on SP-aware alignment and is a good fit for a solid ML venue. Minor revision to tighten the surrogate discussion and the risk-control limitation is sufficient."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing is that this is the first paper that actually optimizes selective prediction (AURC) as the RL objective for LLMs instead of correctness or calibration. They replace the usual binary or Brier reward with a signed “lifted” AURC weight that both rewards correct answers and penalizes errors according to their rank, then pool the B×G rollouts inside each mini-batch so the ranking can be computed without a full-dataset pass. That is new, and the technical devices are simple enough to be reproducible.\n\nWhat they do well is the framing and the experiments. The CC-versus-SP distinction is drawn cleanly with a figure that actually shows the two orderings can disagree; the multi-model (Qwen-7B, Llama-8B), multi-domain (HotPotQA, BigMath + six OOD sets) tables, and the MedQA risk-controlled deployment all point the same way: RLSR lowers AURC and raises Acc@10/25/50 relative to RLVR and RLCR, even when full-coverage accuracy is comparable. Ablations on batch size, std-normalization, and confidence scorers are present and the degradation is graceful. The math is just the known weighted form of AURC plus a sign flip; no circular lemmas.\n\nThe softest spot is exactly the one the stress-test flags: there is no proof that batch ranking is an unbiased surrogate for population AURC, only an empirical check that smaller B still works. That is a real approximation, but it is already stress-tested and does not look load-bearing for the reported gains. The other limitation they themselves note—optimizing the whole AURC curve rather than a risk-constrained coverage objective—is honest and minor for a first paper. Free parameters (effective batch, LoRA rank 1, temperatures) are standard and shared across baselines.\n\nThis is for anyone working on LLM reliability, abstention, or high-stakes deployment. The empirical pattern is strong enough that a serious referee should see it; I would bring it to reading group and expect to cite the method when I next need a selective-prediction baseline. Send it to peer review.","headline":"First clean attempt to put AURC inside GRPO for LLM alignment; gains over RLVR/RLCR look real on ID/OOD and risk-controlled MedQA, batch ranking is the only soft approximation.","tokens_in":30520,"tokens_out":558,"would_cite":true,"duration_ms":8531,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Training language models to optimize the risk-coverage curve yields better selective prediction than accuracy or calibration rewards.","keywords":["selective prediction","risk-coverage curve","AURC","LLM alignment","reinforcement learning","verbalized confidence","group relative policy optimization"],"falsifier":"Train the same base models with identical hyperparameters but systematically smaller effective batch sizes; if the risk-coverage curves on held-out sets collapse toward the calibration or correctness baselines once the batch ranking becomes too noisy, the surrogate is inadequate.","tokens_in":30476,"feed_emoji":"🎯","tokens_out":588,"duration_ms":5270,"temperature":0.7,"pith_summary":"When large language models make high-stakes decisions, the safer design is often to answer only when the model is likely correct and to abstain otherwise, handing the rest to a human. That selective-prediction goal is measured by the area under the risk-coverage curve: how quickly error falls as coverage shrinks. Existing post-training methods reward correctness or calibrated confidence, but those objectives do not automatically produce the right ranking of correct versus incorrect answers. This paper introduces Reinforcement Learning for Selection Reward (RLSR), which replaces those rewards with a lifted, batch-ranked form of the area-under-risk-coverage objective inside a group-relative policy-optimization loop. Across multi-hop QA and multi-step math training sets, and on both in-domain and out-of-domain benchmarks, the resulting models separate correct from incorrect predictions more cleanly and therefore achieve lower risk at every useful coverage level, including a controlled 75 percent accuracy regime on a medical exam dataset.","feed_headline":"Train LLMs on risk-coverage, not just accuracy","feed_subtitle":"A selection reward beats correctness and calibration objectives on both in-domain and out-of-domain tasks.","key_machinery":"Reinforcement Learning for Selection Reward (RLSR): a lifted, signed form of the weighted AURC that rewards correct rollouts and penalizes incorrect ones according to their rank inside a pooled mini-batch of B prompts times G samples, then feeds those ranks into group-relative policy optimization.","core_discovery":"Directly aligning a language model with a selection-aware reward based on the area under the risk-coverage curve produces substantially better risk-coverage trade-offs than either correctness-only or correctness-plus-calibration alignment, on both the training distribution and held-out domains.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Align LLMs on AURC selection rewards not accuracy","RLSR improves LLM risk-coverage over correctness objectives","Selective prediction alignment yields better LLM coverage trade-offs","Post-train LLMs with selection reward for risk-coverage gains","Selection-aware AURC rewards beat accuracy calibration alignment"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Ranking the pooled rollouts inside each training batch is a close enough stand-in for the true population ranking that defines the area under the risk-coverage curve.","fun_headline_variants_meta":{"raw":{"variants":["Align LLMs on AURC selection rewards not accuracy","RLSR improves LLM risk-coverage over correctness objectives","Selective prediction alignment yields better LLM coverage trade-offs","Post-train LLMs with selection reward for risk-coverage gains","Selection-aware AURC rewards beat accuracy calibration alignment"]},"model":"grok-4.5","effort":"low","cost_usd":0.00516,"raw_usage":{"total_tokens":1420,"prompt_tokens":744,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":51600000,"prompt_tokens_details":{"text_tokens":744,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":611,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":744,"tokens_out":65,"duration_ms":11453,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T01:50:35.935452+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same base models with identical hyperparameters but systematically smaller effective batch sizes; if the risk-coverage curves on held-out sets collapse toward the calibration or correctness baselines once the batch ranking becomes too noisy, the surrogate is inadequate.","supporting_citations":[],"review_version":1}