{"id":"345f2b8b-2229-4fe5-bb4d-34f5d64b708b","arxiv_id":"2508.02150","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A self-supervised RL framework with curriculum-decomposed constraints and a constraint-wise binary reward model improves instruction following in reasoning LLMs while preserving reasoning performance.","lead":"The paper proposes a reinforcement learning method that improves instruction following in reasoning language models by using the model's own responses at increasingly difficult constraint levels as training rewards. It reports strong gains on instruction-following benchmarks with stable reasoning scores, though 'no external supervision' is overstated because GPT-4o and Claude are used to build the training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-label accuracy in §3.2 is never measured; if weak models fail the newly added constraint in positives or satisfy it in negatives, the reward model can learn spurious cues and Table 3 gains may be reward hacking.","rationale":"I agree with the reader that the pseudo-label assumption in Section 3.2 is the most load-bearing risk. It sits directly under the reward model that drives RL; if the labels are wrong, all downstream gains are suspect. The existing Table 2 validation is not responsive because it ranks five artificially constructed responses per instruction rather than checking binary labels in the training set. The ablations show the reward model contributes to the gain, which increases the stakes: if the reward model is learning a spurious shortcut, the improvement is not real constraint following. OOD generalization to WritingBench and Collie is encouraging but does not rule out shortcuts shared across benchmarks. The external-supervision caveat (GPT-4o in Appendix A.1.2 and Claude cold-start in Appendix A.3.4) is real, but it affects the 'self-supervised' framing more than the empirical improvement, so I treat it as secondary. The proposed check is cheap and directly resolves the concern; until it is run, CONDITIONAL remains the right verdict, so the reader's verdict needs no adjustment.","tokens_in":15937,"tokens_out":7889,"duration_ms":94725,"concrete_test":"Human-annotate a random stratified sample of the Section 3.2 reward-model training pairs, e.g., 200 positives and 200 negatives per base model (R1-Distill-Qwen-7B and Qwen2.5-7B-Instruct), asking two annotators whether each response satisfies the listed constraint. Compute pseudo-label accuracy and Cohen's kappa. If negative-label accuracy is below roughly 85% or positive-label accuracy below roughly 70%, retrain the reward model with human-filtered labels and check whether the Table 3 deltas persist or shrink.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 builds the soft-constraint reward model on the assumption that, for each added constraint c_i, the response r_i generated with c_i present satisfies c_i, while r_{i-1} generated without c_i does not. The paper never reports the accuracy of these pseudo-labels on the actual training pairs. The only validation, Table 2 and Appendix A.1.3, is a 50-group ranking correlation on responses deliberately constructed to satisfy 1..5 constraints; it does not measure binary label correctness on real model rollouts. Because the base models are weak multi-constraint followers (the paper's own motivation), positives can systematically fail the newly added constraint and negatives can accidentally satisfy it, especially for soft constraints. If this noise is biased, the reward model may learn superficial cues such as response length or formatting rather than true constraint satisfaction. Then the IFEval and CFBench gains in Table 3 could reflect reward hacking instead of genuine instruction following, which is the central empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-supervised reinforcement learning framework for improving instruction following in reasoning models without relying on a stronger external reward model. The method consists of three stages: (1) constructing multi-constraint instruction datasets and decomposing them into an incremental curriculum of constraint counts; (2) training a constraint-wise binary classification reward model for soft constraints using pseudo-labels derived from responses to adjacent curriculum levels, while hard constraints are verified by rules; and (3) optimizing the policy with GRPO using a composite reward. Experiments on R1-Distill-Qwen-1.5B/7B, R1-0528-Qwen3-8B, and a cold-started Qwen2.5-7B variant report improved instruction following on IFEval, CFBench, FollowBench, ComplexBench, WritingBench, and Collie, with reasoning performance roughly maintained (e.g., Table 3: R1-Distill-Qwen-7B IFEval 61.7→71.7; Table 4: average reasoning score 52.0 unchanged). Ablations show that removing the rule-based reward, removing the probability reward, or removing the curriculum each degrades performance, and training dynamics are analyzed.","tokens_in":16059,"tokens_out":5673,"duration_ms":63494,"significance":"If the central claims hold, the paper offers a practical recipe for strengthening soft constraint following without a larger or proprietary reward model: curriculum decomposition for dense signals, self-supervised binary classification for soft constraints, and hybrid reward aggregation. The gains are consistent across model families and sizes, with out-of-domain generalization and stable reasoning scores, and the public release of code and data is a concrete strength that supports reproducibility. The ablations are informative and mostly support the design choices. However, the headline 'without external supervision' claim is broader than what is actually implemented, and the central pseudo-label assumption in §3.2 is never directly validated on real training rollouts, so the empirical gains are not yet fully established.","major_comments":[{"comment":"The soft-constraint reward model is trained on pseudo-labels whose correctness is assumed but never measured. §3.2 states that for constraint c_i, response r_i (generated with c_i) 'is likely to satisfy it' and r_{i-1} (generated without c_i) 'does not'; the paper reports no accuracy of these labels on the actual training pairs. The only validation, Table 2 and Appx. A.1.3, uses 50 hand-constructed groups with responses deliberately designed to satisfy 1..5 constraints and measures ranking correlation, not binary label accuracy on real model rollouts. If the base policy frequently fails the newly added constraint in positives or satisfies it in negatives, the reward model can learn spurious cues such as length or formatting. Given that the paper's motivation is that the base models are weak multi-constraint followers, this is a real risk: the Table 3 gains may partly reflect reward hacking rather than genuine instruction following. I request per-curriculum pseudo-label accuracy on a random sample of training pairs (with human or rule verification) and binary accuracy of the reward model on held-out human-labeled constraint-level examples.","section":"§3.2, Table 2, Appx. A.1.3"},{"comment":"The central claim of 'without external supervision' is broader than what is implemented. The constraint construction prompt in Table 11 uses GPT-4o to generate the five constraints for each seed instruction, and the cold-start phase in Appx. A.3.4 uses Claude-Sonnet-3.7-thinking to generate SFT responses for Qwen2.5-7B-Instruct-R. The method is self-supervised for reward signals, but not end-to-end free of stronger external models. I recommend either narrowing the claim (for example, 'without a stronger external reward model') or adding an ablation in which constraints are generated without GPT-4o, such as from templates or from the policy's own generations.","section":"Abstract, §1, §3.1, §A.1.2, §A.3.4"},{"comment":"The generator of r_i and r_{i-1} is not specified. This matters because the pseudo-label assumption and the self-supervision claim depend on who produced these responses: if they are sampled from the base policy, the analysis in the first major comment applies; if they are produced by an external model, the method relies on external supervision for reward-model training. Please state the generator explicitly and provide sampling details (temperature, number of rollouts, filtering, and the number of examples per curriculum level).","section":"§3.2"}],"minor_comments":[{"comment":"Please define 'IF-Verifier-7B' and clarify the relationship between the Kendall's tau values in Table 2 (94.0) and Table 6 (61.2 for 'Our Reward Model'); as written, the two tables appear to report different evaluation setups, and the reader cannot tell which one validates the pseudo-labeled training data.","section":"Table 6"},{"comment":"The row label 'Distill-Qwen-1.5B-7B' is inconsistent with the model naming in the text ('R1-Distill-Qwen-1.5B'); please align the nomenclature.","section":"Table 3"},{"comment":"The caption says 'Avg@30' but the table column is 'Avg'; please align the caption with the column and state whether the reported reasoning scores are averaged over 30 sampled responses at the inference temperature used.","section":"Table 4"},{"comment":"Setting kl_coef to 0.0 in GRPO is unusual and deserves one sentence of justification given known reward-hacking risks; the ablation in Table 5 partially addresses this, but a brief comment would help.","section":"Appx. A.3.1"}],"recommendation":"major_revision","confidential_remarks":"The reader's central concern about pseudo-label accuracy is legitimate and should be addressed before acceptance. The 'no external supervision' overclaim is fixable in wording, but the pseudo-label accuracy measurement is a substantive addition. I would not reject on circularity grounds because the reward model is frozen and the policy is evaluated on held-out benchmarks; however, the unmeasured label quality is the main technical risk to the empirical claims. The paper is within scope for cs.AI and the public code/data release is a positive factor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey,\n\nThe core recipe here – decompose multi-constraint instructions into a curriculum, then use the model's own outputs at successive levels to build positive/negative pairs for a per-constraint binary reward model, and run GRPO on the composite reward – is genuinely new to me, and the empirical results are consistent: IFEval and CFBench climb across three base models while reasoning benchmarks stay roughly flat. The ablations support the curriculum and the hybrid rule/probability reward design. That earns the paper a serious look.\n\nThe soft spots are real. The 'without external supervision' framing is inflated: GPT-4o is used to synthesize the constraint curricula and Claude-Sonnet-3.7-thinking produces the cold-start SFT data. One can argue the reward labels themselves come from the self-generated pairs, but that is not 'no external supervision' in a plain reading. More importantly, the pseudo-label assumption in Section 3.2 is never tested on actual rollouts. Table 2's high Kendall tau comes from 50 groups of responses deliberately constructed to satisfy 1–5 constraints; that checks ranking quality, not binary label accuracy on real model outputs. If the base model often fails the newly added constraint in the positive examples, or satisfies it in the negatives, the reward model can latch onto superficial cues like length or formatting, and the Table 3 gains could be partly reward hacking. The rule-based component for hard constraints helps, but it does not cover soft constraints.\n\nWho's this for: people working on RL-based alignment and instruction following. Reviewers should ask for a pseudo-label accuracy breakdown and a precise statement of which external LMs are used where. The paper deserves peer review – it's a concrete, falsifiable method with code and data – but the headline 'self-supervised' claim needs to be restated and the reward model's training labels need direct validation.","headline":"Genuinely new curriculum-decomposed self-labeled reward model, with consistent gains, but 'no external supervision' is overstated and pseudo-label accuracy is unvalidated.","tokens_in":16634,"tokens_out":4019,"would_cite":false,"duration_ms":46652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-supervised reinforcement learning improves a reasoning model's instruction following with no external teacher, while keeping reasoning performance intact.","keywords":["self-supervised reinforcement learning","instruction following","reasoning models","constraint curriculum","reward modeling","soft constraints","GRPO","multi-constraint instructions"],"falsifier":"Take the curriculum-generated instruction pairs, have human annotators or a strong judge label whether each response actually satisfies the target constraint, and measure the pseudo-label accuracy. If accuracy is near chance for multi-constraint instructions, the reward model is learning noise and the reported gains cannot be attributed to the proposed self-supervised mechanism; a direct test would be to retrain with corrected labels and compare final IFEval and CFBench scores.","tokens_in":15702,"feed_emoji":"🎯","tokens_out":6543,"duration_ms":67200,"temperature":0.7,"pith_summary":"Reasoning models typically trade instruction following for raw problem-solving power, and past fixes rely on stronger teacher models to supervise them. This paper claims that the trade-off is not necessary: a self-supervised reinforcement learning loop, using only the model's own outputs, substantially improves constraint following while leaving reasoning scores unchanged. On a 7B distilled reasoning model, the method raises IFEval from 61.7 to 71.7 and CFBench Overall from 55.2 to 60.7, with the six-benchmark reasoning average unchanged at 52.0. The recipe matters because it removes the dependency on proprietary or expensive external models, making instruction-following improvements cheap and scalable.","feed_headline":"Self-supervised RL lifts instruction following, keeps reasoning","feed_subtitle":"A 7B reasoning model gains 10 points on IFEval and 5.5 on CFBench with no external reward model.","key_machinery":"The load-bearing mechanism is the incremental constraint curriculum feeding a constraint-wise binary classification reward model. Each training instruction is split into levels $c_1, c_2, ..., c_k$, adding one constraint at a time, so that a response generated for level $j$ can be compared with a response generated for level $j-1$ to create positive and negative examples for the single new constraint. The reward model $r(c_i, response)$ outputs the softmax probability that a response satisfies constraint $c_i$; hard constraints bypass the model and use rule-based verification. The sample-level reward is the average of these constraint-level rewards, plus a correctness reward for reasoning tasks, and it drives GRPO policy optimization. The curriculum supplies dense signals—reward is available at every level, not only when all constraints are satisfied.","core_discovery":"The paper's central claim is that a reasoning model can improve its own instruction following by learning a reward model from a curriculum built out of its own responses, and then optimizing itself with GRPO. Concretely, the authors decompose each multi-constraint instruction into levels that add one constraint at a time; for each constraint they treat the response generated for the fuller instruction as a positive example and the response generated for the instruction without that constraint as a negative example. This pseudo-labeled data trains a constraint-wise binary classifier that scores soft constraints, while hard constraints are checked by explicit rules. The per-constraint scores are averaged into a sample-level reward. The paper reports consistent gains on six instruction-following benchmarks and unchanged or slightly improved reasoning averages, including the same 7B reasoning model's average of 52.0 before and after training.","pith_inferences":["If the pseudo-label assumption is even partially wrong, the reward model's mistakes will be amplified by RL; reporting training-set label accuracy would turn this from a hidden assumption into a checkable quantity.","Because the reward model is itself a Qwen-based model trained on the same style of data, there may be a ceiling: it cannot teach constraint satisfaction it cannot recognize, so gains may shrink as constraint difficulty grows.","The unchanged reasoning average hides small per-benchmark movement (e.g., AIME25 drops 2.7 points while MMLU-Pro rises 1.8); deployers should check which directions matter for their use.","The curriculum may function partly as a regularizer or reward-shaping term, so the gains might be reproducible with simpler reward shaping rather than a full learned reward model."],"forward_implications":["The same recipe should apply to any base model that can generate responses, so instruction following can be improved without external data or reward models.","Soft constraints—formatting, tone, audience, style—become trainable through binary classification, not through prompt engineering or stronger judges.","Combining rule-based verification for hard constraints with a learned soft-constraint scorer prevents reward hacking, as the ablation shows dropping either component degrades performance.","The method can be added to the cold-start phase of reasoning-model training, which the paper argues is where the largest response-length gains appear.","The framework's efficiency (0.3s per group for the reward model) makes it practical at scale."],"supporting_citations":[{"why":"Supplies the GRPO reinforcement learning algorithm used for policy optimization.","marker":"(Shao et al., 2024)"},{"why":"Provides the R1-distilled reasoning models that serve as the main test beds.","marker":"(Guo et al., 2025)"},{"why":"Defines the soft-constraint taxonomy and the notion of soft constraints the reward model must judge.","marker":"(Ren et al., 2025)"},{"why":"Motivates rule-based verification for hard constraints in the composite reward.","marker":"(Pyatkin et al., 2025)"},{"why":"Represents the external-supervision approach the paper claims to outperform, and appears as the IR-1.5B baseline.","marker":"(Qin et al., 2025)"},{"why":"Supplies the CFBench benchmark and the constraint taxonomy used to synthesize training instructions.","marker":"(Zhang et al., 2024)"},{"why":"Supplies IFEval, a primary instruction-following evaluation metric.","marker":"(Zhou et al., 2023)"}],"fun_headline_variants":["Self-supervised RL boosts instruction following without harming reasoning","Teaching itself: RL improves instruction following, reasoning stable","No external supervision: RL lifts instruction following, preserves reasoning","Reasoning models self-improve instruction following via RL from own data","Self-RL: better instruction following, same reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a response generated for an instruction with a new constraint reliably satisfies it while the response generated without that constraint reliably does not, a pseudo-labeling assumption whose accuracy on the training set is never reported.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised RL boosts instruction following without harming reasoning","Teaching itself: RL improves instruction following, reasoning stable","No external supervision: RL lifts instruction following, preserves reasoning","Reasoning models self-improve instruction following via RL from own data","Self-RL: better instruction following, same reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2010,"prompt_tokens":814,"completion_tokens":1196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":1116}},"tokens_in":430,"tokens_out":1196,"duration_ms":12801,"temperature":1.0,"reasoning_tokens":1116,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:07:09.228553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the curriculum-generated instruction pairs, have human annotators or a strong judge label whether each response actually satisfies the target constraint, and measure the pseudo-label accuracy. If accuracy is near chance for multi-constraint instructions, the reward model is learning noise and the reported gains cannot be attributed to the proposed self-supervised mechanism; a direct test would be to retrain with corrected labels and compare final IFEval and CFBench scores.","supporting_citations":[],"review_version":1}