{"id":"37528c92-d28e-4f21-bd36-f6d65274fad4","arxiv_id":"2605.26315","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Staged-Competence curriculum reduces out-of-distribution harmful responses by 16% and jailbreak success rates by 20% in DPO safety alignment across three model families while using 75% of the data.","lead":"This paper tests whether organizing safety training data by difficulty in stages improves how well Direct Preference Optimisation aligns large language models against harmful outputs. A smart generalist might read it because safer AI with less training data and better resistance to tricks could affect how companies build and deploy models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No ablation on alternative orderings (random/reverse) leaves open whether gains require the claimed difficulty curriculum or arise from staged reference updates alone.","rationale":"The reader's weakest_assumption already isolates exactly this missing control. The abstract description of the method does not mention any ordering ablations, so the concern stands as the primary internal uncertainty. No other component (e.g., loss function, model families) appears more fragile on the supplied information.","tokens_in":1694,"tokens_out":299,"duration_ms":11808,"concrete_test":"Re-run the three-model-family evaluation with a random-order variant that preserves competence-based sampling and reference-model staging but ignores difficulty labels; if OOD harmful rates and jailbreak ASR stay within 5% of the reported Staged-Competence numbers, the curriculum ordering is not required for the headline gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim credits Staged-Competence's difficulty ordering plus competence sampling for the 16% OOD and 20% jailbreak reductions. The method also includes progressive reference-model updates. Without a control that keeps the staged reference and sampling schedule but replaces the difficulty ordering with random or reverse ordering, any benefit could be driven by the reference update or by using only 75% of the data rather than by curriculum structure. This is load-bearing because the abstract reports averaged gains across model families but supplies no such disambiguation.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Staged-Competence, a curriculum learning framework for DPO-based safety alignment of LLMs. It organizes preference data by difficulty, applies competence-based sampling, and progressively updates the reference model. Averaged across three model families, it reports 16% reduction in OOD harmful response rates and 20% reduction in jailbreak attack success rates, while preserving general capabilities with near-zero over-refusal; it also claims to match baseline safety performance using only 75% of the training data and to improve separation between safe and unsafe responses. The method is presented as loss-agnostic and extensible.","tokens_in":1829,"tokens_out":562,"duration_ms":22449,"significance":"If the gains can be attributed specifically to the difficulty curriculum rather than reference-model staging or data subsampling, the work would offer a practical, data-efficient enhancement to existing DPO safety pipelines with public code release aiding reproducibility. The empirical focus on OOD robustness and jailbreak resistance addresses a known limitation of standard DPO.","major_comments":[{"comment":"The central claim attributes the 16% OOD and 20% jailbreak reductions to the difficulty ordering plus competence sampling within Staged-Competence. However, the experimental evaluation lacks an ablation that retains the staged reference-model updates and competence sampling schedule but replaces the difficulty ordering with random or reverse ordering. Without this control, the reported gains cannot be isolated from the effects of using only 75% of the data or the reference updates alone.","section":"Experiments (results tables and ablation studies)"},{"comment":"Results throughout the experimental section report averaged percentage reductions (16% OOD harmful rates, 20% jailbreak success) across model families but supply no error bars, standard deviations, number of runs, or statistical significance tests. Details on the construction and sampling of the OOD evaluation sets and jailbreak prompts are also insufficient to evaluate whether the improvements generalize beyond the specific test distributions chosen.","section":"§4 (Experimental Setup and Results)"}],"minor_comments":[{"comment":"The abstract states the method 'matches baseline safety with only 75% of the training data' but does not clarify whether the 75% subset is the same across all stages or how the staged reference updates interact with this reduced data regime.","section":"Abstract and §4"},{"comment":"Notation for competence scores and difficulty thresholds is introduced without an explicit equation or pseudocode block, making it difficult to reproduce the sampling schedule exactly from the text alone.","section":"§3 (Method)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and indicate where revisions will be made to strengthen the manuscript.","responses":[{"response":"We agree that the specific ablation isolating difficulty ordering (while retaining staged reference updates and competence sampling) is absent and would better attribute gains to the curriculum component. Existing experiments compare Staged-Competence against standard DPO and partial variants, but do not include this exact control. We will add random-order and reverse-order ablations in the revised version to address this directly.","revision_made":"yes","referee_comment":"[Experiments (results tables and ablation studies)] The central claim attributes the 16% OOD and 20% jailbreak reductions to the difficulty ordering plus competence sampling within Staged-Competence. However, the experimental evaluation lacks an ablation that retains the staged reference-model updates and competence sampling schedule but replaces the difficulty ordering with random or reverse ordering. Without this control, the reported gains cannot be isolated from the effects of using only 75% of the data or the reference updates alone."},{"response":"We acknowledge the value of reporting variability and statistical details. The manuscript will be updated to include error bars, standard deviations, run counts, and significance tests for the averaged results. We will also expand §4 with explicit descriptions of OOD set construction, sampling procedures, and jailbreak prompt sources to support evaluation of generalization.","revision_made":"yes","referee_comment":"[§4 (Experimental Setup and Results)] Results throughout the experimental section report averaged percentage reductions (16% OOD harmful rates, 20% jailbreak success) across model families but supply no error bars, standard deviations, number of runs, or statistical significance tests. Details on the construction and sampling of the OOD evaluation sets and jailbreak prompts are also insufficient to evaluate whether the improvements generalize beyond the specific test distributions chosen."}],"tokens_in":1409,"tokens_out":418,"duration_ms":17897,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a training recipe called Staged-Competence that stages safety preference pairs by difficulty, samples based on model competence, and progressively updates the reference model inside DPO. Across three model families it claims lower harmful OOD rates and lower jailbreak success while using only 75% of the data and keeping general capabilities intact. Code release is a plus for anyone who wants to replicate or extend it.\n\nThe experiments are purely empirical and the numbers look directionally useful for practitioners who already run DPO on safety data. The framework is also loss-agnostic, which broadens its potential reach.\n\nThe soft spots sit in the controls and reporting. No error bars or statistical tests appear in the abstract, and the construction of the OOD sets and jailbreak prompts is not described. The stress-test point holds: there is no ablation that keeps the staged reference updates and sampling schedule but replaces the claimed difficulty ordering with random or reverse ordering. Without that, it is impossible to know whether the reported reductions come from the curriculum structure or from the reference-model progression alone. That distinction matters for anyone trying to apply the method elsewhere.\n\nThis is for groups already doing preference tuning for safety who need a concrete recipe to test. It is worth sending to peer review because the practical angle is clear and the numbers, if they hold under tighter controls, would interest people shipping aligned models. The current version needs the missing ablations and basic statistical details before the central claim can be taken at face value.","headline":"Staged-Competence reports 16-20% averaged gains on OOD safety and jailbreak metrics for DPO but the experiments do not isolate whether the difficulty curriculum is doing the work.","tokens_in":2302,"tokens_out":392,"would_cite":false,"duration_ms":16574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A curriculum that stages safety preference data by difficulty produces more robust alignment than standard direct preference optimization.","keywords":["curriculum learning","safety alignment","direct preference optimization","out-of-distribution generalization","jailbreak attacks","preference data ordering","large language models"],"falsifier":"Applying the identical staging and sampling procedure to a fresh collection of safety preference data drawn from a different source and observing no drop in out-of-distribution harmful responses would indicate the reported gains are not general.","tokens_in":2597,"feed_emoji":"🛡️","tokens_out":750,"duration_ms":21737,"temperature":0.7,"pith_summary":"The paper tests whether curriculum learning principles can address the known brittleness of direct preference optimization when used for safety alignment of language models. It introduces a staged approach that orders training examples from easier to harder preference pairs, samples according to the model's current ability, and refreshes the reference model as training advances. Across three model families this yields lower rates of harmful outputs on out-of-distribution queries and greater resistance to jailbreak attempts, all while using less data and without increasing refusals on safe inputs. A reader would care because brittle safety behavior is a practical obstacle to deploying language models in open settings. If the ordering and sampling steps are what drive the gains, then deliberate data sequencing becomes a lightweight lever for improving generalization.","feed_headline":"Difficulty staging cuts harmful responses by 16% in safety training","feed_subtitle":"A curriculum method for preference optimization reaches baseline safety with 75 percent of the data and lowers jailbreak success rates by 20","key_machinery":"Staged-Competence, the curriculum framework that sequences preference pairs by increasing difficulty, samples them according to the model's current competence, and refreshes the reference model at staged intervals.","core_discovery":"Staged-Competence organises preference data by difficulty, applies competence-based sampling, and progressively updates the reference model. Averaged across three model families, the method reduces out-of-distribution harmful response rates by 16 percent and jailbreak attack success rates by 20 percent while preserving general capabilities with near-zero over-refusal. It reaches the safety level of the baseline using only 75 percent of the training data and produces clearer separation between safe and unsafe responses. The framework is independent of the underlying policy optimisation loss and therefore extends to other direct preference optimisation variants and alignment domains.","pith_inferences":["The same difficulty-based staging could be tested on preference data for helpfulness or truthfulness to check whether efficiency gains appear outside safety.","If the ordering effect proves robust, data curation pipelines for large models might shift emphasis from volume to deliberate sequencing.","The approach suggests a low-cost way to improve safety on models that have already undergone initial alignment, without full retraining.","Extending the staging logic to reinforcement learning from human feedback loops would test whether curriculum ideas transfer beyond preference optimisation."],"forward_implications":["The method reaches baseline safety performance with only three-quarters of the usual training data.","It produces clearer separation between safe and unsafe responses than standard training.","The gains hold across three different model families without increasing over-refusal on benign queries.","Because the framework does not depend on a particular optimisation loss, it can be combined with other direct preference optimisation variants."],"fun_headline_variants":["Staged-Competence cuts OOD harms 16% using 75% data","Staged-Competence lowers jailbreak success 20% across models","Difficulty curriculum matches DPO safety at 75% training data","Staged-Competence clarifies safe unsafe response separation"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That ordering preference pairs by the authors' chosen difficulty metric and sampling by competence actually produces genuine robustness rather than results tied to the specific datasets and model families tested.","fun_headline_variants_meta":{"raw":{"variants":["Staged-Competence cuts OOD harms 16% using 75% data","Staged-Competence lowers jailbreak success 20% across models","Difficulty curriculum matches DPO safety at 75% training data","Staged-Competence clarifies safe unsafe response separation"]},"model":"grok-4.3","cost_usd":0.006705,"raw_usage":{"total_tokens":3129,"prompt_tokens":679,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":67049500,"prompt_tokens_details":{"text_tokens":679,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2376,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":679,"tokens_out":74,"duration_ms":22673,"temperature":1.0,"reasoning_tokens":2376,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T22:23:28.986800+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Applying the identical staging and sampling procedure to a fresh collection of safety preference data drawn from a different source and observing no drop in out-of-distribution harmful responses would indicate the reported gains are not general.","supporting_citations":[],"review_version":1}