{"id":"cae5aa88-002a-4a15-a277-829f7f9ec746","arxiv_id":"2412.00074","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On three small harmfulness benchmarks, DPO alignment raised Llama-2-7B safe-response rates from about 40 percent to 93-100 percent while QA accuracy stayed roughly flat.","lead":"This paper compares three ways to make a Llama-2-7B chatbot refuse harmful requests: adding safety examples during fine-tuning, reward-ranked fine-tuning (RAFT), and Direct Preference Optimization (DPO). On small safety test sets, the DPO model produced safe responses about 93 to 100 percent of the time, up from roughly 40 percent for the base model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DPO-vs-SIT/RAFT comparison is confounded by different training data sources; the paper's own tables also show SIT-1000 beating DPO on I-CoNA, so 'DPO outperforms SIT' is not isolated.","rationale":"The reader's weakest_assumption—that DPO, SIT, and RAFT are not trained on matched data—is exactly the load-bearing concern. The paper frames DPO as the most effective alignment method, but the DPO run uses SafeRLHF preference data while SIT and RAFT use Bianchi's safety dataset, so the method and data vary simultaneously. The reported Llama Guard numbers are also not fully consistent with the 'outperform SIT' claim, since SIT-1000 achieves 94% on I-CoNA versus DPO's 93%. This is not an external-consensus disagreement; it is an internal control failure in the experimental design. A matched-data retraining test would settle whether the DPO advantage is real. The paper does provide useful evidence elsewhere: hyperparameters are specified, the SIT scaling trend is reported, and the RAFT error analysis correctly identifies reward-model quality as a failure mode. No code is released, but the training details are sufficiently concrete that the proposed test is feasible. The reader's CONDITIONAL verdict is appropriate; this stress-test does not move it, because the unresolved confound and the numeric inconsistency both point to the same needed condition: a controlled comparison.","tokens_in":48,"tokens_out":4332,"duration_ms":73749,"concrete_test":"Retrain DPO on preference pairs constructed from the exact Bianchi safety dataset used for SIT/RAFT: for each of the ~2.5k unsafe prompts, use the GPT-3.5-generated safe response as chosen and the SFT base model's own generation (or a known harmful response) as rejected, then balance/upsample to roughly 8k instances while keeping all DPO hyperparameters from Section 7.3 identical. Evaluate Llama Guard safe % on I-CoNA, I-MaliciousInstructions, and I-Controversial and compare to Table 23 and the SIT-1000/2000 rows of Table 16. If the DPO advantage disappears or flips, the paper's 'DPO outperforms' claim is driven by data source rather than algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 7.1 and 7.2 train SIT and RAFT on the 20k Alpaca instruction set plus the 2.5k Bianchi et al. (2023) safety samples, while Section 7.3 trains DPO on 8,066 preference pairs from the SafeRLHF/BeaverTails corpus (Table 2). These differ in prompt distribution, response style, and dataset size, so the DPO model's high safe rates in Table 23 could reflect the training data rather than the DPO objective. Section 9's attribution of DPO's success to 'learning from chosen and rejected responses' is therefore an unsupported mechanism claim. Moreover, the reported numbers do not uniformly support 'outperform SIT': Table 16 shows SIT with 1000 safety samples at 94% on I-CoNA vs DPO's 93%, 92% vs 95% on I-MaliciousInstructions, and 97% vs 100% on I-Controversial. Thus the central comparative claim is both confounded and, on one benchmark, contradicted by the paper's own tables.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies three alignment recipes for a Llama-2-7B model instruction-tuned on 20k Alpaca samples: Safety Instruction-Tuning (SIT) by adding n safety samples from Bianchi et al., RAFT with two reward models, and DPO on SafeRLHF preference data. The authors evaluate safety with Llama Guard and reward-model scores on I-CoNA, I-MaliciousInstructions, and I-Controversial, and helpfulness with BoolQ, PIQA, OpenBookQA, Alpaca test, and NLI/lexical metrics. They claim that safe responses increase from about 40% to over 90% without helpfulness loss and that DPO outperforms SIT and RAFT.","tokens_in":16037,"tokens_out":11778,"duration_ms":98772,"significance":"If the central claims were established, this would be a practically useful demonstration that lightweight post-hoc alignment, especially DPO, can improve the safety of an open instruction-tuned 7B model while retaining QA performance, and it would provide a useful comparison of three alignment recipes. The paper has several strengths: it reports a wide range of metrics, includes negative and abandoned evaluation results such as LLM-as-judge positional bias and Bert classifier failure on HEx-PHI, and contains an error analysis of RAFT that identifies reward-model mis-ranking. However, the DPO comparison is confounded with training data, the harmfulness test sets are extremely small, and the helpfulness evaluation has scoring ambiguities, so the current evidence is suggestive but not decisive.","major_comments":[{"comment":"The central conclusion that DPO outperforms SIT and RAFT is confounded with training-data identity. SIT and RAFT are trained on 20k Alpaca instructions plus 2.5k Bianchi safety samples (§7.1, §7.2), while DPO is trained on 8,066 preference pairs from the SafeRLHF/BeaverTails corpus (§7.3, Table 2). The methods differ simultaneously in dataset size, prompt distribution, and response style, so Table 23's high DPO safe rates cannot be attributed to the DPO objective. In addition, Table 16 shows SIT-1000 reaching 94% on I-CoNA versus DPO's 93%, so the paper's own numbers do not uniformly support the superiority claim; Section 9's mechanism explanation, 'learning from chosen and rejected responses,' is therefore unsupported. The comparison should be rerun with matched training data, or the conclusion should be restricted to 'DPO trained on preference data achieved the highest rates.'","section":"§7.1–§7.3; Tables 16, 23; §9"},{"comment":"The baseline safe percentages are inconsistent across tables. Table 16 reports Base as 0.41/0.41/0.45 on I-CoNA/I-MaliciousInstructions/I-Controversial, Table 18 reports the same base as 39.9%/50%/42.5%, and Table 23 reports 39.9%/42.5%/50%. Since the abstract's 'from 40% to over 90%' hinges on the baseline, this discrepancy must be resolved. Furthermore, the three harmfulness sets contain only 178, 100, and 40 examples, and every Llama Guard result is a single-run point estimate; a difference such as 92% versus 94% is only two or three examples. Please provide exact counts, bootstrap confidence intervals or multiple runs, and a validation of Llama Guard agreement on these particular sets.","section":"§4.2; Tables 16, 18, 23"},{"comment":"The helpfulness evaluation does not yet support the 'no compromise' claim. The BoolQ instruction template says 'Answer the following question (True/False)', but BoolQ answers are 'yes/no'; the paper does not explain how the exact-match scoring maps these, despite using max_tokens=1. The Alpaca-test lexical scores (BLEU ≈ 0.038, ROUGE-L ≈ 0.22, BERTScore ≈ 0.19) are low enough that their differences are hard to interpret, and Tables 17, 20, 21, and 24 report QA accuracies without variance or significance testing. Please clarify the scoring protocol and state the uncertainty before using these numbers to claim helpfulness is retained.","section":"§4.1, §5.1; Table 17"},{"comment":"The RAFT results are close to baseline and are overinterpreted. In Table 19 (B=100, five iterations, Deberta reward model), I-CoNA safety is 38.2%, 39.32%, 40.01%, and 39.9% over iterations against a 39.9% baseline, so there is no improvement on that set; only I-MaliciousInstructions moves from 50% to 54%. In Table 18 (B=500, one iteration), the Deberta-filtered model ties the baseline on I-CoNA at 39.9%. The text's statement that safety accuracies 'have increased on some safety datasets' is accurate only for a subset and should be quantified and hedged accordingly.","section":"§7.2.3; Tables 18, 19"},{"comment":"The OpenAssistant Deberta model is used both as a RAFT training filter (§7.2.2) and as a harmfulness evaluation metric (§5.2). I do not believe this invalidates the Llama Guard tables, because those classifications come from a separate model, but the paper should state that reward-model-based safety scores for RAFT-Deberta are not independent of the training signal. Any conclusion about RAFT-Deberta should be based on Llama Guard or another held-out judge, and the paper should make explicit which table uses which metric.","section":"§5.2; §7.2.2"}],"minor_comments":[{"comment":"There are numerous typos, including 'choosen' (§3.2), 'harmfuless' (Table 4), 'Futher' (§5.1), 'contrained' (Appendix §10), and 'volumn' (Figure 11).","section":"Throughout"},{"comment":"The cell 'Median # words in prompt ll' appears to be a corrupted value and should be replaced by a number.","section":"Table 2"},{"comment":"References to figures are inconsistent: the text says 'graph 3' and '11' where it should say 'Figure 3' and 'Figure 11', and several table references in Section 7 do not match the appendix numbering.","section":"§7"},{"comment":"The 'safe RLHF data' is not cited; the paper should cite the SafeRLHF/BeaverTails source and clarify the relationship to the preference dataset described in Section 3.2.","section":"§7.3"},{"comment":"The phrasing 'The results of DPO are pretty impressive' is informal and should be replaced with a quantitative, hedged summary.","section":"§9"},{"comment":"The manually curated helpfulness set is described as verified via 'self-reflection, online searches, and cross-checks with GPT-3.5'; please provide the actual prompts or a link and state any agreement measure, since this set is used for single-word answer accuracy in Tables 20, 21, and 24.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"This is an empirical report whose central comparison is confounded and whose evaluation lacks uncertainty quantification; the underlying phenomenon is plausible and the issues are reparable. I recommend major revision rather than rejection. The key fixes are a matched-data DPO control, exact per-example counts with confidence intervals, clarification of BoolQ scoring, and a more cautious abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a useful replication of Bianchi-style safety instruction tuning, but the headline 'DPO outperforms SIT and RAFT' is not supported as stated. The DPO model trains on SafeRLHF preference data while SIT and RAFT train on Bianchi's 2.5k instruction samples, so algorithm and data are entangled. And the paper's own numbers show SIT-1000 at 94% on I-CoNA vs DPO's 93%; DPO does better on I-Malicious (95 vs 92) and I-Controversial (100 vs 97), so 'outperform' is at best mixed.\n\nWhat is new: the head-to-head comparison itself and the RAFT error analysis showing a reward model can pick unsafe responses and amplify them in fine-tuning. The SIT scaling results (n=100..2000) are clean and consistent with Bianchi et al.; the helpfulness tables show no large drop on BoolQ/PIQA/OpenBookQA. Credit where due: the authors are transparent about discarded metrics (winrate positional bias, Bert classifier failure) and the RAFT reward-model issue.\n\nSoft spots, in rough order of severity. First, the confound above is load-bearing. Second, the safety test sets are tiny—178, 100, 40—and there are no error bars or multiple seeds, so a 90% vs 94% difference is noise. Third, Section 9 attributes DPO's success to 'learning from chosen and rejected responses'; nothing in the design isolates that mechanism. Fourth, the abstract says 'increased safe responses from 40% to over 90% across various harmfulness benchmarks'—true only if you pick the right rows; RAFT doesn't get there, and SIT-2000 is 89% on I-CoNA. Fifth, calling the evaluation 'rigorous' oversells a set of heuristic metrics (Llama Guard, reward-model scores, BLEU/ROUGE on generation).\n\nNone of this is fatal to the descriptive claim that adding a few safety instruction samples helps a lot. A reader who wants a replication datapoint for SIT can use the paper. But the comparative DPO claim needs either matched training data or heavy rephrasing.\n\nAudience: practitioners doing safety alignment who want quick numbers; not researchers looking for a mechanism. I'd send it to a workshop or short-paper track with a request to fix the comparison claim, not to a main conference. A serious referee can help the authors isolate the data effect from the algorithm effect.","headline":"Useful replication of safety instruction tuning, but the DPO-vs-SIT/RAFT comparison is confounded by different training data and contradicted on one benchmark by the paper's own tables.","tokens_in":16578,"tokens_out":3348,"would_cite":false,"duration_ms":29750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding safety examples to instruction tuning or applying DPO after tuning raises safe responses on three harmfulness benchmarks from roughly 40% to over 90% while leaving QA accuracy essentially unchanged.","keywords":["safety alignment","instruction tuning","direct preference optimization","reward ranked fine-tuning","harmfulness evaluation","helpfulness","LLaMA-2","Llama Guard"],"falsifier":"Train all three methods on the same safety data—for instance, construct preference pairs from the same safety instruction-tuning set used for SIT and RAFT, and also run SIT/RAFT on safe-RLHF-style pairs—and compare Llama Guard safe rates on I-CoNA, I-MaliciousInstructions, and I-Controversial. If the DPO advantage disappears or reverses, the paper's attribution of the gain to the preference-loss algorithm is falsified.","tokens_in":15629,"feed_emoji":"🛡️","tokens_out":9581,"duration_ms":69084,"temperature":0.7,"pith_summary":"This paper sets out to show that a modest amount of safety-focused training can make an instruction-tuned open-weight 7B language model safe to serve: adding a few hundred curated (unsafe instruction, safe response) examples to instruction tuning, or applying preference optimization afterward, raises the fraction of responses judged safe by a guardrail classifier from about 40% to above 90% on three harmfulness benchmarks. It further claims that Direct Preference Optimization (DPO), which trains on both chosen and rejected responses, outperforms two alternatives—Safety Instruction-Tuning (SIT) and Reward Ranked FineTuning (RAFT)—while leaving helpfulness on common QA benchmarks essentially unchanged. The practical interest is that this safety gain is achieved with QLoRA-style efficient fine-tuning on limited compute, without a full RLHF loop.","feed_headline":"Preference tuning lifts safe responses from 40% to 90%+","feed_subtitle":"On a 7B model, direct preference optimization beats the other two alignment methods at the same QA accuracy.","key_machinery":"The argument turns on three training recipes compared on the same base model. Safety Instruction-Tuning (SIT) just mixes a small number of (unsafe instruction, safe response) pairs into the instruction-tuning data. Reward Ranked FineTuning (RAFT) samples several responses per prompt, scores them with a reward model, and fine-tunes on the top-ranked responses; the paper shows the reward model's safety accuracy is the deciding factor, because a model that scores unsafe responses highly can misalign the model. Direct Preference Optimization (DPO) is the central mechanism: it optimizes the policy directly on preference pairs so the model is pushed toward chosen responses and away from rejected ones, which the paper identifies as the reason DPO outperforms the other two. The evaluated base model is Llama-2-7B fine-tuned on 20,000 Alpaca instructions with QLoRA.","core_discovery":"The paper's central claim is that safety can be aligned into an instruction-tuned Llama-2-7B model cheaply and without a helpfulness tax. Adding 100 to 2,000 safety samples to the 20,000-sample Alpaca instruction set reduces Llama Guard unsafe rates substantially, with the 1,000-sample model reaching 94% safe on I-CoNA, 92% on I-MaliciousInstructions, and 97% on I-Controversial. Applied after instruction tuning, DPO on the SafeRLHF preference dataset reaches even higher safety (93%, 95%, and 100% on the same three sets) and slightly improves single-word answer accuracy from 40% to 43% while keeping BoolQ, OpenBookQA, and PIQA accuracy near or above the baseline. The authors attribute DPO's advantage to its contrastive learning signal, which moves the model toward chosen responses and away from rejected ones. RAFT, in contrast, only helps when its reward model is safety-accurate; a weak reward model can select unsafe responses and reduce safety.","pith_inferences":["Because DPO was trained on SafeRLHF preference pairs while SIT and RAFT used a different safety dataset, the paper does not isolate whether DPO's advantage comes from the algorithm or from the training data; a controlled cross-dataset test would settle this.","The claim of 'over 90% safe' rests on Llama Guard as the primary safety metric; an independent judge or a jailbreak-style evaluation could show whether the safety margin generalizes beyond this one classifier.","The NLI claim-recall metric introduced in the appendix could serve as a lightweight helpfulness check in settings where API-based judges are unavailable.","The winrate-with-reference prompt template documents a practical way to reduce positional bias in LLM-as-a-judge evaluations, and could be reused by other low-resource alignment studies."],"forward_implications":["Adding 100 safety samples to instruction tuning already raises the Llama Guard safe rate on I-CoNA from 41% to 65%, and 1,000 samples reach 94%.","DPO reaches 93% safe on I-CoNA, 95% on I-MaliciousInstructions, and 100% on I-Controversial, against a base of roughly 40-50%.","Helpfulness is largely preserved: BoolQ accuracy stays around 70%, OpenBookQA and PIQA improve by a few points, and single-word answers rise from 40% to 43% with DPO.","RAFT improves safety only when the reward model is safety-accurate; with a weaker reward model, safety can decline.","The 20k-instruction QLoRA recipe offers a low-compute path to safety alignment that does not require a separate RLHF pipeline."],"supporting_citations":[{"why":"Supplies the safety instruction-tuning dataset and the baseline SIT recipe the paper replicates and extends.","marker":"(Bianchi et al., 2023)"},{"why":"Introduces DPO, the preference-optimization algorithm the paper claims outperforms SIT and RAFT.","marker":"(Rafailov et al., 2023)"},{"why":"Introduces RAFT, the reward-ranked fine-tuning method the paper compares against DPO and SIT.","marker":"(Dong et al., 2023)"},{"why":"Provides the BeaverTails/SafeRLHF preference dataset used for DPO training.","marker":"(Ji et al., 2023)"},{"why":"Source of the red-teaming prompts from which the safety instruction-tuning samples are drawn.","marker":"(Ganguli et al., 2022)"},{"why":"Defines Llama Guard, the primary classifier used to measure safe vs unsafe responses.","marker":"(Inan et al., 2023)"},{"why":"Supplies the Alpaca instruction dataset underlying the 20k-sample base instruction tuning.","marker":"(Taori et al., 2023)"},{"why":"Provides RewardBench safety datasets used to evaluate the reward models for RAFT.","marker":"(Lambert et al., 2024)"}],"fun_headline_variants":["DPO boosts safe answers from 40% to 90%+ without hurting helpfulness","Safety tuning: DPO lifts safe responses to 90%+ on 7B LLM","From 40% to 94% safe: DPO aligns LLMs with no helpfulness loss","Instruction tuning with DPO: 40% to 90%+ safe, zero helpfulness tax"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats DPO, SIT, and RAFT as comparable alignment methods, but DPO was trained on a different preference dataset than the safety examples used for SIT and RAFT, so if the data source rather than the algorithm drives the improvement, the headline comparison is not isolated.","fun_headline_variants_meta":{"raw":{"variants":["DPO boosts safe answers from 40% to 90%+ without hurting helpfulness","Safety tuning: DPO lifts safe responses to 90%+ on 7B LLM","From 40% to 94% safe: DPO aligns LLMs with no helpfulness loss","Instruction tuning with DPO: 40% to 90%+ safe, zero helpfulness tax"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2909,"prompt_tokens":933,"completion_tokens":1976,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1876}},"tokens_in":549,"tokens_out":1976,"duration_ms":12815,"temperature":1.0,"reasoning_tokens":1876,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:26:23.610955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train all three methods on the same safety data—for instance, construct preference pairs from the same safety instruction-tuning set used for SIT and RAFT, and also run SIT/RAFT on safe-RLHF-style pairs—and compare Llama Guard safe rates on I-CoNA, I-MaliciousInstructions, and I-Controversial. If the DPO advantage disappears or reverses, the paper's attribution of the gain to the preference-loss algorithm is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the safety instruction-tuning dataset and the baseline SIT recipe the paper replicates and extends."},{"cited_title":"D., and Finn, C","cited_arxiv_id":null,"evidence_quote":"Introduces DPO, the preference-optimization algorithm the paper claims outperforms SIT and RAFT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces RAFT, the reward-ranked fine-tuning method the paper compares against DPO and SIT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the red-teaming prompts from which the safety instruction-tuning samples are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Llama Guard, the primary classifier used to measure safe vs unsafe responses."}],"review_version":1}