{"id":"60e72d28-4259-49a1-8d53-37802796ceb4","arxiv_id":"2412.06843","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A supervised fine-tuning loss that maximizes an Earth-Mover-Distance-style semantic penalty away from model-generated unsafe responses achieves safety with roughly 100 harmful examples.","lead":"This paper shows that a language model can be made safer during supervised fine-tuning by penalizing responses that resemble a small set of the model's own unsafe outputs, using an embedding-space distance. The method needs only about 100 unsafe examples and, according to the authors, outperforms a baseline that uses high-quality safe responses from another LLM.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-efficiency claim rests on counts from the same OpenAI Moderation API used to select training data, with lambda tuned without a described held-out set; an independent judge is needed.","rationale":"The paper is honest and interesting: it releases code and data, reports over-alignment curves, and explicitly notes limitations such as the dangers of learning with AI-generated data. The DeBERTa plots in Fig. 2 provide independent evidence that TA-SFT reduces mean harmfulness, which is real support for the core safety effect. My concern is specifically about the data-efficiency headline: the numbers that establish it are computed with the same OpenAI Moderation API that filtered the training data, and lambda selection is not described as validation-based. Neither point by itself invalidates the method, but together they make the strongest_claim dependent on a circular and possibly post-selected measurement. The concrete test above would settle it. The reader's weakest_assumption (representativeness of the 100-1,000 harmful examples) is related: the circular judge also affects representativeness, but the sharper issue is that the evaluation judge and training filter are identical. I agree with the CONDITIONAL verdict; the condition should include an independent safety judge and a documented validation procedure for lambda.","tokens_in":15503,"tokens_out":8136,"duration_ms":81188,"concrete_test":"Re-run the data-efficiency sweep (100, 300, 500, 1,000 harmful examples) for Llama-7b with a fixed seed. Tune lambda on a held-out set of ~100 toxic prompts disjoint from the four benchmarks, then count harmful responses on the four benchmarks using an independent judge: either DeBERTa with a calibrated threshold, or human annotation, not the OpenAI Moderation API. Repeat with at least three seeds and report mean and standard error. If EMD still gives near-zero harmful counts and beats NLCL by a similar margin, the concern is resolved; if counts rise substantially or the EMD advantage shrinks, the data-efficiency claim in the abstract should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline contribution is data efficiency: EMD reaches near-zero harmful responses with 100 harmful examples. The evidence for this is Table 2 (and Fig. 2d), which counts harmful responses. Section 4.1 constructs D_safety-related by generating responses and then using the OpenAI Moderation API to extract 1,000 harmful responses; Section 4.3.1 then uses the same API as the secondary safety judge, and the counts in Table 2 appear to come from this API. Consequently, the model is trained on examples selected by a classifier and then tested with that same classifier. A model can reduce API-flagged harmfulness without reducing human-judged harmfulness, so the data-efficiency comparison between EMD and NLCL may partly reflect overfitting to the moderation API rather than general safety. This is not a full indictment: Fig. 2(a-c) uses an independent DeBERTa harmfulness score and shows clear drops in mean harmfulness for the 1,000-example setting, so the core safety effect is not purely an artifact. But the specific 100/300/500-example data-efficiency claim is not independently supported. A second issue compounds this: Appendix A.2 reports grid-searched lambda values for each model and each dataset size, but no validation split or selection criterion is described. If lambda was selected on the four evaluation benchmarks, the reported counts are post-selection numbers. Together these make the central quantitative claim less secure than it appears.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Toxicity Avoiding SFT (TA-SFT), a supervised fine-tuning scheme that augments a standard instruction dataset with a small set of harmful responses to toxic prompts, generated by the base model itself. The main loss is a negative Earth Mover Distance term with cosine embedding cost, implemented through a lower bound that is optimized as the squared distance between the one-hot embedding of the observed unsafe token and the model's expected next-token embedding; the authors also introduce an NLCL likelihood-penalty baseline. Experiments on Llama 7B/13B, Mistral 7B, and Llama 3.1 8B report large safety improvements on four harmfulness benchmarks, preservation of response quality on AlpacaEval and multiple-choice tasks, data efficiency down to 100 harmful examples, and analyses of over-alignment and contrastive augmentation.","tokens_in":15740,"tokens_out":8725,"duration_ms":77087,"significance":"The practical claim is attractive and, if confirmed, would lower the cost of safety tuning by using only model-generated harmful responses. The paper has real strengths: experiments span four base models; response-quality results are reported for several benchmarks; code and data are promised; Proposition 1 is proved correctly in Appendix A.3; and the over-alignment analysis is a useful negative result. The central data-efficiency claim, however, currently rests on an evaluation judge that appears to be the same API used to select training data, and the hyperparameter search is not described as a valid selection procedure. The theoretical efficiency motivation for the lower bound is also not established. These issues are fixable without changing the core method, but they must be addressed before the quantitative claims can be accepted.","major_comments":[{"comment":"The judge used to count \"harmful responses\" in Tables 2, 3, 6 and 7 is not stated in Section 4.3.3. The only binary harmful-rate judge introduced in the paper is the OpenAI Moderation API, which is also used in Section 4.1 to select the 1,000 harmful responses for D_safety-related. If the table counts come from that API, then the 100/300/500-example comparisons are evaluations by the same classifier that filtered the training data, and the headline data-efficiency result may partly reflect overfitting to the API rather than general safety. The DeBERTa curves in Figure 2(a-c) provide an independent signal, but only for the 1,000-example setting. Please (i) state the judge for each safety table, and (ii) report DeBERTa scores or human annotations for the reduced-data settings, using a judge not involved in data construction.","section":"§4.3.3 / Table 2; §4.3.1 / Figure 2(d); §4.1"},{"comment":"The penalty weight lambda is grid-searched separately for each model and each number of toxic prompts, but no validation split or selection criterion is described. If the grid search was evaluated on the four harmfulness benchmarks, all counts in Tables 2, 3, 6 and 7 are post-selection numbers and the reported safety levels are optimistic. Please define a held-out validation set, choose lambda on that set, and report test-benchmark numbers for the selected lambda only.","section":"Appendix A.2 / Table 4"},{"comment":"The claimed efficiency advantage of the lower bound is not established. For the one-hot data distribution P used in Equation (5), the exact EMD equals EMD(P,Q;d_c)=Σ_y Q(y)||e_t−e_y||²/2, which requires the same O(|V|) embedding inner products as the proposed lower bound and no linear-programming solver; the bound is therefore not needed for tractability in the setting actually used. In addition, the 1/(2|V|²) factor makes the numerical bound vacuous for realistic vocabulary sizes (below 10^-9 for |V|=32K). Please either use the exact one-hot EMD and remove the \"enabling more efficient optimization\" claim, or provide evidence that the lower-bound objective behaves better than the exact EMD under optimization.","section":"§3.1 / Proposition 1 / Appendix A.3"},{"comment":"The summary statement that EMD \"enables LLMs to learn safe responses with only 100 harmful examples\" is stronger than the appendix data. At 100 examples the EMD rows contain non-zero harmful counts in several cells (e.g., Llama 13b in Table 2: 2, 2, 1, 2; Llama3.1-8b in Table 6: 0, 7, 2, 1; Mistral 7b in Table 6: 0, 4, 0, 1). The claim should be quantified as \"near-zero on most benchmarks\" or \"lower than NLCL/STL in all settings,\" and should include variance across seeds if single-run counts are used.","section":"§4.3.3 / Table 6"}],"minor_comments":[{"comment":"The word \"seperately\" should be \"separately\".","section":"Figure 2 caption"},{"comment":"The text references \"Section 4.4.1\", but no such section exists; the evaluation protocol is described in Section 4.3.1.","section":"§4.3.6"},{"comment":"The appendix refers to the NLCL loss as the \"ORPO loss\"; the two names should be reconciled throughout the paper.","section":"Appendix A.4.1"},{"comment":"Table 8 (ASR and Mean Judge Score) is not mentioned in the main text and does not specify the judge, the dataset, or how the mean score is computed; add a reference and details, or remove the table.","section":"Table 8"},{"comment":"These tables report single counts without standard errors; given the small magnitudes, a few responses can change the comparison, so multi-seed ranges or a significance test would help.","section":"Tables 2, 3, 6, 7"},{"comment":"The number of generated responses before filtering to 1,000 is not reported, so the reader cannot assess how representative the selected harmful examples are of the base model's unsafe outputs.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main risk is the API-circularity issue; if the authors can supply an independent judge for the reduced-data settings and a clean validation protocol for lambda, the paper would be a solid contribution. I do not see a reason to reject on the current evidence, but the central claim cannot be accepted as it stands. The lower-bound contribution should be reframed or cut in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper makes a genuinely useful empirical claim — you can make an open-weight LLM refuse unsafe prompts by fine-tuning on 100 model-generated harmful responses, no safe responses needed, with a semantic penalty during SFT. I think the core effect is real; the theory around it is not, and the data-efficiency numbers are less secure than they look.\n\nWhat's new and good: the setup uses only the base model's own harmful outputs as training data, which is cheap and practical. The comparison to STL is fair (they give STL its safer data and still win). The over-alignment result is a nice observation: refusal examples in the SFT data are not the cause of the refusals-on-benign-prompts behavior; all methods trace the same safety-vs-overrefusal curve. DeBERTa scores give an independent check on the main safety direction, and the paper is transparent about code/data availability.\n\nSoft spots, in order of importance. First, the theoretical motivation is overstated. Proposition 1 is correct but vacuous: the 1/(2|V|^2) factor makes the lower bound useless for any real vocabulary, and for the one-hot target distribution used in Eq. 5 the exact EMD is cheaper to compute anyway. What is actually optimized is a mean-embedding difference, which is a known objective. The 'novel lower bound' framing should go. Second, the data-efficiency claim rests on Table 2, and those counts come from the OpenAI Moderation API — the same API used to select the training examples. That is a partial circularity. The DeBERTa plot for 1,000 examples shows real safety gains, so the method works at that size, but the 100/300/500 comparison has no independent judge. Third, lambda is grid-searched per model and data size without a described validation split; the reported results may be post-selection. Fourth, no error bars anywhere, and the appendix introduces ASR and Mean Judge Score without defining them.\n\nThese are all addressable. The empirical result is interesting enough to deserve a careful referee. I would send it to review and ask for an independent judge in the low-data regime, a validation protocol for lambda, and a rewritten theory section that says honestly what the loss is. The paper will be weaker without those fixes but it is not wrong in its main empirical direction.","headline":"Data-efficient safety tuning with harmful examples only is a real empirical result; the EMD theory is overstated and the low-data counts need an independent judge.","tokens_in":16329,"tokens_out":2707,"would_cite":true,"duration_ms":24268,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Supervised fine-tuning can make a base LLM refuse toxic prompts using only 100 model-generated harmful responses and no safe-response data.","keywords":["LLM safety","supervised fine-tuning","Earth Mover Distance loss","semantic loss","toxic prompt","over-refusal","data efficiency","contrastive data"],"falsifier":"Hold out a set of toxic prompts whose harmful-response categories were deliberately excluded from the moderation-filtered training examples, fine-tune with the 100-example EMD loss, and measure the harmful response rate; if the rate remains high, the penalty only transfers within the practiced categories rather than making the model generally safe.","tokens_in":15257,"feed_emoji":"🛡️","tokens_out":9160,"duration_ms":77733,"temperature":0.7,"pith_summary":"This paper claims that an LLM can be made considerably safer at the supervised fine-tuning stage without collecting safe responses or human feedback. The proposed approach, Toxicity Avoiding SFT, adds a semantic penalty to the standard fine-tuning objective: it pushes the model's next-token distribution away from the distributions seen in a small set of harmful responses that the model itself produced. The paper reports near-zero harmful responses on four harmfulness benchmarks with as few as 100 toxic examples, while preserving instruction-following quality and multiple-choice accuracy. It also reports that over-refusal follows a fixed trade-off with safety and is not caused by including refusal examples in the training data.","feed_headline":"100 harmful examples make an LLM refuse toxic prompts","feed_subtitle":"A semantic penalty pushes model output away from unsafe responses using only the model's own bad answers.","key_machinery":"The load-bearing object is the Earth Mover Distance between next-token distributions, computed with a semantic cost on normalized token embeddings and optimized through a novel lower bound. For next-token distributions $P$ (data) and $Q_\\theta$ (model), the cost is $d_c(\\hat e_w,\\hat e_{w'})=\\|\\hat e_w-\\hat e_{w'}\\|^2/2$; EMD is the minimal transport cost between $P$ and $Q_\\theta$. Because this cost is a squared norm and not a proper metric, standard EMD lower bounds do not apply, so the paper proves $EMD(P,Q_\\theta;d_c)\\ge \\frac{1}{2|V|^2}\\|\\sum_w P(w)\\hat e_w-\\sum_w Q_\\theta(w)\\hat e_w\\|^2$. Optimizing this lower bound separates the mean embedding of the model's predicted next token from the mean embedding of the harmful token, and the data distribution is treated as a one-hot vector at the observed harmful token. This lets a small set of harmful examples push away an entire semantic neighborhood rather than just a single token.","core_discovery":"The central claim is that safety can be learned from negative examples alone: instead of showing the model safe responses, one keeps the ordinary fine-tuning loss on safe instruction data and adds a penalty that makes the model's next-token distribution semantically far from the next-token distribution of recorded harmful responses. Concretely, given a toxic prompt and the harmful response the base model gave, the penalty maximizes the Earth Mover Distance between the model's predicted next-token distribution $Q_\\theta(\\cdot|w_{<t})$ and the observed data distribution $P(\\cdot|w_{<t})$ under the cosine-distance cost $d_c(\\hat e_w,\\hat e_{w'})=\\|\\hat e_w-\\hat e_{w'}\\|^2/2$. The paper proves a lower bound for this EMD: $EMD(P,Q_\\theta;d_c)\\ge \\frac{1}{2|V|^2}\\|\\sum_w P(w)\\hat e_w-\\sum_w Q_\\theta(w)\\hat e_w\\|^2$, which is maximized by separating the mean embeddings of the predicted and observed next tokens. With this loss, 100 harmful examples, filtered from the model's own outputs, produce near-zero harmful responses across four evaluation sets while maintaining response quality.","pith_inferences":["Beyond the paper, the penalty's dependence on the mean embedding of the next-token distribution suggests that data efficiency may vary across base models with different token-embedding geometries, and this is directly testable by running the same 100-example recipe on diverse base LLMs.","Beyond the paper, the reported evaluation sets may overlap thematically with the 100 to 1000 filtered training examples, so a conservative reading is that part of the gain reflects distribution matching; measuring transfer to disjoint, newly constructed attack categories would strengthen the transfer claim.","Beyond the paper, the over-refusal finding points to a lexical-trigger mechanism rather than a refusal-string mechanism; a natural experiment is to remove toxic-topic vocabulary from benign prompts and see whether over-refusal drops.","Beyond the paper, the lower bound could be tightened using Sinkhorn iterations or other EMD approximations; if a tighter bound improves safety per harmful example, the current lower bound is leaving signal on the table, and if not, the bound already captures the useful semantics."],"forward_implications":["Safety can be injected during ordinary supervised fine-tuning, so no reinforcement learning, human preference labels, or safe-response data are needed.","A training-data ratio of 0.005 between harmful and ordinary instruction examples is enough to achieve near-zero harmful responses on the four evaluated harmfulness sets.","The EMD penalty preserves or slightly improves instruction-following quality and multiple-choice accuracy compared with standard SFT, whereas a likelihood-based penalty needs more harmful examples.","Over-refusal and safety move together along the same curve across training stages and loss functions, indicating that refusal examples in the data are not the cause of over-refusal.","Augmenting the data with LLM-generated seemingly toxic but benign contrastive samples can reduce safety and, at high penalty weight, push the model into producing non-English responses."],"supporting_citations":[{"why":"Provides the Safety Tuned Llamas baseline that TA-SFT compares against; it trains on safe responses to toxic prompts and supplies the advantageously matched comparison.","marker":"Bianchi et al. [2023]"},{"why":"Provides the ORPO relative-odds loss that the paper's NLCL likelihood-penalty baseline adapts when only the losing response is available.","marker":"Hong et al. [2024]"},{"why":"Supplies the one-hot treatment of the next-token data distribution used in the EMD loss implementation.","marker":"Ren et al. [2023]"},{"why":"Gives the standard EMD lower-bound machinery for metric costs, which the paper extends to the squared-norm cosine distance.","marker":"Cohen and Guibas [1997]"},{"why":"Provides the XSTest benchmark used to measure over-refusal on seemingly toxic benign prompts.","marker":"Röttger et al. [2023]"},{"why":"Supplies the procedure for extracting toxic words and generating seemingly toxic contrastive prompts used in the augmentation study.","marker":"Cui et al. [2024]"},{"why":"Documents the degradation from training on generated data, cited to explain the non-English-response collapse observed after contrastive augmentation.","marker":"Shumailov et al. [2023]"},{"why":"Supports the observation that instruction-tuned base LLMs are vulnerable to toxic prompts and can generate harmful responses used as training data.","marker":"Qi et al. [2023]"},{"why":"KTO is the non-pairwise RLHF baseline included in the safety and quality comparisons.","marker":"Ethayarajh et al. [2024]"},{"why":"The Alpaca dataset supplies the 20,000 safety-unrelated instruction pairs used in the fine-tuning mixture.","marker":"Taori et al. [2023]"}],"fun_headline_variants":["100 bad answers teach LLM to refuse toxic prompts","LLM learns safety from its own harmful responses, no human labels","Semantic EMD penalty: 100 self-generated toxic replies train safe LLM","One hundred harmful samples flip an LLM's toxic behavior","From toxic to safe: 100 negative examples suffice for LLM safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The handful of model-generated harmful examples used for training must be representative of the unsafe responses the model would give to unseen toxic prompts; if whole classes of harmful behavior never appear in those examples, the learned penalty will not transfer to them.","fun_headline_variants_meta":{"raw":{"variants":["100 bad answers teach LLM to refuse toxic prompts","LLM learns safety from its own harmful responses, no human labels","Semantic EMD penalty: 100 self-generated toxic replies train safe LLM","One hundred harmful samples flip an LLM's toxic behavior","From toxic to safe: 100 negative examples suffice for LLM safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001164,"raw_usage":{"total_tokens":4823,"prompt_tokens":957,"completion_tokens":3866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":3776}},"tokens_in":573,"tokens_out":3866,"duration_ms":24698,"temperature":1.0,"reasoning_tokens":3776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:28:16.753445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a set of toxic prompts whose harmful-response categories were deliberately excluded from the moderation-filtered training examples, fine-tune with the 100-example EMD loss, and measure the harmful response rate; if the rate remains high, the penalty only transfers within the practiced categories rather than making the model generally safe.","supporting_citations":[],"review_version":1}