REVIEW 4 major objections 6 minor 27 references
Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Supervised fine-tuning can make a base LLM refuse toxic prompts using only 100 model-generated harmful responses and no safe-response data.
desk verdict Data-efficient safety tuning with harmful examples only is a real empirical result; the EMD theory is overstated and the low-data counts need an independent judge. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Earth Mover Distance between next-token distributions, computed with a semantic cost on normalized token embeddings and optimized through a novel lower bound. For next-token distributions $P$ (data) and $Q_\theta$ (model), the cost is $d_c(\hat e_w,\hat e_{w'})=\|\hat e_w-\hat e_{w'}\|^2/2$; EMD is the minimal transport cost between $P$ and $Q_\theta$. Because this cost is a squared norm and not a proper metric, standard EMD lower bounds do not apply, so the paper proves $EMD(P,Q_\theta;d_c)\ge \frac{1}{2|V|^2}\|\sum_w P(w)\hat e_w-\sum_w Q_\theta(w)\hat e_w\|^2$. Optimizing this lower bound separates the mean embedding of the model's predicted next token from the mean embedding of the harmful token, and the data distribution is treated as a one-hot vector at the observed harmful token. This lets a small set of harmful examples push away an entire semantic neighborhood rather than just a single token.
What would settle it
Hold out a set of toxic prompts whose harmful-response categories were deliberately excluded from the moderation-filtered training examples, fine-tune with the 100-example EMD loss, and measure the harmful response rate; if the rate remains high, the penalty only transfers within the practiced categories rather than making the model generally safe.
Extended reading notes
Core claim
The central claim is that safety can be learned from negative examples alone: instead of showing the model safe responses, one keeps the ordinary fine-tuning loss on safe instruction data and adds a penalty that makes the model's next-token distribution semantically far from the next-token distribution of recorded harmful responses. Concretely, given a toxic prompt and the harmful response the base model gave, the penalty maximizes the Earth Mover Distance between the model's predicted next-token distribution $Q_\theta(\cdot|w_{<t})$ and the observed data distribution $P(\cdot|w_{<t})$ under the cosine-distance cost $d_c(\hat e_w,\hat e_{w'})=\|\hat e_w-\hat e_{w'}\|^2/2$. The paper proves a lower bound for this EMD: $EMD(P,Q_\theta;d_c)\ge \frac{1}{2|V|^2}\|\sum_w P(w)\hat e_w-\sum_w Q_\theta(w)\hat e_w\|^2$, which is maximized by separating the mean embeddings of the predicted and observed next tokens. With this loss, 100 harmful examples, filtered from the model's own outputs, produce near-zero harmful responses across four evaluation sets while maintaining response quality.
Load-bearing premise
The handful of model-generated harmful examples used for training must be representative of the unsafe responses the model would give to unseen toxic prompts; if whole classes of harmful behavior never appear in those examples, the learned penalty will not transfer to them.
Editorial extensions
If this is right
- Safety can be injected during ordinary supervised fine-tuning, so no reinforcement learning, human preference labels, or safe-response data are needed.
- A training-data ratio of 0.005 between harmful and ordinary instruction examples is enough to achieve near-zero harmful responses on the four evaluated harmfulness sets.
- The EMD penalty preserves or slightly improves instruction-following quality and multiple-choice accuracy compared with standard SFT, whereas a likelihood-based penalty needs more harmful examples.
- Over-refusal and safety move together along the same curve across training stages and loss functions, indicating that refusal examples in the data are not the cause of over-refusal.
- Augmenting the data with LLM-generated seemingly toxic but benign contrastive samples can reduce safety and, at high penalty weight, push the model into producing non-English responses.
Reading between the lines
- Beyond the paper, the penalty's dependence on the mean embedding of the next-token distribution suggests that data efficiency may vary across base models with different token-embedding geometries, and this is directly testable by running the same 100-example recipe on diverse base LLMs.
- Beyond the paper, the reported evaluation sets may overlap thematically with the 100 to 1000 filtered training examples, so a conservative reading is that part of the gain reflects distribution matching; measuring transfer to disjoint, newly constructed attack categories would strengthen the transfer claim.
- Beyond the paper, the over-refusal finding points to a lexical-trigger mechanism rather than a refusal-string mechanism; a natural experiment is to remove toxic-topic vocabulary from benign prompts and see whether over-refusal drops.
- Beyond the paper, the lower bound could be tightened using Sinkhorn iterations or other EMD approximations; if a tighter bound improves safety per harmful example, the current lower bound is leaving signal on the table, and if not, the bound already captures the useful semantics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Toxicity Avoiding SFT (TA-SFT), a supervised fine-tuning scheme that augments a standard instruction dataset with a small set of harmful responses to toxic prompts, generated by the base model itself. The main loss is a negative Earth Mover Distance term with cosine embedding cost, implemented through a lower bound that is optimized as the squared distance between the one-hot embedding of the observed unsafe token and the model's expected next-token embedding; the authors also introduce an NLCL likelihood-penalty baseline. Experiments on Llama 7B/13B, Mistral 7B, and Llama 3.1 8B report large safety improvements on four harmfulness benchmarks, preservation of response quality on AlpacaEval and multiple-choice tasks, data efficiency down to 100 harmful examples, and analyses of over-alignment and contrastive augmentation.
Significance. The practical claim is attractive and, if confirmed, would lower the cost of safety tuning by using only model-generated harmful responses. The paper has real strengths: experiments span four base models; response-quality results are reported for several benchmarks; code and data are promised; Proposition 1 is proved correctly in Appendix A.3; and the over-alignment analysis is a useful negative result. The central data-efficiency claim, however, currently rests on an evaluation judge that appears to be the same API used to select training data, and the hyperparameter search is not described as a valid selection procedure. The theoretical efficiency motivation for the lower bound is also not established. These issues are fixable without changing the core method, but they must be addressed before the quantitative claims can be accepted.
major comments (4)
- [§4.3.3 / Table 2; §4.3.1 / Figure 2(d); §4.1] The judge used to count "harmful responses" in Tables 2, 3, 6 and 7 is not stated in Section 4.3.3. The only binary harmful-rate judge introduced in the paper is the OpenAI Moderation API, which is also used in Section 4.1 to select the 1,000 harmful responses for D_safety-related. If the table counts come from that API, then the 100/300/500-example comparisons are evaluations by the same classifier that filtered the training data, and the headline data-efficiency result may partly reflect overfitting to the API rather than general safety. The DeBERTa curves in Figure 2(a-c) provide an independent signal, but only for the 1,000-example setting. Please (i) state the judge for each safety table, and (ii) report DeBERTa scores or human annotations for the reduced-data settings, using a judge not involved in data construction.
- [Appendix A.2 / Table 4] The penalty weight lambda is grid-searched separately for each model and each number of toxic prompts, but no validation split or selection criterion is described. If the grid search was evaluated on the four harmfulness benchmarks, all counts in Tables 2, 3, 6 and 7 are post-selection numbers and the reported safety levels are optimistic. Please define a held-out validation set, choose lambda on that set, and report test-benchmark numbers for the selected lambda only.
- [§3.1 / Proposition 1 / Appendix A.3] The claimed efficiency advantage of the lower bound is not established. For the one-hot data distribution P used in Equation (5), the exact EMD equals EMD(P,Q;d_c)=Σ_y Q(y)||e_t−e_y||²/2, which requires the same O(|V|) embedding inner products as the proposed lower bound and no linear-programming solver; the bound is therefore not needed for tractability in the setting actually used. In addition, the 1/(2|V|²) factor makes the numerical bound vacuous for realistic vocabulary sizes (below 10^-9 for |V|=32K). Please either use the exact one-hot EMD and remove the "enabling more efficient optimization" claim, or provide evidence that the lower-bound objective behaves better than the exact EMD under optimization.
- [§4.3.3 / Table 6] The summary statement that EMD "enables LLMs to learn safe responses with only 100 harmful examples" is stronger than the appendix data. At 100 examples the EMD rows contain non-zero harmful counts in several cells (e.g., Llama 13b in Table 2: 2, 2, 1, 2; Llama3.1-8b in Table 6: 0, 7, 2, 1; Mistral 7b in Table 6: 0, 4, 0, 1). The claim should be quantified as "near-zero on most benchmarks" or "lower than NLCL/STL in all settings," and should include variance across seeds if single-run counts are used.
minor comments (6)
- [Figure 2 caption] The word "seperately" should be "separately".
- [§4.3.6] The text references "Section 4.4.1", but no such section exists; the evaluation protocol is described in Section 4.3.1.
- [Appendix A.4.1] The appendix refers to the NLCL loss as the "ORPO loss"; the two names should be reconciled throughout the paper.
- [Table 8] Table 8 (ASR and Mean Judge Score) is not mentioned in the main text and does not specify the judge, the dataset, or how the mean score is computed; add a reference and details, or remove the table.
- [Tables 2, 3, 6, 7] These tables report single counts without standard errors; given the small magnitudes, a few responses can change the comparison, so multi-seed ranges or a significance test would help.
- [§4.1] The number of generated responses before filtering to 1,000 is not reported, so the reader cannot assess how representative the selected harmful examples are of the base model's unsafe outputs.
Circularity Check
Data-efficiency headline is measured with the same OpenAI Moderation API used to curate the harmful training set, making the 100-example claim partly circular; independent DeBERTa curves only partially rescue it.
-
fitted input called prediction
[Section 4.1 (training data curation) and Section 4.3.1/4.3.3, Table 2 (evaluation)]
"We use the SFTed LLM to generate harmful responses, and then apply the OpenAI moderation API to extract 1,000 responses that are harmful from the LLM under consideration. ... we used the OpenAI Moderation API as a secondary evaluation method. This API provides both a harmfulness score (in [0,1]) ... and a binary tag indicating whether the response is safe. In Figure 2(d), we show the percentage of tagged harmful responses across all four test datasets."
The same classifier is used on both sides of the experiment: it selects the 1,000 harmful examples that form D_safety-related, and the 'number of harmful responses' reported in Table 2 (Section 4.3.3) is counted with that same binary tag, since DeBERTa yields a continuous score rather than counts. The headline data-efficiency result — EMD safe with as few as 100 harmful examples — therefore measures how well the model suppresses the exact filter that curated its only negative examples, rather than measuring general safety with an independent judge. A model can reduce API-flagged harmfulness without reducing human-judged harmfulness, so the 100/300/500-example comparison is partly circular.
full rationale
The method's derivation is not circular by construction: Proposition 1 is proved in Appendix A.3 using Cauchy-Schwarz, the EMD loss in Eq. (5) is a standard negative-margin training objective, and the paper contains no load-bearing self-citation chain; the one external citation used for the one-hot approximation (Ren et al. 2023) is not the authors' own work. The circularity is at the evaluation level: Section 4.1 filters D_safety-related with the OpenAI Moderation API, and Section 4.3.1 introduces the same API as the judge whose binary tag underlies the harmful-response counts in Table 2 and Table 3. This reduces the paper's headline data-efficiency claim to 'suppress the training filter', which is a partial circularity, not a full one, because the benchmark prompts are unseen and because the independent DeBERTa harmfulness scores in Fig 2(a-c) show genuine safety improvement for the 1000-example setting. A compounding but distinct issue: Appendix A.2 reports grid-searched lambda values per model and dataset size with no validation split or selection criterion described; if lambda was selected on the four evaluation benchmarks, the Table 2 numbers are post-selection. That is a missing-support concern, not a by-construction circularity. Overall score 4: the central claim still has independent content via DeBERTa and quality benchmarks, but the strongest quantitative claim is entangled with its own training filter.
Assumptions & free parameters
free parameters (2)
- penalty weight lambda (EMD loss) =
0.49 to 9.00 across models and dataset sizes (Table 4)
- penalty weight lambda (NLCL loss) =
2.40 to 55.00 across models and dataset sizes (Table 4)
assumptions (5)
- standard math Cauchy-Schwarz inequality and the fact that a coupling gamma has entries in [0,1].
- domain assumption Token embeddings in normalized space are meaningful for semantic similarity; cosine distance approximates semantic distance.
- domain assumption The data next-token distribution P can be replaced by a one-hot vector of the observed token.
- domain assumption Penalizing the model's next-token distribution on teacher-forced unsafe prefixes transfers to the model's own generated responses.
- domain assumption OpenAI Moderation API labels correctly identify harmful responses.
Cite this review
Pith. "Pith review of Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs." pith.science (2026). https://pith.science/paper/RLOFPROP
@misc{pith2026241206843,
author = {Pith},
title = {Pith review of: Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/RLOFPROP}},
note = {Machine review of arXiv:2412.06843}
}
read the original abstract
Large Language Models (LLMs) generating unsafe responses to toxic prompts is a significant issue in their applications. While various efforts aim to address this safety concern, previous approaches often demand substantial human data collection or rely on the less dependable option of using another LLM to generate corrective data. In this paper, we aim to take this problem and overcome limitations of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the unsafe LLM itself. By employing a semantic cost combined with a negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe responses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over-alignment and potential degradation of language capabilities when using contrastive data.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[6]
Fine-tuning language models from human preferences
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593,
arXiv 1909
-
[7]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,
-
[8]
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul R ¨ottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07875,
-
[9]
The curse of recursion: Training on generated data makes models forget
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493,
-
[10]
R-judge: Benchmarking safety risk awareness for llm agents
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. R-judge: Benchmarking safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019,
-
[11]
Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691, 2(4):5,
-
[12]
Kto: Model alignment as prospect theoretic optimization
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306,
-
[13]
Safety fine-tuning at (almost) no cost: A baseline for vision large language models
Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. arXiv preprint arXiv:2402.02207,
Show all 27 references
-
[14]
Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693,
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693,
-
[15]
Safe lora: the silver lining of reducing safety risks when fine-tuning large language models
Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models. arXiv preprint arXiv:2405.16833,
-
[17]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,
-
[18]
Mistral 7b
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,
-
[19]
The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
11 Semantic loss guided data efficient supervised fine tuning for Safe Responses in LLMs A PREPRINT Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of mo...
-
[20]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,
-
[22]
Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543,
-
[23]
Boolq: Exploring the surprising difficulty of natural yes/no questions
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,
1905 arXiv
-
[24]
Can a suit of armor conduct electricity? a new dataset for open book question answering
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789,
-
[25]
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le
URL https://zenodo.org/records/12608602. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652,
-
[26]
Xstest: A test suite for identifying exaggerated safety behaviours in large language models.arXiv preprint arXiv:2308.01263,
Paul R ¨ottger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models.arXiv preprint arXiv:2308.01263,
-
[27]
Mixtral of experts
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,
-
[1997]
Emo: Earth mover distance optimization for auto-regressive language modeling
Siyu Ren, Zhiyong Wu, and Kenny Q Zhu. Emo: Earth mover distance optimization for auto-regressive language modeling. arXiv preprint arXiv:2310.04691,
-
[2019]
Language models are few-shot learners
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165,
2005 arXiv
-
[2020]
The woman worked as a babysitter: On biases in language generation
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326,
1909 arXiv
-
[2021]
Or-bench: An over-refusal benchmark for large language models
Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947,
-
[2022]
Unveiling the implicit toxicity in large language models
Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the implicit toxicity in large language models. arXiv preprint arXiv:2311.17391,
-
[2023]
Realtoxicityprompts: Evaluating neural toxic degeneration in language models
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462,
2009 arXiv
-
[2024]
Predictability and surprise in large generative models
1https://anonymous.4open.science/r/ICLR-2025-anonymous-code-submission-3CC8 10 Semantic loss guided data efficient supervised fine tuning for Safe Responses in LLMs A PREPRINT Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova ...
2025
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.