Pith. sign in

REVIEW 4 major objections 5 minor 59 references

DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DataRx claims that safety samples should be chosen by the size of the gap between a model's current response and its reference refusal, and shows this cuts average attack success rate from 59.23% to 13.70%.

desk verdict A sensible gap-based safety data selection method with broad but currently unvalidated empirical claims; needs a dedup check and variance before the headline numbers can be trusted. read the letter →

arxiv 2608.04322 v1 pith:4CQASH7Y submitted 2026-08-05 cs.CL

Junbo Zhang , Qianli Zhou , Xinyang Deng , Wen Jiang This is my paper · ORCID
classification cs.CL
keywords largelanguagemodelssafetyalignmentfine-tuningdataselectionattacksuccessratehiddenrepresentationsadaptationscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the best safety examples to mix into task-specific fine-tuning are the ones that supply refusal signals the target model currently lacks, not the ones it already behaves safely on. It proposes DataRx, which ranks candidate safety examples by a Safety Adaptation Score: the gap between the refusal signal of the reference safe response and the refusal signal of the model's own native response, measured in the model's layer-wise hidden representations. With only 1% additional safety samples from BeaverTails, DataRx reduces the average attack success rate of Llama3-8B-Instruct from 59.23% under random mixing to 13.70%, with similar gains on Qwen2.5 and Mistral. A sympathetic reader would care because it suggests safety can be preserved without expensive rewriting or extra inference-time guardrails, just by choosing the right samples from existing safety datasets.

What carries the argument

The load-bearing object is the refusal score R(d), defined for an instruction-response pair d as the mean over all layers of the difference cos(h_l(d), C^l_r) - cos(h_l(d), C^l_c), where h_l(d) is the hidden state of the final end-of-sequence token at layer l and C^l_r and C^l_c are the average hidden states of 100 refusal and 100 compliance anchor examples drawn from a paired circuit-breaker style dataset. The selection mechanism uses the Safety Adaptation Score SAS_i = R(safe reference pair) - R(native pair), which requires first generating the target model's own response to each harmful prompt. This score does the work of turning 'what the model gets wrong' into a ranking over existing safety data, so only the top-K samples are mixed into task-specific fine-tuning.

What would settle it

Train three copies of the same model on top-ranked, bottom-ranked, and randomly selected safety samples at several mixing ratios and learning rates; if bottom-ranked samples ever match or beat top-ranked samples on held-out safety benchmarks across multiple safety datasets, then the gap-based ordering is not the cause of the safety gains. A more direct check is to label responses by human refusal quality and test whether the refusal score R(d) increases monotonically with judged quality.

Watch

Extended reading notes

Core claim

The central discovery is a gap-based, representation-level scoring for safety data selection. For a candidate sample consisting of a harmful prompt paired with a refusal reference, DataRx computes the refusal scores of the safety reference pair and of the target model's native response pair, each being the mean over transformer layers of the difference between cosine similarity to a refusal anchor center and cosine similarity to a compliance anchor center. The Safety Adaptation Score, defined as the difference between those two refusal scores, measures how much refusal capability the sample would add to the current model. Sampling the top-K candidates and mixing them with downstream data at a ratio of 100:1 preserves downstream accuracy within roughly two percentage points while sharply lowering attack success; on BeaverTails, for example, DataRx brings the average attack success rate to 13.70% for Llama3, 12.29% for Qwen2.5, and 62.43% for Mistral, whereas random mixing reaches 59.23%, 50.08%, and 88.31%, respectively.

Load-bearing premise

The entire ranking rests on assuming that a scalar refusal score built from cosine similarity to two averaged center vectors in hidden state space faithfully tracks how much refusal supervision a sample truly adds; if that score does not reflect genuine refusal quality, the selected samples may not be the critical ones.

Editorial extensions

If this is right

  • Safety can be preserved during task-specific fine-tuning with a fixed 100:1 task-to-safety mixing ratio, and downstream accuracy changes by roughly two percentage points or less.
  • Response length, category balance, and paraphrase-based distribution matching are not reliable proxies for corrective value; the paper reports that long-response selection can even raise attack success above downstream-only fine-tuning.
  • DataRx composes with safety-data synthesis: applying it to generated safety data further lowers average attack success, for instance on the GR-SAP generated dataset from 25.21% to 19.49% for Llama3.
  • Higher learning rates and weaker baseline safety make degradation more severe, but the same gap-based selection procedure still mitigates it across all three tested models and seven downstream tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its gap score could double as a diagnostic for other alignment failures: replacing the refusal and compliance anchors with anchors for over-refusal, sycophancy, or hallucination would yield the same missingness-aware selection recipe.
  • A testable extension is an iterative active-learning loop: use SAS to pick samples, fine-tune, recompute SAS on the updated model, and pick again; the paper's representation-shift evidence suggests successive rounds may need different samples.
  • Because SAS is computed on a single final-token hidden state averaged over all layers, it likely tracks a direction-style refusal signal, which would imply the method transfers to other open-weight models without retraining the scorer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. DataRx proposes a missingness-aware sampling method for selecting safety-critical examples from external safety datasets to mitigate safety degradation during task-specific fine-tuning. The method defines a refusal score as the layer-averaged difference in cosine similarity to refusal and compliance anchor centers (Eqs. 3–8), then ranks candidate safety samples by the Safety Adaptation Score (SAS), i.e., the gap between the refusal signal of the reference safety response and the target model's native response (Eq. 11), and selects the top-K samples (Eq. 12). The paper reports extensive experiments across three LLMs, seven downstream tasks, two candidate safety datasets, and three safety benchmarks, claiming that DataRx reduces the average attack success rate of Llama3-8B-Instruct from 59.23% under random mixing to 13.70% with 1% BeaverTails safety data, while preserving downstream task accuracy.

Significance. If the central claims hold, DataRx is a useful and cheap data-centric defense: it requires no inference-time overhead, no external strong model for rewriting, and it is complementary to safety-data synthesis methods. The paper provides broad and consistent evidence that hard safety examples, defined by the target model's own behavioral gap, are more effective than easy ones (Figs. 3–4), and the representation-based selection criterion is falsifiable and easy to reproduce. The main weaknesses are that the evaluation protocol lacks variance measures and an overlap analysis between selected safety samples and evaluation benchmarks, and the paper does not validate that the proposed refusal score actually measures refusal quality. These issues currently leave the strongest quantitative claims insufficiently supported.

major comments (4)
  1. [III-C, Eq. (12); IV-B, Tables II–III] The reported ASR improvements may be inflated by overlap between the selected safety samples and the evaluation benchmarks. DataRx deliberately ranks and selects prompts for which the target model's native response is unsafe, and it selects from BeaverTails/Aegis, which are broad red-team collections; DirectHarm4, HarmBench, and HEx-PHI are likewise collections of harmful prompts. If any selected prompt, or a near paraphrase, also appears in an evaluation set, fine-tuning directly memorizes a refusal for a test instance and the measured gain does not reflect general safety improvement. The paper reports no deduplication or overlap check. I ask the authors to report exact-match and embedding-based overlap between the selected subsets and each evaluation benchmark, re-run the main comparisons after removing overlapping or near-duplicate examples, and state whether any headline numbers change.
  2. [III-B, Eqs. (3)–(8)] The refusal score R(d) is the core measurement of the method, but its validity is not demonstrated. There is no evidence that a layer-averaged cosine difference to two anchor centroids is monotonic with actual refusal quality, that the final eos-token hidden state represents the safety of a multi-token response, or that 100 refusal and 100 compliance anchors are sufficient and stable. Because Eq. (11) ranks samples by differences in this score, any failure of these assumptions breaks the method's rationale. Please add validation experiments: correlation of R(d) with external safety labels and with the model's observed refusal rate; an ablation of token position (final token vs first response token vs mean pooling); an anchor-set sensitivity analysis; and a qualitative error analysis showing that the ranking aligns with human judgment.
  3. [IV-A, 'Evaluation metrics'; Tables II–III] All reported ASR and ACC values come from a single training run with no variance or significance testing, and safety labels come from a single judge model (Llama-Guard-3-8B) without validation against a second judge or human labels. Several claimed advantages over strong baselines are small (e.g., Llama3 with Aegis: DataRx 4.31% vs SSS-B 6.04%; Mistral with Aegis: DataRx 29.96% vs Paraphrase 30.14%), so without error bars the central claim of consistent superiority is not established. Please run at least three seeds per condition and report mean±standard deviation, and validate the safety judge on a sample against human labels or a second strong judge.
  4. [IV-D, Figs. 6–8] The paper's main narrative focuses on ASR reduction, but the mixing-ratio study shows that DataRx can substantially increase over-refusal on XSTest, especially at higher ratios on Aegis. Over-refusal is itself a safety and utility cost, and the abstract's headline result (59.23% to 13.70% on BeaverTails) is not accompanied by the over-refusal numbers for that configuration. The main tables should include XSTest refusal rates, or a combined safety-utility metric, so that the reader can judge whether the ASR gains come at the cost of excessive brittleness on benign prompts. As written, the claim that DataRx preserves utility is supported only for downstream task accuracy, not for benign-prompt behavior.
minor comments (5)
  1. [II, Eq. (1)] The Huber contamination model is written as an unnormalized sum P = P_task + P_safe without a mixing coefficient, so it is not a probability distribution as stated; it is also not used in the rest of the paper. Please clarify or remove it.
  2. [Tables II and III; Fig. 3] The column header 'DockBlocks' is inconsistent with the dataset name 'DocBlocks' used in Table I and Fig. 2; please correct the spelling throughout.
  3. [III-C and Table VI] Section III-C states that selection is restricted to harmful instructions paired with refusal responses, but the low-score example in Table VI has an alignment response that is not a refusal ('I have a friend who is very fond of eating bugs...'). Please clarify the filtering process or correct the example.
  4. [IV-A and VI] The paper does not include a reproducibility statement or a data/code availability section; given the empirical nature of the work and the single-run evaluation, please add a clear statement about code, data, and random seeds.
  5. [IV-I] The section 'Why less is more?' is speculative and cites reasoning-data papers without establishing a direct connection to the safety setting; please either move this material to the discussion or integrate it with concrete evidence from the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DataRx's safety adaptation score is an independently computed selection heuristic, not a fitted predictor of the post-fine-tuning attack success rate it is evaluated against.

full rationale

DataRx defines a per-sample Safety Adaptation Score, SAS_i = R_safe,i - R_orig,i (Eq. 11), where R_safe,i and R_orig,i are refusal scores computed from the target model's hidden states and the candidate safety dataset's reference response (Eqs. 3-10), using anchor centers built from an external dataset (Circuit Breaker Dataset). The selection rule (Eq. 12) ranks candidates and takes the top-K samples before fine-tuning. The evaluation target, post-fine-tuning ASR on DirectHarm4, HarmBench, and HEx-PHI, is not used to set any constant, center, or threshold in this score, and no parameter of DataRx is fit to the evaluation outcome. The claim that high-SAS samples provide stronger corrective supervision is a causal hypothesis tested empirically, not an identity. The only self-citations appear as background or peripheral references (e.g., [7] in a list of studies on fine-tuning safety degradation, and [46] on over-refusal) and are not load-bearing: the central derivation does not reduce to them. The paper itself explicitly leaves the mechanism open ('Future research can further investigate why a small number of safety-critical examples can effectively mitigate safety degradation'), which further indicates the reported effect is an empirical finding rather than a definitional consequence. Concerns about possible overlap between selected safety samples and evaluation benchmarks are a validity or leakage threat, not a circularity of the derivation, and no equation in the paper encodes the evaluation outcome into the selection score. The derivation chain is therefore self-contained.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

DataRx is a heuristic, parameter-free scoring method. Its central claim rests on domain assumptions about the geometry of hidden representations and about the validity of the evaluation judge. The anchor dataset size, layer aggregation, and selection budget are hand-chosen. There are no fitted parameters in the scoring formula, and no new entities are introduced.

free parameters (3)
  • Number of anchor examples per behavior class = 100 safe, 100 unsafe
    Hand-chosen: 5 examples per 20 safety categories sampled from Circuit Breaker Dataset (Section IV-A). The refusal and compliance centers in Eq. (3) depend on these anchors, and the paper does not study sensitivity to this choice.
  • Selection budget K = top-100 in ablations; 1% of downstream data (ratio 100:1) in main experiments
    Chosen by hand as a fixed mixing ratio; the ratio study in Section IV-D shows safety and over-refusal trade-offs vary with K.
  • Layer aggregation = all L layers, equal weight
    Chosen to avoid single-layer selection issues (Section III-C). The paper provides no evidence that equal weighting is optimal; it is a modeling decision.
assumptions (5)
  • domain assumption Hidden states of the final <eos> token encode sufficient safety-relevant information to distinguish refusal from compliance.
    Invoked in Section III-B when defining centers and refusal scores from h_l(d).
  • domain assumption Cosine similarity in the residual stream is an appropriate distance for comparing safety behavior.
    Used in Eq. (4); no justification is given other than convention.
  • domain assumption The 100-example anchor datasets from Circuit Breaker are representative of refusal and compliance behaviors for each target model.
    Section IV-A constructs anchors from 20 categories; the paper does not analyze coverage or sensitivity to the anchor choice.
  • standard math Huber contamination model applies to the mixture of task and safety data distributions.
    Eq. (1) states the mixture P = P_task + P_safe without the mixing coefficient epsilon; the formulation is informal and the method never uses this model quantitatively.
  • domain assumption Attack success rate measured by Llama-Guard-3-8B is a faithful proxy for model safety after fine-tuning.
    Used in all experiments; no validation against human judgments or other classifiers is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuning." pith.science (2026). https://pith.science/paper/4CQASH7Y

@misc{pith2026260804322,
  author       = {Pith},
  title        = {Pith review of: DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4CQASH7Y}},
  note         = {Machine review of arXiv:2608.04322}
}
read the original abstract

Task-specific fine-tuning can improve the performance of large language models (LLMs) on downstream tasks. However, our study reveals that task-specific fine-tuning can also weaken the safety guardrails of aligned LLMs. A widely adopted strategy for preserving safety during fine-tuning is to incorporate safety data. Although previous studies have shown that randomly mixing safety data can alleviate safety degradation, the underlying principle determining why some safety examples are more effective than others still remains unclear. In this paper, we propose DataRx, a missingness-aware sampling method for selecting safety-critical examples. DataRx is based on the hypothesis that a safety sample is more effective when the selected examples provide safety signals that fill the missing parts of LLMs' safety capabilities. DataRx's key insight is leveraging high-dimensional hidden representations rather than discrete tokens to quantify the safety signal gap between the target model's native response and the safety reference response. The results show that, with only 1% additional safety samples from BeaverTails, DataRx reduces the average attack success rate of Llama3-8B-Instruct across seven downstream tasks from 59.23% under random sampling to 13.70%. In addition, DataRx can be combined with the existing safety data synthesis method to further enhance safety defenses during fine-tuning. We hope that DataRx will inspire more data-centric defense research.

Figures

Figures reproduced from arXiv: 2608.04322 by the authors.

Figure 1
Figure 1. Overview of the DataRx framework. DataRx identifies safety-critical [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of safety degradation and fine-tuning accuracy under different learning rates for task-specific LLM fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Attack success rates of Llama3, Qwen2.5, and Mistral after fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Overview of the DataRx framework. (a) Contrastive safety pattern extraction: DataRx bulids layer-wise refusal-related safety representations by [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Impact of safety data mixing ratios on safety performance and over [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Impact of safety data mixing ratios on safety performance and over [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 9
Figure 9. Figure 9: PCA shift of safety representations under fine-tuning with top- and [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Response length distribution of top- and bottom-ranked safety [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 30 canonical work pages

  1. [1]

    Building efficient llm pipeline for human mobility prediction,

    C. Wang, S. Zhou, L. Chen, and S. Shang, “Building efficient llm pipeline for human mobility prediction,” inProceedings of the 33rd ACM International Conference on Advances in Geographic Information Systems, 2025, pp. 1230–1233

    work page 2025

  2. [2]

    Bridging cross-domain time series: Efficient federated anomaly detection with sharded llms,

    R. Xu, H. Miao, S. Wang, Y . Zhao, B. Yang, Y . Gao, P. S. Yu, and C. S. Jensen, “Bridging cross-domain time series: Efficient federated anomaly detection with sharded llms,”IEEE Transactions on Knowledge and Data Engineering, 2026

    work page 2026

  3. [3]

    Llm-based agents for tool learning: A survey: W. xu et al

    W. Xu, C. Huang, S. Gao, and S. Shang, “Llm-based agents for tool learning: A survey: W. xu et al.”Data Science and Engineering, vol. 10, no. 4, pp. 533–563, 2025

    work page 2025

  4. [4]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  5. [5]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  6. [6]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 30 988–31 043. JOURNAL NAME, VOL. XX, NO. XX, MONTH YEAR 12

    work page 2024

  7. [7]

    DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning

    J. Zhang, Q. Zhou, X. Deng, W. Jiang, J. Pan, and J. Zhu, “Datashield: Safety-degrading data filtering for llm benign instruction fine-tuning,” arXiv preprint arXiv:2606.00160, 2026

    work page Pith review arXiv 2026

  8. [8]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,

    F. Bianchi, M. Suzgun, G. Attanasio, P. R ¨ottger, D. Jurafsky, T. Hashimoto, and J. Y . Zou, “Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 34 196–34 216

    work page 2024

Show all 59 references
  1. [9]

    Token-level adversarial prompt detection based on perplexity measures and contextual information,

    Z. Hu, G. Wu, S. Mitra, R. Zhang, T. Sun, H. Huang, and V . Swami- nathan, “Token-level adversarial prompt detection based on perplexity measures and contextual information,”arXiv preprint arXiv:2311.11509, 2023

    2023 arXiv

  2. [10]

    Rain: Your lan- guage models can align themselves without finetuning,

    Y . Li, F. Wei, J. Zhao, C. Zhang, and H. Zhang, “Rain: Your lan- guage models can align themselves without finetuning,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 20 580– 20 595

    2024

  3. [11]

    Exploring the limits of transfer learning with a unified text- to-text transformer,

    R. Colin, “Exploring the limits of transfer learning with a unified text- to-text transformer,”J. Mach. Learn. Res., vol. 21, 2020

    2020

  4. [12]

    Data to defense: The role of curation in aligning large language models against safety compromise,

    X. Liu, J. Liang, L. Tang, M. Ye, W. Ma, and Z. Xi, “Data to defense: The role of curation in aligning large language models against safety compromise,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 12 822–12 837

    2025

  5. [13]

    Do as i do (safely): Mitigating task-specific fine-tuning risks in large language models,

    F. Eiras, A. Petrov, P. Torr, M. P. Kumar, and A. Bibi, “Do as i do (safely): Mitigating task-specific fine-tuning risks in large language models,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 45 614–45 632

    2025

  6. [14]

    Self-distillation bridges distribution gap in language model fine-tuning,

    Z. Yang, T. Pang, H. Feng, H. Wang, W. Chen, M. Zhu, and Q. Liu, “Self-distillation bridges distribution gap in language model fine-tuning,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 1028– 1043

    2024

  7. [15]

    Gr-sap: Generative replay for safety alignment preservation during fine-tuning,

    Z. Fang, J. Zhou, and H. Chen, “Gr-sap: Generative replay for safety alignment preservation during fine-tuning,”arXiv preprint arXiv:2603.10243, 2026

    2026

  8. [16]

    Aegis2. 0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails,

    S. Ghosh, P. Varshney, M. N. Sreedhar, A. Padmakumar, T. Rebedea, J. R. Varghese, and C. Parisien, “Aegis2. 0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the As...

    2025

  9. [17]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset,

    J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y . Wang, and Y . Yang, “Beavertails: Towards improved safety alignment of llm via a human-preference dataset,”Advances in Neural Information Processing Systems, vol. 36, pp. 24 678–24 704, 2023

    2023

  10. [18]

    Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection,

    H. Shen, P.-Y . Chen, P. Das, and T. Chen, “Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 31 243–31 264

    2025

  11. [19]

    How to fine-tune safely on a budget: Model adaptation using minimal resources,

    A. C. Pham, M. Thalanki, M. Sun, A. Chaloo, A. Gupta, T. Xia, A. Mate, E. Nosakhare, and S. Srinivasan, “How to fine-tune safely on a budget: Model adaptation using minimal resources,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: In...

    2025

  12. [20]

    Sft doesn’t always hurt general capa- bilities: Revisiting domain-specific fine-tuning in llms,

    J. Lin, Z. Wang, K. Qian, T. Wang, A. Srinivasan, H. Zeng, R. Jiao, X. Zhou, J. Gesi, D. Wanget al., “Sft doesn’t always hurt general capa- bilities: Revisiting domain-specific fine-tuning in llms,” inInternational Conference on Learning Representations, vol. 2026, 2026, pp. 4...

    2026

  13. [21]

    Robust estimation of a location parameter,

    P. J. Huber, “Robust estimation of a location parameter,” inBreak- throughs in statistics: Methodology and distribution. Springer, 1992, pp. 492–518

    1992

  14. [22]

    Refusal in language models is mediated by a single direction,

    A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,”Advances in Neural Information Processing Systems, vol. 37, pp. 136 037–136 083, 2024

    2024

  15. [23]

    On the role of attention heads in large language model safety,

    Z. Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, K. Wang, Y . Liu, J. Fang, and Y . Li, “On the role of attention heads in large language model safety,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 84 042–84 071

    2025

  16. [24]

    Detoxifying large language models via knowledge editing,

    M. Wang, N. Zhang, Z. Xu, Z. Xi, S. Deng, Y . Yao, Q. Zhang, L. Yang, J. Wang, and H. Chen, “Detoxifying large language models via knowledge editing,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 3093–3118

    2024

  17. [25]

    Qwen2.5 technical report,

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

  18. [26]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnieret al., “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  19. [27]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.” Iclr, vol. 1, no. 2, p. 3, 2022

    2022

  20. [28]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakanoet al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  21. [29]

    Measuring mathematical problem solving with the math dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,”arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  22. [30]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” inProceedings of the 57th annual meeting of the association for computational linguistics, 2019, pp. 4791–4800

    2019

  23. [31]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 8732–8740

    2020

  24. [32]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,”Applied Sciences, vol. 11, no. 14, p. 6421, 2021

    2021

  25. [33]

    Magicoder: Empow- ering code generation with oss-instruct,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Empow- ering code generation with oss-instruct,” 2024

    2024

  26. [34]

    Multi- lingual contextualization of large language models for document-level machine translation,

    M. M. Ramos, P. Fernandes, S. Agrawal, and A. F. Martins, “Multi- lingual contextualization of large language models for document-level machine translation,”arXiv preprint arXiv:2504.12140, 2025

    2025 arXiv

  27. [35]

    Improving alignment and robustness with circuit breakers,

    A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, J. Z. Kolter, M. Fredrikson, and D. Hendrycks, “Improving alignment and robustness with circuit breakers,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  28. [36]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  29. [37]

    Comet-22: Unbabel-ist 2022 submission for the metrics shared task,

    R. Rei, J. G. De Souza, D. Alves, C. Zerva, A. C. Farinha, T. Glushkova, A. Lavie, L. Coheur, and A. F. Martins, “Comet-22: Unbabel-ist 2022 submission for the metrics shared task,” inProceedings of the Seventh Conference on Machine Translation (WMT), 2022, pp. 578–585

    2022

  30. [38]

    Keeping llms aligned after fine-tuning: The crucial role of prompt templates,

    K. Lyu, H. Zhao, X. Gu, D. Yu, A. Goyal, and S. Arora, “Keeping llms aligned after fine-tuning: The crucial role of prompt templates,”Ad- vances in Neural Information Processing Systems, vol. 37, pp. 118 603– 118 631, 2024

    2024

  31. [39]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Liet al., “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,”arXiv preprint arXiv:2402.04249, 2024

    2024 arXiv

  32. [40]

    Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning,

    H. Zhao, M. Andriushchenko, F. Croce, and N. Flammarion, “Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning,”arXiv preprint arXiv:2402.04833, 2024

    2024 arXiv

  33. [41]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms,

    S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y . Lin, N. Lambert, Y . Choi, and N. Dziri, “Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms,”Advances in neural information processing systems, vol. 37, pp. 8093–8131, 2024

    2024

  34. [42]

    Qwen3guard technical report,

    H. Zhao, C. Yuan, F. Huang, X. Hu, Y . Zhang, A. Yang, B. Yu, D. Liu, J. Zhou, J. Linet al., “Qwen3guard technical report,”arXiv preprint arXiv:2510.14276, 2025

    2025 arXiv

  35. [43]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” inProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), 201...

    2019

  36. [44]

    Safety alignment should be made more than just a few tokens deep,

    X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson, “Safety alignment should be made more than just a few tokens deep,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 54 911–54 941

    2025

  37. [45]

    XSTest: A test suite for identifying exaggerated safety behaviours in large language models,

    P. R ¨ottger, H. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “XSTest: A test suite for identifying exaggerated safety behaviours in large language models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu...

    2024

  38. [46]

    Understanding and mitigating over-refusal for large language models via safety repre- sentation,

    J. Zhang, R. Chen, Q. Zhou, X. Deng, and W. Jiang, “Understanding and mitigating over-refusal for large language models via safety repre- sentation,”arXiv preprint arXiv:2511.19009, 2025

    2025

  39. [47]

    Un- learning isn’t deletion: Investigating reversibility of machine unlearning in llms,

    X. Xu, X. Yue, Y . Liu, Q. Ye, H. Zheng, P. Hu, M. Du, and H. Hu, “Un- learning isn’t deletion: Investigating reversibility of machine unlearning in llms,”arXiv preprint arXiv:2505.16831, 2025

    2025 arXiv

  40. [48]

    Lima: Less is more for alignment,

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. Yuet al., “Lima: Less is more for alignment,”Advances in Neural Information Processing Systems, vol. 36, pp. 55 006–55 021, 2023

    2023

  41. [49]

    Limo: Less is more for reasoning,

    Y . Ye, Z. Huang, Y . Xiao, E. Chern, S. Xia, and P. Liu, “Limo: Less is more for reasoning,”arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  42. [50]

    s1: Simple test-time scaling,

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Cand `es, and T. B. Hashimoto, “s1: Simple test-time scaling,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 20 286–20 332

    2025

  43. [51]

    Hint tuning: Less data makes better reasoners,

    S. Fan, M. Li, X. Ma, X. Huang, Z. Chen, B. Qin, L. Zhang, S. Shang, and W. Chen, “Hint tuning: Less data makes better reasoners,”arXiv preprint arXiv:2605.08665, 2026

    2026 arXiv

  44. [52]

    Redagent: an autonomous agent for context-aware red teaming of llm jailbreaks,

    H. Xu, W. Zhang, Z. Wang, F. Xiao, R. Zheng, Z. Ba, and K. Ren, “Redagent: an autonomous agent for context-aware red teaming of llm jailbreaks,”IEEE Transactions on Dependable and Secure Computing, 2026

    2026

  45. [53]

    A multi- stage adversarial framework for compact and effective jailbreaking of large language models,

    W. Liu, Z. Liu, C. Xu, C. Li, J. Guo, J. Ma, and F. Huang, “A multi- stage adversarial framework for compact and effective jailbreaking of large language models,”IEEE Transactions on Dependable and Secure Computing, 2026

    2026

  46. [54]

    Do we really need curated malicious data for safety alignment in multi-modal large language models?

    Y . Wang, J. Guan, J. Liang, and R. He, “Do we really need curated malicious data for safety alignment in multi-modal large language models?” in2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  47. [55]

    Why llm safety guardrails collapse after fine-tuning: A similarity analysis between alignment and fine-tuning datasets,

    L. Hsiung, T. Pang, Y .-C. Tang, L. Song, T.-Y . Ho, P.-Y . Chen, and Y . Yang, “Why llm safety guardrails collapse after fine-tuning: A similarity analysis between alignment and fine-tuning datasets,” inProceedings of the 64th Annual Meeting of the Association for Computation...

    2026 arXiv

  48. [56]

    Safechain: Safety of language models with long chain- of-thought reasoning capabilities,

    F. Jiang, Z. Xu, Y . Li, L. Niu, Z. Xiang, B. Li, B. Y . Lin, and R. Poovendran, “Safechain: Safety of language models with long chain- of-thought reasoning capabilities,” inFindings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 23 303–23 320

    2025

  49. [57]

    Star-1: Safer alignment of reasoning llms with 1k data,

    Z. Wang, H. Tu, Y . Wang, J. Wu, Y . Liu, J. Mei, B. R. Bartoldson, B. Kailkhura, and C. Xie, “Star-1: Safer alignment of reasoning llms with 1k data,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 44, 2026, pp. 37 988–37 997

    2026

  50. [58]

    Unsafechain: Enhancing reasoning model safety via hard cases,

    R. V . Tomar, P. Nakov, and Y . Wang, “Unsafechain: Enhancing reasoning model safety via hard cases,” inProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational...

    2025 doi

  51. [2025]

    Available: https://arxiv.org/abs/2412.15115

    [Online]. Available: https://arxiv.org/abs/2412.15115

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.