Pith. sign in

REVIEW 5 major objections 6 minor 29 references

Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding safety examples to instruction tuning or applying DPO after tuning raises safe responses on three harmfulness benchmarks from roughly 40% to over 90% while leaving QA accuracy essentially unchanged.

desk verdict Useful replication of safety instruction tuning, but the DPO-vs-SIT/RAFT comparison is confounded by different training data and contradicted on one benchmark by the paper's own tables. read the letter →

arxiv 2412.00074 v1 pith:S4J5WNQC submitted 2024-11-26 cs.CL

Avinash Amballa , Durga Sandeep Saluru , Gayathri Akkinapalli , Abhishek Sureddy , Akshay Kumar Sureddy This is my paper · ORCID
classification cs.CL
keywords safetyalignmentinstructiontuningdirectpreferenceoptimizationrewardrankedfine-tuningharmfulnessevaluationhelpfulnessLLaMA-2LlamaGuard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a modest amount of safety-focused training can make an instruction-tuned open-weight 7B language model safe to serve: adding a few hundred curated (unsafe instruction, safe response) examples to instruction tuning, or applying preference optimization afterward, raises the fraction of responses judged safe by a guardrail classifier from about 40% to above 90% on three harmfulness benchmarks. It further claims that Direct Preference Optimization (DPO), which trains on both chosen and rejected responses, outperforms two alternatives—Safety Instruction-Tuning (SIT) and Reward Ranked FineTuning (RAFT)—while leaving helpfulness on common QA benchmarks essentially unchanged. The practical interest is that this safety gain is achieved with QLoRA-style efficient fine-tuning on limited compute, without a full RLHF loop.

What carries the argument

The argument turns on three training recipes compared on the same base model. Safety Instruction-Tuning (SIT) just mixes a small number of (unsafe instruction, safe response) pairs into the instruction-tuning data. Reward Ranked FineTuning (RAFT) samples several responses per prompt, scores them with a reward model, and fine-tunes on the top-ranked responses; the paper shows the reward model's safety accuracy is the deciding factor, because a model that scores unsafe responses highly can misalign the model. Direct Preference Optimization (DPO) is the central mechanism: it optimizes the policy directly on preference pairs so the model is pushed toward chosen responses and away from rejected ones, which the paper identifies as the reason DPO outperforms the other two. The evaluated base model is Llama-2-7B fine-tuned on 20,000 Alpaca instructions with QLoRA.

What would settle it

Train all three methods on the same safety data—for instance, construct preference pairs from the same safety instruction-tuning set used for SIT and RAFT, and also run SIT/RAFT on safe-RLHF-style pairs—and compare Llama Guard safe rates on I-CoNA, I-MaliciousInstructions, and I-Controversial. If the DPO advantage disappears or reverses, the paper's attribution of the gain to the preference-loss algorithm is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that safety can be aligned into an instruction-tuned Llama-2-7B model cheaply and without a helpfulness tax. Adding 100 to 2,000 safety samples to the 20,000-sample Alpaca instruction set reduces Llama Guard unsafe rates substantially, with the 1,000-sample model reaching 94% safe on I-CoNA, 92% on I-MaliciousInstructions, and 97% on I-Controversial. Applied after instruction tuning, DPO on the SafeRLHF preference dataset reaches even higher safety (93%, 95%, and 100% on the same three sets) and slightly improves single-word answer accuracy from 40% to 43% while keeping BoolQ, OpenBookQA, and PIQA accuracy near or above the baseline. The authors attribute DPO's advantage to its contrastive learning signal, which moves the model toward chosen responses and away from rejected ones. RAFT, in contrast, only helps when its reward model is safety-accurate; a weak reward model can select unsafe responses and reduce safety.

Load-bearing premise

The comparison treats DPO, SIT, and RAFT as comparable alignment methods, but DPO was trained on a different preference dataset than the safety examples used for SIT and RAFT, so if the data source rather than the algorithm drives the improvement, the headline comparison is not isolated.

Editorial extensions

If this is right

  • Adding 100 safety samples to instruction tuning already raises the Llama Guard safe rate on I-CoNA from 41% to 65%, and 1,000 samples reach 94%.
  • DPO reaches 93% safe on I-CoNA, 95% on I-MaliciousInstructions, and 100% on I-Controversial, against a base of roughly 40-50%.
  • Helpfulness is largely preserved: BoolQ accuracy stays around 70%, OpenBookQA and PIQA improve by a few points, and single-word answers rise from 40% to 43% with DPO.
  • RAFT improves safety only when the reward model is safety-accurate; with a weaker reward model, safety can decline.
  • The 20k-instruction QLoRA recipe offers a low-compute path to safety alignment that does not require a separate RLHF pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because DPO was trained on SafeRLHF preference pairs while SIT and RAFT used a different safety dataset, the paper does not isolate whether DPO's advantage comes from the algorithm or from the training data; a controlled cross-dataset test would settle this.
  • The claim of 'over 90% safe' rests on Llama Guard as the primary safety metric; an independent judge or a jailbreak-style evaluation could show whether the safety margin generalizes beyond this one classifier.
  • The NLI claim-recall metric introduced in the appendix could serve as a lightweight helpfulness check in settings where API-based judges are unavailable.
  • The winrate-with-reference prompt template documents a practical way to reduce positional bias in LLM-as-a-judge evaluations, and could be reused by other low-resource alignment studies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper studies three alignment recipes for a Llama-2-7B model instruction-tuned on 20k Alpaca samples: Safety Instruction-Tuning (SIT) by adding n safety samples from Bianchi et al., RAFT with two reward models, and DPO on SafeRLHF preference data. The authors evaluate safety with Llama Guard and reward-model scores on I-CoNA, I-MaliciousInstructions, and I-Controversial, and helpfulness with BoolQ, PIQA, OpenBookQA, Alpaca test, and NLI/lexical metrics. They claim that safe responses increase from about 40% to over 90% without helpfulness loss and that DPO outperforms SIT and RAFT.

Significance. If the central claims were established, this would be a practically useful demonstration that lightweight post-hoc alignment, especially DPO, can improve the safety of an open instruction-tuned 7B model while retaining QA performance, and it would provide a useful comparison of three alignment recipes. The paper has several strengths: it reports a wide range of metrics, includes negative and abandoned evaluation results such as LLM-as-judge positional bias and Bert classifier failure on HEx-PHI, and contains an error analysis of RAFT that identifies reward-model mis-ranking. However, the DPO comparison is confounded with training data, the harmfulness test sets are extremely small, and the helpfulness evaluation has scoring ambiguities, so the current evidence is suggestive but not decisive.

major comments (5)
  1. [§7.1–§7.3; Tables 16, 23; §9] The central conclusion that DPO outperforms SIT and RAFT is confounded with training-data identity. SIT and RAFT are trained on 20k Alpaca instructions plus 2.5k Bianchi safety samples (§7.1, §7.2), while DPO is trained on 8,066 preference pairs from the SafeRLHF/BeaverTails corpus (§7.3, Table 2). The methods differ simultaneously in dataset size, prompt distribution, and response style, so Table 23's high DPO safe rates cannot be attributed to the DPO objective. In addition, Table 16 shows SIT-1000 reaching 94% on I-CoNA versus DPO's 93%, so the paper's own numbers do not uniformly support the superiority claim; Section 9's mechanism explanation, 'learning from chosen and rejected responses,' is therefore unsupported. The comparison should be rerun with matched training data, or the conclusion should be restricted to 'DPO trained on preference data achieved the highest rates.'
  2. [§4.2; Tables 16, 18, 23] The baseline safe percentages are inconsistent across tables. Table 16 reports Base as 0.41/0.41/0.45 on I-CoNA/I-MaliciousInstructions/I-Controversial, Table 18 reports the same base as 39.9%/50%/42.5%, and Table 23 reports 39.9%/42.5%/50%. Since the abstract's 'from 40% to over 90%' hinges on the baseline, this discrepancy must be resolved. Furthermore, the three harmfulness sets contain only 178, 100, and 40 examples, and every Llama Guard result is a single-run point estimate; a difference such as 92% versus 94% is only two or three examples. Please provide exact counts, bootstrap confidence intervals or multiple runs, and a validation of Llama Guard agreement on these particular sets.
  3. [§4.1, §5.1; Table 17] The helpfulness evaluation does not yet support the 'no compromise' claim. The BoolQ instruction template says 'Answer the following question (True/False)', but BoolQ answers are 'yes/no'; the paper does not explain how the exact-match scoring maps these, despite using max_tokens=1. The Alpaca-test lexical scores (BLEU ≈ 0.038, ROUGE-L ≈ 0.22, BERTScore ≈ 0.19) are low enough that their differences are hard to interpret, and Tables 17, 20, 21, and 24 report QA accuracies without variance or significance testing. Please clarify the scoring protocol and state the uncertainty before using these numbers to claim helpfulness is retained.
  4. [§7.2.3; Tables 18, 19] The RAFT results are close to baseline and are overinterpreted. In Table 19 (B=100, five iterations, Deberta reward model), I-CoNA safety is 38.2%, 39.32%, 40.01%, and 39.9% over iterations against a 39.9% baseline, so there is no improvement on that set; only I-MaliciousInstructions moves from 50% to 54%. In Table 18 (B=500, one iteration), the Deberta-filtered model ties the baseline on I-CoNA at 39.9%. The text's statement that safety accuracies 'have increased on some safety datasets' is accurate only for a subset and should be quantified and hedged accordingly.
  5. [§5.2; §7.2.2] The OpenAssistant Deberta model is used both as a RAFT training filter (§7.2.2) and as a harmfulness evaluation metric (§5.2). I do not believe this invalidates the Llama Guard tables, because those classifications come from a separate model, but the paper should state that reward-model-based safety scores for RAFT-Deberta are not independent of the training signal. Any conclusion about RAFT-Deberta should be based on Llama Guard or another held-out judge, and the paper should make explicit which table uses which metric.
minor comments (6)
  1. [Throughout] There are numerous typos, including 'choosen' (§3.2), 'harmfuless' (Table 4), 'Futher' (§5.1), 'contrained' (Appendix §10), and 'volumn' (Figure 11).
  2. [Table 2] The cell 'Median # words in prompt ll' appears to be a corrupted value and should be replaced by a number.
  3. [§7] References to figures are inconsistent: the text says 'graph 3' and '11' where it should say 'Figure 3' and 'Figure 11', and several table references in Section 7 do not match the appendix numbering.
  4. [§7.3] The 'safe RLHF data' is not cited; the paper should cite the SafeRLHF/BeaverTails source and clarify the relationship to the preference dataset described in Section 3.2.
  5. [§9] The phrasing 'The results of DPO are pretty impressive' is informal and should be replaced with a quantitative, hedged summary.
  6. [§4.1] The manually curated helpfulness set is described as verified via 'self-reflection, online searches, and cross-checks with GPT-3.5'; please provide the actual prompts or a link and state any agreement measure, since this set is used for single-word answer accuracy in Tables 20, 21, and 24.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central safety and helpfulness results are measured by external benchmarks (Llama Guard, BoolQ, PIQA, OpenBookQA) independent of the training procedures.

full rationale

The paper's headline claim—safe responses increasing from 40% to over 90%—is supported by Llama Guard classifications (Section 5.2, Tables 16 and 23), an external safety classifier not used as a training signal for SIT, RAFT, or DPO. The DPO model is trained on SafeRLHF/BeaverTails preference data and evaluated on Bianchi et al.'s I-CoNA, I-MaliciousInstructions, and I-Controversial sets; the SIT models are trained on Alpaca plus Bianchi safety samples and evaluated on the same external harmfulness sets plus independent QA benchmarks. No fitted parameter is renamed as a prediction, and no equation or construction forces an evaluation result from the training data. The only overlap worth noting is that OpenAssistant Deberta appears both as a RAFT reward model (Section 7.2) and as one reported harmfulness metric (Section 5.2), but the RAFT safety tables (18 and 19) report Llama Guard percentages, and Deberta scores are reported for SIT models that were not trained with Deberta. Thus the central safety claims do not reduce to the reward model's own scores. There are no load-bearing self-citations, and the cited algorithms originate from external groups with no author overlap. The comparison between DPO and SIT/RAFT is confounded by different training data sources and is not directly tabulated head-to-head, but that is a validity concern, not circularity, and does not raise the circularity score.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on evaluation assumptions rather than derivations. No new entities are introduced. The only hand-chosen quantity that affects the headline number is the SIT safety sample count, selected from a sweep rather than pre-specified. Training hyperparameters such as learning rate and LoRA rank are reported but not treated as fitted constants.

free parameters (1)
  • SIT safety-sample count n = 1000 (of swept values 100, 300, 500, 1000, 1500, 2000)
    The paper sweeps n and reports results for each; the 90 percent plus headline for SIT corresponds to the n=1000 point, a post-hoc selection from the evaluation sweep rather than a pre-specified choice.
assumptions (3)
  • domain assumption Llama Guard's safe/unsafe verdict is treated as ground truth for harmfulness.
    Section 5.2 introduces Llama Guard as the primary safety metric and Tables 16, 18, 19, and 23 use it for the headline percentages.
  • domain assumption The three harmfulness datasets are representative of unsafe instruction distributions.
    Section 4.2 describes I-CoNA (178), I-MaliciousInstructions (100), and I-Controversial (40) as sufficient to evaluate safety, but their small sizes and topic coverage are not justified.
  • domain assumption SafeRLHF filtered preference pairs provide valid chosen/rejected safety signals.
    Section 3.2 says the data is filtered to cases where the chosen response is safe and better, but the filtering rule and quality checks are not specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness." pith.science (2026). https://pith.science/paper/S4J5WNQC

@misc{pith2026241200074,
  author       = {Pith},
  title        = {Pith review of: Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S4J5WNQC}},
  note         = {Machine review of arXiv:2412.00074}
}
abstract

Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning and text generation. However, these models can inadvertently generate unsafe or biased responses when prompted with problematic inputs, raising significant ethical and practical concerns for real-world deployment. This research addresses the critical challenge of developing language models that generate both helpful and harmless content, navigating the delicate balance between model performance and safety. We demonstrate that incorporating safety-related instructions during the instruction-tuning of pre-trained models significantly reduces toxic responses to unsafe prompts without compromising performance on helpfulness datasets. We found Direct Preference Optimization (DPO) to be particularly effective, outperforming both SIT and RAFT by leveraging both chosen and rejected responses for learning. Our approach increased safe responses from 40$\%$ to over 90$\%$ across various harmfulness benchmarks. In addition, we discuss a rigorous evaluation framework encompassing specialized metrics and diverse datasets for safety and helpfulness tasks ensuring a comprehensive assessment of the model's capabilities.

Figures

Figures reproduced from arXiv: 2412.00074 by the authors.

Figure 1
Figure 1. Instruction-Template In absence of input, we provide only Instruction and Response. Refer to the example shown in fig￾ure 2 Statistics Dataset Alpaca Data Safety Data Size 20k 2.5k Median # words in instructions 10 9 Max # words in instructions 70 25 Num. of instructions with empty 12.5k 2.5k Median # words in output 30 53 Max # words in output 609 256 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Instruction-Template with no Input 3.2 Preference Dataset For algorithms like DPO, we required preference dataset i.e., chosen and rejected response for a given instruction. We leverage the ”Beaver-Tails” datasets collection, which was introduced by (Ji et al., 2023). In our paper, as we aim to align to￾wards safety + helpfulness, we filter out the data where the choosen response is safe and better. The final datase… view at source ↗
Figure 3
Figure 3. Basemodel training (Left). Safety Instruction tuning training (Right) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: SIT Evaluation Results - Llama guard safe % (Left), Helpfulness QA Tasks + Single-Word Answers [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: RAFT - Llama Guard Safe % - Different RM (left) and Different Iterations (right) [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: RAFT - Accuracy on helpfulness datasets - Different RM (left) and Different Iterations (right) [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: DPO eval loss vs num. of steps [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: DPO - Llama Guard Safe % (Left), Helpfulness performance (Right) [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Win Rate prompt template #1 Data Base as A Base as B Base wins 50 28 2000 wins 27 50 Tie 23 22 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Win Rate prompt template #2 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: NLI prompt claims [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: NLI sample response scores 10.4 Safety Instruction Tuning Results [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 22 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bianchi, F., Suzgun, M., Attanasio, G., R \"o ttger, P., Jurafsky, D., Hashimoto, T., and Zou, J. (2023). Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. ArXiv

    work page 2023

  4. [4]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. (2019). Piqa: Reasoning about physical commonsense in natural language. ArXiv

    work page 2019

  5. [5]

    Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324--345

    1952

  6. [6]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. (2022). Scaling instruction-finetuned language models. ArXiv

    work page 2022

  7. [7]

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. (2019). Boolq: Exploring the surprising difficulty of natural yes/no questions. ArXiv

    work page 2019

  8. [8]

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). Qlora: Efficient finetuning of quantized llms. ArXiv

    work page 2023

Show all 29 references
  1. [9]

    Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. (2023). Raft: Reward ranked finetuning for generative foundation model alignment

    2023

  2. [10]

    Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022). Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ArXiv preprint

    2022

  3. [11]

    Gao, T., Yen, H., Yu, J., and Chen, D. (2023). Enabling large language models to generate text with citations

    2023

  4. [12]

    Ge, S., Zhou, C., Hou, R., Khabsa, M., Wang, Y.-C., Wang, Q., Han, J., and Mao, Y. (2023). Mart: Improving llm safety with multi-round automatic red-teaming

    2023

  5. [13]

    Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., and Khabsa, M. (2023). Llama guard: Llm-based input-output safeguard for human-ai conversations

    2023

  6. [14]

    Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Zhang, C., Sun, R., Wang, Y., and Yang, Y. (2023). Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    2023

  7. [15]

    Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N

    Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. (2024). Rewardbench: Evaluating reward models for language modeling

    2024

  8. [16]

    Lin, C.-Y. (2004). ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  9. [17]

    Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., Dong, H., Pi, R., Zhao, H., Jiang, N., Ji, H., Yao, Y., and Zhang, T. (2023). Mitigating the alignment tax of rlhf. ArXiv

    2023

  10. [18]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022). Training language model...

    2022

  11. [19]

    Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. (2023). Fine-tuning aligned language models compromises safety, even when users do not intend to!

    2023

  12. [20]

    D., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model

    2023

  13. [21]

    Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., and Ma, S. (2020). Codebleu: a method for automatic evaluation of code synthesis

    2020

  14. [22]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  15. [23]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....

    2023

  16. [24]

    M., Martin, L., Batra, S., Celikyilmaz, A., Wang, W

    Tuan, Y.-L., Chen, X., Smith, E. M., Martin, L., Batra, S., Celikyilmaz, A., Wang, W. Y., and Bikel, D. M. (2024). Towards safety and helpfulness balanced responses via controllable large language models

    2024

  17. [25]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. (2023a). Self-instruct: Aligning language models with self-generated instructions. In Proc. of ACL , pages 13484--13508, Toronto, Canada. Association for Computational Linguistics

    2023

  18. [26]

    N., Egert, D., Delalleau, O., Scowcroft, J

    Wang, Z., Dong, Y., Zeng, J., Adams, V., Sreedhar, M. N., Egert, D., Delalleau, O., Scowcroft, J. P., Kant, N., Swope, A., and Kuchaiev, O. (2023b). Helpsteer: Multi-attribute helpfulness dataset for steerlm

    2023

  19. [27]

    Q., and Artzi, Y

    Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2020). Bertscore: Evaluating text generation with bert

    2020

  20. [28]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena

    2023

  21. [29]

    Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J. (2023). Starling-7b: Improving llm helpfulness & harmlessness with rlaif

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.