AnswerCarefully: A Dataset for Improving the Safety of Japanese LLM Output

Hisami Suzuki; Kouta Nakayama; Satoru Katsumata; Satoshi Sekine; Takashi Kodama; Tetsuro Takahashi

REVIEW 3 major objections 5 minor 1 cited by

AnswerCarefully: A Dataset for Improving the Safety of Japanese LLM Output

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AnswerCarefully, a human-written Japanese safety dataset of 1,800 question-answer pairs, reports that adding it to supervised fine-tuning cuts a Japanese LLM's harmful-answer rate from 44.5% to 15.3% without lowering general usefulness.

desk verdict A genuinely useful Japanese safety dataset with reference answers and a solid human-evaluated benchmark; the SFT safety gains are plausible but rest on a judge that sees the reference answers, so treat that table as suggestive. read the letter →

arxiv 2506.02372 v1 pith:TZ74DFRQ submitted 2025-06-03 cs.CL

Hisami Suzuki , Satoru Katsumata , Takashi Kodama , Tetsuro Takahashi , Kouta Nakayama , Satoshi Sekine This is my paper

classification cs.CL

keywords JapaneseLLMsafetyalignmentinstructiontuningreferenceanswersLLM-as-a-judgebenchmarkculturaladaptationsupervisedfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AnswerCarefully is a Japanese-language safety dataset: 1,800 questions that need careful answers, each paired with a human-written reference answer, and covering a broad taxonomy of risk categories. The paper's central claim is that adding these pairs to supervised fine-tuning of a Japanese LLM makes its outputs safer—the harmful-answer rate falls from 44.5% to 15.3% with the v2 data duplicated sixteen times—without lowering usefulness on general questions. The reference answers do double duty: they are training targets, and when shown to an automatic judge they raise judge-human agreement on safety ratings almost to the level of agreement between two human raters. The paper also uses the same questions as a benchmark and finds large safety differences across twelve Japanese LLMs. If the claim holds, the dataset is a reusable safety-alignment resource and benchmark for Japanese, with annotations aimed at helping other languages adapt it.

What carries the argument

The load-bearing object is the paired reference answer. Each of the 1,800 questions comes with a human-written answer designed under the paper's stated values: say nothing harmful or unsafe, oppose illegal activity and discrimination, protect human rights and democracy, present multiple viewpoints on sensitive topics, avoid over-anthropomorphism, and mark the output as coming from an AI assistant. These references do two jobs: as training targets in supervised fine-tuning they teach the model how to answer carefully, and as context for an LLM-as-a-judge they raise the judge's agreement with human raters from 0.45-0.55 to 0.57-0.68 (Spearman), close to human-human agreement. The risk-category taxonomy comes from the English safeguard-evaluation dataset the paper adapts, with Level-3 categories rebalanced to 56 for the Japanese data collection.

What would settle it

Run the same fine-tuned model's test responses past three blinded human raters using the paper's five-level rubric; if the human violation rate does not drop from the usefulness-only baseline, the reported safety improvement is an artifact of the reference-aware judge.

Watch

Extended reading notes

Core claim

The paper's discovery is that safety behavior in a Japanese LLM can be moved substantially by instruction data alone, using a small human-written dataset with reference answers, without sacrificing general helpfulness. In the reported experiments, fine-tuning a 13-billion-parameter Japanese LLM with the usual usefulness data alone yields a violation rate of 0.445 and an acceptable-response rate of 0.436 on the dataset's test questions; adding version 1 data once brings these to 0.357 and 0.538, and duplicating version 2 data sixteen times brings them to 0.153 and 0.719, while the general-domain usefulness score stays at roughly 3.8 in all conditions. The paper further claims that giving the automatic judge the reference answers improves its correlation with human ratings (Spearman 0.57-0.68) to the level of human-human correlation, and that using the dataset as a benchmark exposes wide variation in safety among twelve Japanese LLMs. The intended takeaway is that this dataset is an effective, openly usable alignment resource for Japanese, not just an evaluation set.

Load-bearing premise

The safety improvement is judged by an automatic judge that is shown the dataset's reference answers, and the paper assumes that judge is a faithful stand-in for human safety ratings.

Editorial extensions

If this is right

Repeating safety data during fine-tuning is an effective low-cost lever: duplicating the v2 data sixteen times lowers the violation rate from 0.274 to 0.153 and raises the acceptable-response rate from 0.595 to 0.719.
A safety dataset with reference answers can serve two purposes at once: training targets for alignment and context that makes automatic safety evaluation more reliable.
Ordinary, non-jailbreak questions are enough to separate well-aligned from poorly aligned Japanese LLMs; the twelve-model benchmark shows pronounced differences in violation and acceptable-response rates.
Because 27% of the collected questions contain culture-specific content, simply translating an English safety dataset into Japanese would not capture the same risks; locally written questions add coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the automatic judge in the fine-tuning experiment is shown the reference answers, part of the measured improvement could be stylistic mimicry of those references rather than a general safety policy; a blinded human evaluation of the fine-tuned model would test this directly.
The usefulness check rests on a single general-domain benchmark; testing on Japanese-specific instruction-following and factual-accuracy benchmarks would show whether the no-degradation result generalizes.
The paper's reference-aware judge result suggests a cheap recipe for other languages: publishing paired reference answers may itself improve the reliability of automatic safety evaluation, independent of whether the answers are used for training.
As models improve, a static safety set loses discriminative power; the planned borderline and regionally sensitive additions are a natural extension, but the same dynamic would apply to any language derived from this dataset.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: the dataset is the contribution, and it's a good one. The fine-tuning table is a demo, and its headline numbers are weaker than they look because the judge was given the same reference answers the model was trained on. Send this to review, but ask for a human or blind evaluation of the tuned models before taking the SFT claim at face value.

The new thing is AnswerCarefully: 1,800 manually written Japanese QA pairs with reference answers, covering the DNA taxonomy. That's genuinely missing from the existing Japanese datasets (JBBQ is bias-only, TruthfulQA-J and JSocialFact are factuality-only). The reference answers are a smart addition; they make the dataset usable for SFT and they demonstrably improve judge reliability, which the paper shows with human-rater correlations. The mlmc annotations in v2.2 are a practical step toward porting the data to other languages.

The human evaluation of 12 LLMs in Section 5 is the most solid part. Three annotators, adjudication for score disagreements, and the results separate the models. That's a real benchmark result and it's not contaminated by the judge issue.

Now the soft spot. Table 3's safety gains—violation rate from 0.445 to 0.153—were measured by GPT-4-as-a-judge that receives the AC reference answer for each test question. The fine-tuned models were trained on those same reference answers. The paper reports only moderate human-judge agreement (Spearman ~0.68 with reference, ~0.56 without). So it's entirely plausible that part of the measured improvement is the judge rewarding reference-like phrasing rather than a genuine reduction in harm. The paper doesn't report any human evaluation of the tuned models, so we can't tell. That doesn't make the dataset circular, but it does mean the SFT claim is not yet established at the level the abstract implies.

Minor: the test set is 336 questions, no confidence intervals on the violation rates, and the questions are deliberately non-jailbreak, so the benchmark measures everyday safety, not adversarial robustness. Those are limitations the authors mostly acknowledge.

Bottom line: this deserves serious review. The dataset and the human benchmark are valuable and will be used. The fine-tuning experiment needs additional evidence—either a blind judge or human ratings—before the safety-improvement claim is fully credible.

Referee Report

3 major / 5 minor

Summary. The paper introduces AnswerCarefully (AC), a Japanese-language safety dataset consisting of 1,800 manually written question--reference-answer pairs organized under the Do-Not-Answer three-level risk taxonomy. The authors use AC for supervised fine-tuning of LLM-jp-13B-v2.0 and report that safety, measured by GPT-4-as-a-judge, improves substantially: violation rate falls from 0.445 to 0.153 and acceptable response rate rises from 0.436 to 0.719 when ACv2 is duplicated 16 times, while MT-Bench scores remain around 3.8. They also report a human-rated benchmark evaluation of 12 Japanese LLMs on an ACv1 evaluation set, showing large variation across systems, and describe a recent update that adds English translations and adaptation annotations.

Significance. If the fine-tuning results are accepted, the paper provides a useful, culturally grounded safety resource for Japanese LLMs: the dataset is manually created rather than GPT-generated, it includes reference answers that serve both instruction tuning and automatic evaluation, and the human-rated benchmark of 12 systems is a valuable independent contribution. The dataset construction is not circular: the taxonomy comes from an external source and the questions and reference answers are authored by human annotators. However, the central claim that AC-based SFT reduces harmful outputs rests on an automatic judge that is given the same reference answers used in training, and the paper reports no human evaluation of the fine-tuned checkpoints. That confound must be addressed before the safety-improvement claim can be taken at face value.

major comments (3)

[Section 3.2 / Table 3] The safety evaluation in Section 4 uses GPT-4-as-a-judge that has access to AC reference answers, and the SFT models were trained on those same reference answers. This creates a potential confound: the judge may reward outputs that are stylistically or structurally similar to the reference answers, inflating the measured safety improvement. Table 2 shows only moderate judge--human agreement even with references (Pearson 0.667, Spearman 0.679), and no human evaluation of the Table 3 checkpoints is reported. The human-rated benchmark in Section 5 covers 12 systems but not the SFT checkpoints. To support the claim of reduced harmful answer rate, the authors should report human ratings for at least the usefulness-only baseline, ACv1 x16, and ACv2 x16 on the ACv2 test set, or equivalently provide judge-without-reference scores as a conservative complement.
[Table 3] Violation Rate and Acceptable Response Rate are reported as point estimates without confidence intervals, even though they are computed on only 336 test samples. For example, the difference between 0.218 (ACv1 x16) and 0.153 (ACv2 x16) may be within sampling noise. The authors should report binomial confidence intervals or perform significance tests for each pairwise comparison, not only the single 'p < 0.05' statement for ACv1 versus ACv2 in Section 4.
[Section 3.2] The reliability of the automatic judge is established through correlation between scores (Pearson/Spearman), but the headline metrics are dichotomized aggregates: Violation Rate (ratings 1-2) and Acceptable Response Rate (ratings 4-5). Correlation in the full rating scale does not directly validate agreement on these binary decisions. The authors should report agreement on the safety boundary (e.g., Cohen's kappa or precision/recall for violation versus non-violation) between GPT-4 and human raters, with and without reference answers, to justify the use of the judge for the specific metrics in Table 3.

minor comments (5)

[Section 4] The fine-tuning setup is described only by the list of instruction datasets; the authors should state the number of epochs, learning rate, batch size, and any regularization details, or at least refer to a reproducible configuration, since the effect sizes in Table 3 are the main empirical result.
[Table 3] Some MT-Bench entries are reported with confidence intervals and others without (e.g., ACv1 x16 shows 3.68 with no error bar). For consistency, report all MT-Bench scores with the same uncertainty format.
[Section 4] The sentence reporting a statistically significant difference 'comparing these two' should name the exact conditions compared (presumably ACv1 x16 versus ACv2 x16) and the statistical test used, such as McNemar's test or a bootstrap test on violation rate.
[Section 5] The human benchmark uses the ACv1 evaluation set (183 questions), while the fine-tuning evaluation uses the ACv2 test set (336 questions). The authors should clarify why the benchmark was not updated to ACv2 and whether the conclusions drawn from Figure 2 are intended to generalize to the current dataset version.
[Abstract / References] There are several typographical errors: an isolated 'fc' appears after the abstract, 'concludethe paperin' is missing spaces, reference [8] has 'Sekine. Satoshi' in the author list, and reference [6] contains a stray ':' after 'LLM-jp'. These should be corrected in a revision.

Circularity Check

1 steps flagged · score 4.0 of 10

Table 3 safety gains are partly benchmarked against the dataset's own reference answers: the same AC references are used both to fine-tune the model and to cue the GPT-4 judge.

fitted input called prediction [Section 3.2 (Automatic Evaluation); Section 4 (Safety Fine-Tuning), Table 3]
"Interestingly, the correlation coefficients improve to the range from 0.57 to 0.68 when LLM-as-a-judge can use reference answers... Based on these results, we consider that LLM-as-a-judge is sufficiently reliable in evaluations using AC, and in Section 4 below, we will investigate the effectiveness of safety fine-tuning using automatic evaluation. For safety evaluation, we used the ACv2 evaluation set (336 samples) and performed automatic evaluation with GPT-4-as-a-judge as described in the previous section."

The SFT model is trained on AC dev reference answers (Section 4), then evaluated on ACv2 test with GPT-4-as-a-judge 'as described in the previous section,' i.e., a judge that is given the AC reference answers. The paper explicitly says these references improve automatic evaluation accuracy. The Table 3 improvement (Violation Rate 0.445→0.153; Acceptable Response Rate 0.436→0.719) is therefore partly measuring how closely the tuned model reproduces the AC reference style the judge is cued with, not an independently measured safety gain. Table 2 validates this judge only moderately against humans (Spearman 0.679 with reference; 0.557 without), and no human evaluation of the fine-tuned models is reported.

full rationale

The dataset construction itself is not circular: AC questions and reference answers are manually created by annotators, the risk taxonomy is taken from the external Do-Not-Answer dataset, and the Section 5 benchmark uses human raters on 12 LLMs, providing independent content. The main circularity risk is confined to Section 4: the fine-tuned model is trained on AC reference answers and then scored by a GPT-4 judge that receives those same reference answers; the paper even states that the references improve the judge's human correlation. With only moderate human-judge agreement and no human evaluation of the tuned checkpoints, the reported drop in violation rate is partly an artifact of the model learning to imitate the dataset's reference style rather than a proven reduction in harmfulness. This is a partial circularity in the evaluation instrument, not a wholesale identity between input and output. Self-citations ([4], [8], [11]) are used for implementation details and detailed criteria, not as load-bearing external theorems, so they do not raise the score further. The appropriate score is 4.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted numerical parameters appear in the paper; the main load-bearing assumptions are about the validity of the risk taxonomy, the reference answers, and the automatic judge. These are listed as domain assumptions.

assumptions (3)

domain assumption The Do-Not-Answer risk taxonomy is comprehensive enough for Japanese LLM safety.
Section 2 adopts DNA categories with Level-3 reduced from 61 to 56 to balance sample counts. No independent validation of coverage for Japanese cultural contexts is provided, though 27 percent of samples contained culture-specific content.
domain assumption Reference answers written by annotators following G7 Hiroshima AI Process values represent appropriate safe responses.
Section 2 states the answers follow G7 values and multiple viewpoints, but also says 'It is not our goal to offer the right response'. The quality of these answers is load-bearing for SFT.
domain assumption GPT-4-as-a-judge with AC reference answers is a reliable proxy for human safety ratings.
Section 3.2 and Table 2 show correlations with human raters that are moderate (0.57-0.68 with reference answers). All Table 3 SFT safety numbers rely on this judge.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AnswerCarefully: A Dataset for Improving the Safety of Japanese LLM Output." pith.science (2026). https://pith.science/paper/TZ74DFRQ

@misc{pith2026250602372,
  author       = {Pith},
  title        = {Pith review of: AnswerCarefully: A Dataset for Improving the Safety of Japanese LLM Output},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TZ74DFRQ}},
  note         = {Machine review of arXiv:2506.02372}
}

read the original abstract

In this paper we present AnswerCarefully, a dataset for promoting the safety and appropriateness of Japanese LLM outputs. The dataset consists of 1,800 pairs of questions and reference answers, where the questions require special attention in answering. It covers a wide range of risk categories established in prior English-language datasets, but the data samples are original in that they are manually created to reflect the socio-cultural context of LLM usage in Japan. We show that using this dataset for instruction to fine-tune a Japanese LLM led to improved output safety without compromising the utility of general responses. We also report the results of a safety evaluation of 12 Japanese LLMs using this dataset as a benchmark. Finally, we describe the latest update on the dataset which provides English translations and annotations of the questions, aimed at facilitating the derivation of similar datasets in different languages and regions.

Figures

Figures reproduced from arXiv: 2506.02372 by the authors.

**Figure 2.** Results of human evaluation of 12 LLMs using AC [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Sample mlmc annotation in ACv2.2 or region) [15]. Furthermore, we have recently added what we call multi-language-multi-culture (mlmc) annotations to ACv2 as version 2.2, in line with the efforts of AI Safety Institute International Network 18 promoting the creation of safety data for multiple languages and regions. A sample annotation is shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Additional samples from AnswerCarefully (translated into English) [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
cs.CL 2025-06 conditional novelty 3.0 of 10

A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.

Reference graph

Works this paper leans on

18 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

arXiv 2022
[2]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

work page 2022
[3]

Llama Guard: LLM-based input-output safeguard for human-AI conversations, 2023

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based input-output safeguard for human-AI conversations, 2023. URL https: //arxiv.org/abs/2312.06674

arXiv 2023
[4]

Investigating tuning methods for achieving usefulness and safety for Japanese large language models (in Japanese)

Satoru Katsumata, Takashi Kodama, and Yusuke Miyao. Investigating tuning methods for achieving usefulness and safety for Japanese large language models (in Japanese). In The 31st Annual Conference of the Association for Natural Language Processing, 2025

work page 2025
[5]

Japanese safety boundary test for large language models (in Japanese)

Tomoya Kurosawa, Junya Takayama, Koki Wataoka, Koga Kobayashi, Masayuki Asahara, and Sae Nishiuchi. Japanese safety boundary test for large language models (in Japanese). In The 31st Annual Conference of the Association for Natural Language Processing, 2025

work page 2025
[6]

LLM-jp: A Cross- organizational Project for the Research and Development of Fully Open Japanese LLMs, 2024

LLM-jp, :, Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiraoka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, Hiroki Kanezashi, Hiroshi Kataoka, Satoru Katsumata, Daisuke Kawahara, Seiy...

arXiv 2024
[7]

Construction of the Japanese TruthfulQA Dataset (in Japanese)

Yusuke Nakamura and Daisuke Kawahara. Construction of the Japanese TruthfulQA Dataset (in Japanese). In The 30th Annual Conference of the Association for Natural Language Processing, 2024

work page 2024
[8]

Kouta Nakayama, Takashi Kodama, Hisami Suzuki, Yusuke Miyao, and Sekine. Satoshi. llm-jp- judge: Japanese LLM-as-a-Judge evaluation tool (in Japanese). In The 31st Annual Conference of the Association for Natural Language Processing, 2025

work page 2025

Show all 18 references

[9]

JSocialFact: a misinfor- mation dataset from social media for benchmarking LLM safety

Tomoka Nakazato, Masaki Onishi, Hisami Suzuki, and Yuya Shibuya. JSocialFact: a misinfor- mation dataset from social media for benchmarking LLM safety. In 2024 IEEE International Conference on Big Data (BigData), pages 3017–3025, 2024. doi: 10.1109/BigData62323.2024. 10825770

2024
[10]

GPT-4 technical report, 2024

OpenAI. GPT-4 technical report, 2024

2024
[11]

Large-scale human evaluation of LLM safety (in Japanese)

Tetsuro Takahashi, Hisami Suzuki, and Satoshi Sekine. Large-scale human evaluation of LLM safety (in Japanese). In The 31st Annual Conference of the Association for Natural Language Processing, 2025

2025
[12]

Gemini: A family of highly capable multimodal models, 2024

Gemini Team. Gemini: A family of highly capable multimodal models, 2024

2024
[13]

Llama 2: Open foundation and fine-tuned chat models, 2023

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

2023
[14]

Do-Not-Answer: Evaluating safeguards in LLMs

Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-Not-Answer: Evaluating safeguards in LLMs. In Yvette Graham and Matthew Purver, editors, Findings of the Association for Computational Linguistics: EACL 2024, pages 896–911, St. Julian’s, Malta, March 20...

2024
[15]

A Chinese Dataset for Evaluating the Safeguards in Large Language Models

Yuxia Wang, Zenan Zhai, Haonan Li, Xudong Han, Shom Lin, Zhenxuan Zhang, Angela Zhao, Preslav Nakov, and Timothy Baldwin. A Chinese Dataset for Evaluating the Safeguards in Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Associa...

2024 doi
[16]

Analyzing social biases in Japanese large language models

Hitomi Yanaka, Namgi Han, Ryoma Kumon, Jie Lu, Masashi Takeshita, Ryo Sekizawa, Taisei Kato, and Hiromi Arai. Analyzing social biases in Japanese large language models. arxiv:2406.02050, 2024

2024 arXiv
[17]

Xing, Hao Zhang, Joseph E

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. InAdvances in Neural Information Processi...

2023
[184]

URL https://aclanthology.org/2024.findings-acl.184/

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

arXiv 2022

[2] [2]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

work page 2022

[3] [3]

Llama Guard: LLM-based input-output safeguard for human-AI conversations, 2023

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based input-output safeguard for human-AI conversations, 2023. URL https: //arxiv.org/abs/2312.06674

arXiv 2023

[4] [4]

Investigating tuning methods for achieving usefulness and safety for Japanese large language models (in Japanese)

Satoru Katsumata, Takashi Kodama, and Yusuke Miyao. Investigating tuning methods for achieving usefulness and safety for Japanese large language models (in Japanese). In The 31st Annual Conference of the Association for Natural Language Processing, 2025

work page 2025

[5] [5]

Japanese safety boundary test for large language models (in Japanese)

Tomoya Kurosawa, Junya Takayama, Koki Wataoka, Koga Kobayashi, Masayuki Asahara, and Sae Nishiuchi. Japanese safety boundary test for large language models (in Japanese). In The 31st Annual Conference of the Association for Natural Language Processing, 2025

work page 2025

[6] [6]

LLM-jp: A Cross- organizational Project for the Research and Development of Fully Open Japanese LLMs, 2024

LLM-jp, :, Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiraoka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, Hiroki Kanezashi, Hiroshi Kataoka, Satoru Katsumata, Daisuke Kawahara, Seiy...

arXiv 2024

[7] [7]

Construction of the Japanese TruthfulQA Dataset (in Japanese)

Yusuke Nakamura and Daisuke Kawahara. Construction of the Japanese TruthfulQA Dataset (in Japanese). In The 30th Annual Conference of the Association for Natural Language Processing, 2024

work page 2024

[8] [8]

Kouta Nakayama, Takashi Kodama, Hisami Suzuki, Yusuke Miyao, and Sekine. Satoshi. llm-jp- judge: Japanese LLM-as-a-Judge evaluation tool (in Japanese). In The 31st Annual Conference of the Association for Natural Language Processing, 2025

work page 2025