Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales

Ayuto Tsutsumi; Yuu Jinnai

REVIEW 4 major objections 7 minor 21 references

Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new 809-question benchmark shows that LLMs continually pretrained on Japanese documents carry far more knowledge of Japanese yokai folktales than English-centric models.

desk verdict A solid, carefully-built benchmark for Japanese folktale knowledge; the headline result is plausible, but the GPT-4o judge is under-validated for the fine-grained ranking. read the letter →

arxiv 2506.03619 v1 pith:SFLOQNUT submitted 2025-06-04 cs.CL

Ayuto Tsutsumi , Yuu Jinnai This is my paper

classification cs.CL

keywords yokaiJapanesefolktalesculturalawarenessofLLMscontinualpretrainingbenchmarkdatasetmultiple-choiceevaluationLLMNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

YokaiEval is a new benchmark of 809 multiple-choice questions about yokai, the supernatural creatures of Japanese folktales, built from Wikipedia articles and manually verified against folklore scholarship. The paper uses it to test how much cultural knowledge 31 large language models carry, and finds a clear pattern: models trained with Japanese language resources beat English-centric and Chinese-centric models, and the best performers are Llama-3-family models that were continually pretrained on Japanese documents. The authors argue that this identifies continual pretraining as the decisive step for acquiring regional folklore knowledge, since supervised fine-tuning and preference learning made little difference in their comparisons. If the finding holds, the practical recipe for culturally aware LLMs is to start from a capable base model and continue pretraining on the target community's own texts, then verify with a benchmark like YokaiEval rather than assuming general fluency implies cultural knowledge.

What carries the argument

The central object is YokaiEval, a dataset of 809 Japanese multiple-choice questions about yokai—supernatural creatures from Japanese folktales—each with four options and one answer linked to a reference from folklore scholarship. The construction pipeline uses GPT-4o to generate questions from 1,054 Wikipedia articles in the Japanese list of yokai, GPT-4o mini to filter incoherent items, and manual review to verify every entry against named authorities such as Kunio Yanagita and Shigeru Mizuki. The argument's analytical machinery is the three-stage training taxonomy—continual pretraining, supervised fine-tuning, and preference learning—used to compare publicly available model families plus the authors' own DPO-trained models, isolating which stage contributes yokai knowledge. A secondary mechanism is the GPT-4o judge that converts free-form answers into true/false scores despite the wide orthographic variation in yokai names.

What would settle it

Run YokaiEval scoring with an independent judge (for example, an open-weight model or normalized exact match) on the same model outputs; if the ranking of Japanese-centric over English-centric models collapses, the paper's central comparison is an artifact of the GPT-4o judge. Separately, continually pretrain a single base model on Japanese Wikipedia for increasing token budgets; if YokaiEval accuracy does not rise with the Japanese corpus budget, the claim that continual pretraining is the key stage would be refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an LLM's knowledge of Japanese folktale creatures tracks the language resources it was trained on more than its overall Japanese ability. On YokaiEval, Japanese-centric models score higher than English- or Chinese-centric models of comparable JMT-Bench ability, with Llama-3-based models continually pretrained on Japanese leading the ranking (llama-3.1-70b-japanese-instruct at 0.635 and llama-3.1-swallow-70b-instruct-v0.1 at 0.615, against 0.496 for the base Llama-3.1-8B-instruct). Using available model versions before and after supervised fine-tuning, and its own DPO runs, the paper argues that post-training steps add little yokai knowledge, so continual pretraining is the key stage. A perplexity comparison of original versus paraphrased Wikipedia articles showed no clear sign of data leakage, which the authors take as support for interpreting the scores as genuine folklore knowledge rather than memorized benchmark items.

Load-bearing premise

Every comparison among the 31 models assumes that GPT-4o's true/false judgements about whether a response picked the correct answer are reliable, but only a small manual check on two models was reported.

Editorial extensions

If this is right

Teams building culturally aware LLMs should treat continual pretraining on community-language documents as the primary investment, since it is the stage that measurably adds folktale knowledge.
Standard supervised fine-tuning and preference-learning datasets do not automatically transfer cultural knowledge, so improvements on general Japanese benchmarks should not be read as improvements in cultural awareness.
YokaiEval is a complement to general Japanese benchmarks like JMT-Bench, useful precisely because high general Japanese ability does not guarantee high yokai knowledge.
The benchmark construction pipeline can be repeated for folktales of other communities, provided each source is manually checked against domain experts, as the appendix argues for single-community analysis before cross-cultural comparison.
The lack of a clear memorization signal in the perplexity check supports treating YokaiEval scores as evidence about pretraining distributions, rather than about test-set contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A controlled experiment not run in the paper would settle the causal claim: continually pretrain the same base model on increasing amounts of Japanese folklore text and check whether YokaiEval accuracy rises with the corpus budget, since the paper's evidence is observational.
If an independent open-weight judge replaced GPT-4o in scoring, the ranking could shift for models that express answers in unusual phrasings, so the benchmark's portability depends on the judge being reproducible.
The result suggests a broader principle: cultural knowledge that lives mainly in non-English documents can be acquired through continued pretraining on those documents, which makes digitizing community archives a higher-leverage activity than further alignment tuning.
It would be informative to test whether retrieval-augmented generation from a curated yokai corpus can close the gap for English-centric models; the paper mentions RAG only as unexplored future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Read the yokai paper. Worth knowing: it's a solid, honest benchmark paper, not a big conceptual leap. The new thing is YokaiEval, 809 manually verified multiple-choice questions about Japanese yokai, plus a first pass at ranking 31 models on it. The construction is careful—questions generated from Wikipedia via GPT-4o, filtered with GPT-4o-mini, then all remaining QAs manually checked with reference verification, excluding anything that lacks expert-backed references. That is real work, and the dataset likely has stand-alone value.

The main descriptive result—Japanese-centric models score higher than English-centric ones—is visible in the raw numbers and is plausible even before any statistical machinery. The continual-pretraining-on-Llama-3 finding is new and interesting, though observational, as they concede. The data-leakage check via perplexity on original vs paraphrased articles is a nice touch, and the limitations section is unusually candid, including the tokenizer confound and the observational nature of the training-stage comparisons.

Soft spots: the evaluation judge is GPT-4o, which also generated the questions. They validate it only with a lexical parser covering ~80% of outputs and a manual check on two models. The remaining 20% is exactly where the judge's discretion is largest—hedged, paraphrased, or multi-format answers—and no per-model error rates are reported. If GPT-4o is systematically lenient toward phrasing common in Japanese instruct models, the fine-grained ranking, especially the llama-3-continual-pretraining claim, could shift. I don't think this sinks the central claim; the gap between Japanese-centric and English-centric models is large and coherent across many pairs. But it is a genuine limitation that needs addressing—for example, judge a hard subset with human annotations, report a confusion matrix, or use a different judge model on a sample.

The observational ablation of SFT/PL is fine but should not be over-read. Also, no confidence intervals anywhere; minor for a benchmark paper, but worth a sentence.

Who this is for: people building cultural-knowledge benchmarks and the Japanese NLP community. It won't reshape the field, but it's a legitimate extension with careful data work. Recommendation: send it to peer review—desk rejection would be wrong. A good referee can push for stronger judge validation and uncertainty estimates; the benchmark itself deserves to be published.

Referee Report

4 major / 7 minor

Summary. The paper introduces YokaiEval, a benchmark of 809 multiple-choice questions about Japanese yokai, constructed from Japanese Wikipedia articles with GPT-4o-generated questions, GPT-4omini-based filtering, and manual verification against expert references. The authors evaluate 31 publicly available LLMs and report that Japanese-centric models, especially those continually pretrained from Llama-3, achieve higher YokaiEval accuracy than English-centric models, and that continual pretraining is the key training stage for acquiring this cultural knowledge. The paper also includes an observational SFT/DPO ablation and a data-leakage analysis based on perplexity comparisons.

Significance. If the judge-validity concern is resolved, YokaiEval is a valuable, carefully curated resource: the dataset is grounded in human-verified references, the code and data are released, and the cultural domain is underserved by existing benchmarks. The suggestive finding that continued pretraining on Japanese from a strong English-centric base yields high yokai knowledge is actionable and of interest to the cultural-NLP community. However, the headline claim currently outruns the evidence because the GPT-4o judge is not validated at a level commensurate with its role, and the abstract states a stronger comparison than the reported numbers support.

major comments (4)

[Section 5.1, Appendix F, footnote 2] The headline comparison rests entirely on GPT-4o as the correctness judge, yet the validation of this judge is not commensurate with its role. Footnote 2 reports a lexical parser that extracts answers from only about 80% of outputs, and manual review of only two models; the 20% of unparseable outputs are exactly the paraphrased, hedged, or multi-format responses where judge discretion is largest. Because GPT-4o also generated the dataset in Section 4.1, a systematic preference for outputs phrased like its own generations, or for the hedging style common in Japanese instruct models, would put Japanese-centric and English-centric models on different scales. Please add per-model judge agreement on a human-annotated sample stratified by model family and by parseability, report a judge confusion matrix, and test whether judge accuracy differs between Japanese-centric and English-centric models. The current 'As far as we are aware, there were no errors' statement should be replaced with these quantitative results.
[Abstract, Section 5.2, Table 8] The abstract's claim that 'models trained with Japanese language resources achieve higher accuracy than English-centric models' is not supported by the paper's own numbers: in Table 8, the English-centric meta-llama-3.3-70b-instruct scores 0.641 on YokaiEval, higher than the best Japanese-centric model (llama-3.1-70b-japanese-instruct-2407, 0.635). The supported finding is the weaker, conditional one stated in Section 5.2: Japanese-centric models tend to score higher than English-centric models of similar JMT-Bench capability, and Llama-3-based continual-pretraining models do especially well. Please revise the abstract and framing accordingly, or present a matched comparison (e.g., regression controlling for JMT-Bench) that quantifies the group difference.
[Section 5.2, Tables 3 and 4] The claim that continual pretraining is the key step is built on an observational comparison and on ablation results with very small differences: Table 3 has three data points spanning 0.496-0.538 with adjacent scores differing by 0.001-0.028, and Table 4 has three models with inconsistent DPO effects. No confidence intervals, bootstrap estimates, or significance tests are reported anywhere, so it is unclear which of the reported differences (e.g., Swallow v0.2 0.510 vs v0.1 0.511) are real. Please provide per-model bootstrap confidence intervals for YokaiEval accuracy, and, if possible, a permutation or regression-based test for the Japanese-vs-English group contrast. The SFT/DPO sections should be explicitly labeled as exploratory given the small sample.
[Appendix G, Section 5.2] The JMT-Bench matching that underlies the 'similar performance' comparison depends on a linear regression (Eq. 1) mapping Swallow-leaderboard scores to Nejumi-leaderboard scores, but the paper does not report how many models were used to fit this regression, which models they were, or the residual/uncertainty of the fit. Since several JMT-Bench values in Table 8 are marked as estimated (*), the matched comparison inherits this uncertainty. Please report the fitting set, N, R-squared, and residual variance, and consider a sensitivity analysis using the original scores without calibration.

minor comments (7)

[Section 4.1] The text says questions are generated from '1,054 Wikipedia articles' but then reports 'Out of 1,055 QAs'; please reconcile the counts.
[Section 5.1] There is a typo: 'Japanse-centric models' should be 'Japanese-centric models'.
[Section 8] The word 'encourge' in the Ethical Considerations section should be 'encourage'.
[Appendix J] The phrase 'biased or calligraphed analysis' appears to be a word-choice error; 'calligraphed' is not the intended term, and the sentence should be rephrased.
[Section 5.2, Figure 6] The perplexity-based data-leakage check uses only 20 articles and two models; please state the sample size explicitly in the text and avoid saying 'no significant difference' without a statistical test, or rephrase as 'no clear evidence of leakage in this sample'.
[Table 2 and References] Romanization is inconsistent: 柳田國男 is listed as 'Kunio Yanagita' in Table 2 but as 'Kunio Yanagida' in the reference list; please unify the transliteration.
[Appendix F] The 1-shot evaluation example instructs the judge to mark an explicitly hedged guess as correct; consider an ablation showing whether this instruction changes judge behavior, given that the judge-bias concern in the major comments depends on the judge's treatment of hedging.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: YokaiEval answers are externally grounded and human-verified; the headline comparison is an empirical result, not a fitted or self-defined quantity.

full rationale

The paper's derivation chain is not circular. The benchmark answers are generated by GPT-4o from Wikipedia articles, but then filtered with GPT-4o-mini and, crucially, manually verified against domain-expert references, with the first author and two additional annotators checking that each answer is verifiable and supported by a cited work by a folklorist or literary researcher. Thus the target quantity (knowledge of yokai) is not defined by the evaluated models, by GPT-4o's own preferences, or by the paper's conclusion. The headline result that Japanese-centric and continually-pretrained models score higher is an empirical outcome of running 31 public models on this externally grounded dataset, not a parameter fitted to produce that outcome. The use of GPT-4o as the judge of model responses is a measurement-validity concern rather than a formal circularity: the judge is instructed to compare responses against the pre-existing correct choice, and the paper reports a lexical-parser cross-check on roughly 80% of outputs plus manual review of two models. The paper itself acknowledges relevant limitations in Section 7 ('the QAs were extracted using GPT-4o, which may introduce bias into the dataset') and in Section 4.1 ('we do not have a record of inter-annotator agreement'), but these are validity caveats, not reductions of the conclusion to its inputs. The only self-citation, Jinnai (2024) providing the ChatbotArena-Ja dataset for the DPO ablation, is a secondary, non-load-bearing component: the main comparative claim does not depend on it, and the DPO effect is measured independently on YokaiEval. The JMT-Bench linear regression in Appendix G is a calibration for a control variable, not a 'prediction' that drives the central result. No equation, definition, or fitted parameter in the paper makes the claimed result true by construction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on no fitted physical parameters. The only fitted numbers are the JMT-Bench calibration coefficients. The main assumptions are about the reliability of Wikipedia as a source, GPT-4o as judge, and the leakage probe.

free parameters (1)

JMT-Bench linear regression coefficients = slope 9.576, intercept 0.868
Used to estimate JMT-Bench scores for models not on the primary leaderboard; fitted on models present in both leaderboards (Appendix G).

assumptions (3)

domain assumption Correct yokai answers are uniquely determined by Wikipedia articles and their cited references.
The dataset construction relies on this to generate and verify questions (Section 4.1).
domain assumption GPT-4o judge correctly maps a model response to true/false/null for the four-choice questions.
All model scores depend on this judge (Section 5.1, Appendix F).
domain assumption Perplexity differences between original and paraphrased articles detect memorization or data leakage.
Data leakage analysis follows Deng et al. (2024) and applies it to 20 articles (Section 5.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales." pith.science (2026). https://pith.science/paper/SFLOQNUT

@misc{pith2026250603619,
  author       = {Pith},
  title        = {Pith review of: Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SFLOQNUT}},
  note         = {Machine review of arXiv:2506.03619}
}

read the original abstract

Although Large Language Models (LLMs) have demonstrated strong language understanding and generation abilities across various languages, their cultural knowledge is often limited to English-speaking communities, which can marginalize the cultures of non-English communities. To address the problem, evaluation of the cultural awareness of the LLMs and the methods to develop culturally aware LLMs have been investigated. In this study, we focus on evaluating knowledge of folktales, a key medium for conveying and circulating culture. In particular, we focus on Japanese folktales, specifically on knowledge of Yokai. Yokai are supernatural creatures originating from Japanese folktales that continue to be popular motifs in art and entertainment today. Yokai have long served as a medium for cultural expression, making them an ideal subject for assessing the cultural awareness of LLMs. We introduce YokaiEval, a benchmark dataset consisting of 809 multiple-choice questions (each with four options) designed to probe knowledge about yokai. We evaluate the performance of 31 Japanese and multilingual LLMs on this dataset. The results show that models trained with Japanese language resources achieve higher accuracy than English-centric models, with those that underwent continued pretraining in Japanese, particularly those based on Llama-3, performing especially well. The code and dataset are available at https://github.com/CyberAgentA ILab/YokaiEval.

Figures

Figures reproduced from arXiv: 2506.03619 by the authors.

**Figure 1.** The Night Parade of One Hundred Demons (百鬼夜行) by Kawanabe Kyosai. It is said that a parade of supernatural creatures known as yokai march through the streets of Japan at night, and anyone who comes across would be spirited away. The folktale became one of the popular motifs in the Edo period portrayed in many media including ukiyo-e, toys, and picture scrolls. decent performance even in resource-limited languages (… view at source ↗

**Figure 2.** Common approach to training a culturally aware LLM is roughly divided into pretraining, supervised [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The procedure for generating YokaiEval involves several steps. First, we use articles from Wikipedia as the sources of information and generate QA pairs using GPT-4o. Next, we use GPT-4omini to remove QA pairs that are not coherent with the articles. Finally, the first author manually reviews all the data entries and their references to verify the validity of the QA pairs. information within the article. See Appendi… view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: Scores of YokaiEval and JMT-Bench. Red: Japanese-centric models (including models continually [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: The prompt for generating the response from [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: The distribution of perplexities of the original [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Prompt for generating a QA from a Wikipedia article using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 8.** Figure 8: Prompt for generating a QA from a Wikipedia article using GPT-4o (Figure [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

**Figure 9.** Figure 9: Prompt for evaluating the correctness of QAs using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: Prompt for evaluating the correctness of QAs using GPT-4o (Figure [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

**Figure 11.** Figure 11: Prompt for identifying the reference of the QA using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

**Figure 12.** Figure 12: Prompt for identifying the reference of the QA using GPT-4o (Figure [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

**Figure 13.** Figure 13: Snapshot of the annotation tool developed for the manual verification step. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

**Figure 14.** Figure 14: Prompt for categorizing the type of the QA using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

**Figure 15.** Figure 15: Prompt for categorizing the type of the QA using GPT-4o (Figure [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

**Figure 16.** Figure 16: Prompt for extracting the region of the yokai from the Wikipedia article using GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

**Figure 17.** Figure 17: Prompt for extracting the region of the yokai from the Wikipedia article using GPT-4o (Figure [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]

**Figure 18.** Figure 18: Prompt for GPT-4o Evaluation of the answer of the LLMs (Section [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]

**Figure 19.** Figure 19: Prompt for GPT-4o Evaluation of the answer of the LLMs (Figure [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages

[1]

Make the question verifiable from the description

work page
[2]

Make the question a multiple-choice question with four options

work page
[3]

その妖怪についての知識がないと正解出来ない質問にしなさい

work page
[4]

Yokai Amusements

AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. Jing Huang and Diyi Yang. 2023. Culturally aware na...

arXiv 2024
[5]

In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand

Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng,...

arXiv 2023
[6]

コードブロックのないjson形式にしなさい

work page
[7]

Understanding the capabilities and limitations of large language models for cultural commonsense. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5668–5680, Mexico City, Mexico. Association for Computational Lin- guistics. Tay...

work page 2024
[8]

Proceedings of the AAAI Conference on Artificial Intelligence , 38(18):19937–19947

Value kaleidoscope: Engaging AI with plural- istic human values, rights, and duties. Proceedings of the AAAI Conference on Artificial Intelligence , 38(18):19937–19947. Katsumi Tada. 1998. 江戸妖怪かるた [Edo Yokai Karuta]. Kokushokankoukai. Sunao Takoshima. 2018. Folkloristic elements and their embellishments among the works of shigeru mizuki. Transactions of t...

arXiv 1998

Show all 21 references

[9]

arXiv preprint arXiv:2404.01954

Hyperclova x technical report. arXiv preprint arXiv:2404.01954. Tsukioka Yoshitoshi. 1902. 新形三十六怪撰大森彦七道に怪異に逢ふ図 [Omori Hikoshichi encoun- tering a demon, new form of thirty-six-ghosts]. Linhao Yu, Yongqi Leng, Yufei Huang, Shang Wu, Haixin Liu, Xinmeng Ji, Jiahui Zhao, Jinwan...

1902 arXiv
[11]

妖怪の名前から推測不可能な質問にしなさい
[12]

妖怪に関する記述は文献によって異なる場合が多々あるため、「とされていることが多いでしょうか？」のように質問し、回答の存在を保証できるような質問にしなさい
[14]

quesiton

必ず予想では正解出来ない質問を作りなさい ###{yokai_name}についての説明文### {detail} ###Example### {{ "quesiton": " 日本の妖怪である「赤えい」の大きさはどの程度とされていることが多いでしょうか？以下の4つから回答を1つ選び出力しなさい。", "choices": [ "10メートル", "100メートル", "1キロメートル", "10キロメートル" ], "answer": "10キロメートル" }} {{ "quesiton": " 日本の妖怪である「鍛冶媼」の正体とされている動物は...
[17]

Make the question one that cannot be answered correctly without knowledge of the yokai
[18]

Make the question one that cannot be guessed from the yokai 's name
[19]

What is often said to be...?

Since descriptions of yokai often vary by source, frame the question in a way that ensures the existence of an answer, such as "What is often said to be...?"
[20]

Format the question in JSON without any code blocks
[21]

question

Ensure the question cannot be answered correctly by guessing. ###Example### {{ "question": "What is often said to be the size of the Japanese yokai 'Akaei'? Choose one answer from the following four options.", "choices": [ "10 meters", "100 meters", "1 kilometer", "10 kilomete...
[2015]

In Proceedings of the Fourth Workshop on Computa- tional Linguistics for Literature, pages 42–47, Den- ver, Colorado, USA

GutenTag: an NLP-driven tool for digital hu- manities research in the Project Gutenberg corpus. In Proceedings of the Fourth Workshop on Computa- tional Linguistics for Literature, pages 42–47, Den- ver, Colorado, USA. Association for Computational Linguistics. Olena Burda-Las...

2022 arXiv
[2016]

In Proceedings of the 8th Global WordNet Conference (GWC), pages 83–87, Bucharest, Romania

Towards a WordNet based classification of actors in folktales. In Proceedings of the 8th Global WordNet Conference (GWC), pages 83–87, Bucharest, Romania. Global Wordnet Association. Thierry Declerck, Nikolina Koleva, and Hans-Ulrich Krieger. 2012. Ontology-based incremental a...

2012 arXiv
[2023]

In Thirty-seventh Conference on Neural Information Processing Sys- tems

Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Sys- tems. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A bench- mark for measuring th...

2024 arXiv
[2024]

In First Conference on Language Modeling

Continual pre-training for cross-lingual LLM adaptation: Enhancing japanese language capabili- ties. In First Conference on Language Modeling. Koyoharu Gotouge. 2016. 鬼滅の刃 [Demon Slayer: Kimetsu no Yaiba]. Shueisha. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lho...

2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Make the question verifiable from the description

work page

[2] [2]

Make the question a multiple-choice question with four options

work page

[3] [3]

その妖怪についての知識がないと正解出来ない質問にしなさい

work page

[4] [4]

Yokai Amusements

AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. Jing Huang and Diyi Yang. 2023. Culturally aware na...

arXiv 2024

[5] [5]

In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand

Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng,...

arXiv 2023

[6] [6]

コードブロックのないjson形式にしなさい

work page

[7] [7]

Understanding the capabilities and limitations of large language models for cultural commonsense. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5668–5680, Mexico City, Mexico. Association for Computational Lin- guistics. Tay...

work page 2024

[8] [8]

Proceedings of the AAAI Conference on Artificial Intelligence , 38(18):19937–19947

Value kaleidoscope: Engaging AI with plural- istic human values, rights, and duties. Proceedings of the AAAI Conference on Artificial Intelligence , 38(18):19937–19947. Katsumi Tada. 1998. 江戸妖怪かるた [Edo Yokai Karuta]. Kokushokankoukai. Sunao Takoshima. 2018. Folkloristic elements and their embellishments among the works of shigeru mizuki. Transactions of t...

arXiv 1998