Pith. sign in

REVIEW 4 major objections 5 minor 63 references

D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Model

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read D-GEN, a fine-tuned language model, generates multiple-choice distractors that preserve the ranking of 42 model configurations (Spearman's $\rho$ 0.99) and match the confidence distribution of human-written distractors.

desk verdict Useful distractor-generation model with honest experiments, but the claim of difficulty preservation is overstated given the ~5% accuracy drop in Table 3. read the letter →

arxiv 2504.13439 v2 pith:V5A6TIP3 submitted 2025-04-18 cs.CL

Grace Byun , Jinho D. Choi This is my paper · ORCID
classification cs.CL
keywords distractorgenerationmultiple-choiceevaluationMMLUrankingalignmententropyanalysislargelanguagemodelshumanbenchmarkconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

D-GEN is a fine-tuned large language model that turns open-ended questions into multiple-choice format by generating three plausible but incorrect distractors for each question. The paper argues that these machine-written distractors are interchangeable with human-written ones: across 21 models and two few-shot settings, model rankings on the rewritten MMLU benchmark match the original with Spearman's $\rho$ 0.99 and Kendall's $\tau$ 0.94, and the entropy of models' answer-choice distributions is statistically indistinguishable from the original for two of three judge models. Human ratings on seven other tasks confirm the distractors are fluent, coherent, distracting, and incorrect. If the claim holds, expensive human distractor writing can be replaced by an automated pipeline, making multiple-choice evaluation fast, scalable, and reliable.

What carries the argument

The central object is the D-GEN model itself: a LLaMA-3.3-70B-Instruct (and an 8B variant) fine-tuned on the MMLU auxiliary training set to output three semantically close but incorrect distractors for a question and its correct answer. An automatic correction loop filters and regenerates distractors until they are unique and non-overlapping with the correct answer, and the correct answer is randomly placed among the four options to cancel position bias. The evaluation rests on two quantitative instruments: ranking alignment (Spearman's $\rho$ and Kendall's $\tau$ between model performance ranks on original MMLU and MMLU-DGEN) and entropy analysis (the Shannon entropy $H(p)=-\sum_i p_i \log p_i$ of the softmax probability distribution over A/B/C/D, compared with Wilcoxon signed-rank tests). These instruments are what carry the argument that the generated distractors have the same discriminatory power and plausibility as human-written ones.

What would settle it

Run the paper's own human-evaluation protocol (fluency, coherence, distractiveness, incorrectness) on a random sample of MMLU-DGEN questions: if the distractiveness or incorrectness ratings fall clearly below the original MMLU distractors', or if naive human test-takers score significantly lower on MMLU-DGEN than on MMLU, the claim that the generated distractors are interchangeable with the originals fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that D-GEN-generated distractors preserve the measurement properties of the original MMLU test set. Concretely, replacing the human-written wrong answers with D-GEN's distractors does not change the relative ordering of 42 model configurations (Spearman's $\rho$ 0.99, Kendall's $\tau$ 0.94), and the distributions of model confidence, measured as entropy over the four choices, are close enough that Wilcoxon signed-rank tests find no significant difference for two of the three judge models and only one domain (Social Sciences on Llama-3.3-70B-Instruct) shows $p < 0.05$. The paper also demonstrates on seven FLAN tasks that the generated distractors receive high average scores (mostly 4-5 on a 1-5 scale) for fluency, coherence, distractiveness, and incorrectness. The authors intend this as evidence that D-GEN is the first open-source distractor generator reliable enough for automated multiple-choice evaluation.

Load-bearing premise

The argument assumes that two aggregate statistics, rank correlation across 42 model configurations and matched entropy distributions on three judge models, prove that D-GEN's distractors are as good as human-written ones, even though no human ever scores the MMLU-DGEN distractors themselves; human scores come only from different FLAN tasks.

Editorial extensions

If this is right

  • Benchmarks like MMLU can be regenerated or extended with fresh distractors without re-running human annotation, at a fraction of the cost.
  • Model evaluation on open-ended tasks can be converted to multiple-choice format automatically, reducing false negatives caused by format inconsistencies in generation.
  • The ranking-alignment and entropy tests give future distractor generators a quantitative, scalable acceptance criterion that does not require expert human judgment.
  • Because the D-GEN distractors slightly increase difficulty (negative mean accuracy differences across all domains), benchmarks built this way may be marginally harder than their originals, a shift the headline rank-correlation statistic does not reveal.
  • The 8B variant and open-source release allow other researchers to generate distractors for new datasets without access to proprietary models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The rank-correlation evidence would survive a uniform difficulty shift, so the paper's strongest statistics do not by themselves prove that absolute difficulty is preserved; a direct human or behavioral comparison of MMLU versus MMLU-DGEN would settle this.
  • The same entropy-matching methodology could be adapted as a screening test for adversarial-robustness benchmarks: distractors that maximize model uncertainty are exactly the options that stress-test calibrated confidence.
  • A natural extension is to train D-GEN on the distractors generated by itself, bootstrapping new MC benchmarks from open-ended data with no human-written gold options, which would remove the ground-truth dependency the paper names as its main scalability limitation.
  • The paper's finding that general-purpose LLM judges penalize correct-but-intentionally-incorrect distractors suggests automated distractor evaluation by such judges needs task-specific calibration, a caveat that applies beyond this paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces D-GEN, an open-source distractor generation model obtained by fine-tuning Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct on the MMLU auxiliary training set. It proposes two automated evaluation metrics for distractor quality: ranking alignment (Spearman/Kendall correlation of model accuracies between original MMLU and MMLU-DGEN) and entropy analysis (comparison of model confidence distributions). Experiments show Spearman rho 0.99 and Kendall tau 0.94 over 42 configurations, and entropy differences that are mostly not statistically significant. Human evaluation on FLAN tasks reports high fluency, coherence, distractiveness, and incorrectness scores. The paper claims that D-GEN can replace human-written distractors in multiple-choice benchmarks without altering relative model rankings.

Significance. If the central claim is established, D-GEN offers a practical, low-cost way to convert open-ended tasks into multiple-choice format and to enlarge or refresh MC benchmarks with automatically generated distractors. The paper's strengths include the release of the models and datasets, the use of a broad set of 21 models with 0- and 5-shot settings, and the honest acknowledgment in Section 4.4 and the Limitations that MMLU-DGEN is slightly harder and that the evaluation methods depend on ground-truth distractors. However, as detailed below, the evidence for difficulty equivalence is incomplete: the headline rank-correlation metric is insensitive to the systematic accuracy drop reported in Table 3, and the human evaluation does not cover the MMLU-DGEN set. With additional targeted analyses, the work could become a solid contribution to reliable MC evaluation.

major comments (4)
  1. [§4.4, Table 3] The paper reports mean accuracy differences (MMLU-DGEN minus MMLU) of -0.05 overall, with per-domain means of -0.03 to -0.07 and minimums of -0.09/-0.10. This is a systematic downward shift in scores. Spearman's rho and Kendall's tau are invariant to monotone transformations of accuracy, so the high rank correlations in Table 2 cannot detect such a uniform difficulty shift. Since the central claim is that D-GEN can replace human-written MMLU distractors in benchmark evaluation, the paper needs additional evidence of difficulty equivalence, e.g., item-level accuracy distributions, calibration plots, or human scoring on a sample of MMLU-DGEN items.
  2. [§6.2–6.3] Human evaluation is conducted only on 700 FLAN examples across seven tasks; no human evaluation is performed on the MMLU-DGEN set that is the primary product of the paper. The abstract's statement that "Human evaluation further confirms the fluency, coherence, distractiveness, and incorrectness" therefore does not apply to MMLU-DGEN, where the replacement claim actually stands. Without human judgment on MMLU-DGEN, the quality of these distractors rests entirely on proxy metrics, which the paper itself shows are imperfect (Table 3 difficulty shift).
  3. [§5.3, Table 4] The entropy analysis finds no statistically significant difference in 11 of 12 model-domain pairs, but the direction is highly consistent: D-GEN entropy is higher in every domain for Llama and Qwen, and in 2 of 4 domains for Mixtral. With domain-level sample sizes of 12–19 subcategories, the Wilcoxon tests have limited power, and no multiple-comparison correction is applied. The paper's conclusion that the entropy distributions "closely match" is therefore not strongly supported; the data are consistent with a small but systematic increase in uncertainty, matching the difficulty shift in Table 3. I recommend reporting effect sizes, confidence intervals, and item-level analyses.
  4. [§6.2, Appendix F.3] The human evaluation section does not report the number of annotators, their qualifications, or inter-annotator agreement. As a result, the reliability of the reported average scores (Table 7) cannot be assessed. Since human evaluation is used to validate distractor quality, this information is essential for the claims.
minor comments (5)
  1. [Abstract, §5] The abstract says "closely matches the entropy distribution," but Section 5 only reports mean entropy and p-values; the full distributions are not shown.
  2. [Appendix D.2] The text states "Most of these p-values exceed 0.05," but Table 65 lists rank-correlation p-values that are all far below 0.05; this appears to be a copy-paste error from the entropy analysis.
  3. [§4.2] Please clarify whether "preserving the original distribution of correct answer indices" means preserving the sequence of correct positions or just the marginal distribution.
  4. [§5.3] The one significant p-value (0.0342) is described as "the only exception," but with 12 tests, one significant result is expected by chance; no multiple-comparison correction is discussed.
  5. [Table 3] Consider reporting the raw accuracy values in addition to the differences, since a mean difference of -0.05 may be small for some models but substantial for others.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the D-GEN evaluation measures generated distractors against held-out ground-truth MMLU distractors, and the reported statistics are measurements, not fitted inputs.

full rationale

The paper's derivation chain is self-contained with respect to circularity. D-GEN is fine-tuned on the MMLU auxiliary training set (Section 3.1) and evaluated on the held-out MMLU test by replacing original distractors with generated ones (Section 4.2), so the target quantities (MMLU-DGEN accuracy, entropy, and human scores) are not inputs to the training procedure. The rank-alignment statistics (Spearman 0.9918, Kendall 0.9413, Table 2) and Wilcoxon entropy comparisons (Tables 4 and 67) are direct measurements against ground-truth distractors, not parameters fitted to reproduce those numbers. No equation in the paper defines distractor quality as the same quantity used to fit or train D-GEN. The entropy analysis includes Llama-3.3-70B-Instruct, the base model from which D-GEN is derived, which is a partial confound, but the comparison is partially controlled by two unrelated models (Qwen and Mixtral), and the paper reports the one significant Llama Social Sciences result (p = 0.0342) rather than suppressing it. The human evaluation is conducted on FLAN tasks rather than on MMLU-DGEN, which is an evidence-scope gap and a correctness risk, but it is not circular because the human scores are not used to define or fit D-GEN's outputs. The paper also candidly discloses that MMLU-DGEN is 'slightly more challenging' (Section 4.4, Table 3, mean accuracy difference -0.05), a finding that qualifies the replacement claim but does not indicate circularity. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation appears in the derivation chain.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to the evaluation outcomes. The paper's claims rest on three domain assumptions about the MMLU training set, rank correlation as a quality proxy, and entropy as a plausibility proxy. No new entities are introduced.

assumptions (3)
  • domain assumption The MMLU auxiliary training set provides a sufficient and representative collection of question-answer-distractor triples to teach distractor generation.
    D-GEN is trained only on this set, and the model's generalization to MMLU test and FLAN tasks is assumed. Stated in Section 3.1.
  • domain assumption Model rankings on MMLU are a valid proxy for the discriminatory power of a distractor set.
    The ranking alignment test in Section 4.2 assumes that preserving the order of 21 models means the distractors separate strong from weak models just as well as the original set.
  • domain assumption Entropy of model confidence over answer choices reflects distractor plausibility.
    Section 5.1 defines entropy as a holistic metric; the paper does not establish a formal link between entropy and human-judged plausibility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Model." pith.science (2026). https://pith.science/paper/V5A6TIP3

@misc{pith2026250413439,
  author       = {Pith},
  title        = {Pith review of: D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5A6TIP3}},
  note         = {Machine review of arXiv:2504.13439}
}
read the original abstract

Evaluating generative models with open-ended generation is challenging due to inconsistencies in response formats. Multiple-choice (MC) evaluation mitigates this issue, but generating high-quality distractors is time-consuming and labor-intensive. We introduce D-GEN, the first open-source distractor generator model that transforms open-ended data into an MC format. To evaluate distractor quality, we propose two novel methods: (1) ranking alignment, ensuring generated distractors retain the discriminatory power of ground-truth distractors, and (2) entropy analysis, comparing model confidence distributions. Our results show that D-GEN preserves ranking consistency (Spearman's rho 0.99, Kendall's tau 0.94) and closely matches the entropy distribution of ground-truth distractors. Human evaluation further confirms the fluency, coherence, distractiveness, and incorrectness. Our work advances robust and efficient distractor generation with automated evaluation, setting a new standard for MC evaluation.

Figures

Figures reproduced from arXiv: 2504.13439 by the authors.

Figure 1
Figure 1. D-GEN Training: We fine-tune LLaMA using the auxiliary training set from MMLU. The model is trained to generate a list of distractors for a given question and its correct answer [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Filtering and Regeneration Process: The system automatically identifies and filters low-quality distractors [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Scatter plots comparing the ranks of 42 configurations (21 models evaluated at both 0-shot and 5-shot [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Proportion of the scores (1–5) across different [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (A) Percentage of distractors with at least [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Prompt used to fine-tune D-GEN Prompt messages = [ { "role": "system", "content": "Your task is to generate 3 incorrect but plausible distractors for the given question. The distractors should be semantically related to the context of the question and close to the corr…
Figure 7
Figure 7. Figure 7: Prompt used to generate three distractors for MMLU questions using [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Entropy Calculation Process [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Prompt used to generate distractors for reading comprehension questions. Demonstrations vary depending [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: A user interface for human evaluation, where evaluators rate distractors generated by [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: Prompt used for GPT evaluation of the distractors generated by [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]
Figure 12
Figure 12. Figure 12: Prompt used for GPT evaluation of the distractors generated by [PITH_FULL_IMAGE:figures/full_fig_p032_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 17 canonical work pages

  1. [1]

    Sheng, Wei Emma Zhang, Munazza Zaib, and Ahoud Alhazmi

    Elaf Alhazmi, Quan Z. Sheng, Wei Emma Zhang, Munazza Zaib, and Ahoud Alhazmi. 2024. https://arxiv.org/abs/2402.01512 Distractor generation in multiple-choice tasks: A survey of methods, datasets, and evaluation . Preprint, arXiv:2402.01512

    arXiv 2024

  2. [2]

    Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, and Teruko Mitamura. 2016. https://aclanthology.org/C16-1107/ Generating questions and multiple-choice answers using semantic analysis of texts . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pages 112...

    work page 2016

  3. [3]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. https://api.semanticscholar.org/CorpusID:208290939 Piqa: Reasoning about physical commonsense in natural language . In AAAI Conference on Artificial Intelligence

    2019

  4. [4]

    Semere Kiros Bitew, Johannes Deleu, Chris Develder, and Thomas Demeester. 2023. https://arxiv.org/abs/2307.16338 Distractor generation for multiple-choice questions with predictive prompting and large language models . Preprint, arXiv:2307.16338

    arXiv 2023

  5. [5]

    Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aur \'e lie N \'e v \'e ol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. ...

    doi:10.18653/v1/w16-2301 2016

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    arXiv 2018

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

Show all 63 references
  1. [9]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/N19-1246 DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  2. [10]

    Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. https://doi.org/10.18653/v1/P19-1102 Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model . In Proceedings of the 57th Annual Meeting of the Association for...

    2019 doi

  3. [11]

    Claire Cardie Faisal Ladhak, Esin Durmus and Kathleen McKeown. 2020. Wikilingua: A new benchmark dataset for multilingual abstractive summarization. In Findings of EMNLP, 2020

    2020

  4. [12]

    Wanyong Feng, Jaewook Lee, Hunter McNichols, Alexander Scarlatos, Digory Smith, Simon Woodhead, Nancy Ornelas, and Andrew Lan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.193 Exploring automated distractor generation for math multiple-choice questions via large langu...

    2024 doi

  5. [13]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  6. [14]

    Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. https://doi.org/10.18653/v1/W17-3518 The W eb NLG challenge: Generating text from RDF data . In Proceedings of the 10th International Conference on Natural Language Generation, pages 124--1...

    2017 doi

  7. [15]

    Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. https://aclanthology.org/S12-1052/ S em E val-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning . In * SEM 2012: The First Joint Conference on Lexical and Computational Sem...

    2012

  8. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  9. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  10. [18]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset . Preprint, arXiv:2103.03874

    2021 arXiv

  11. [19]

    Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. https://arxiv.org/abs/1506.03340 Teaching machines to read and comprehend . Preprint, arXiv:1506.03340

    2015 arXiv

  12. [20]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  13. [21]

    Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1243 Cosmos QA : Machine reading comprehension with contextual commonsense reasoning . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processin...

    2019 doi

  14. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  15. [23]

    ManjulaShenoy, Shashank Goyal, and Chaitanya

    Archana Praveen Kumar, Ashalatha Nayak, K. ManjulaShenoy, Shashank Goyal, and Chaitanya. 2023. https://api.semanticscholar.org/CorpusID:258243349 A novel approach to generate distractors for multiple choice questions . Expert Syst. Appl., 225:120022

    2023

  16. [24]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017 a . https://arxiv.org/abs/1704.04683 Race: Large-scale reading comprehension dataset from examinations . Preprint, arXiv:1704.04683

    2017 arXiv

  17. [25]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017 b . https://doi.org/10.18653/v1/D17-1082 RACE : Large-scale R e A ding comprehension dataset from examinations . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pa...

    2017 doi

  18. [26]

    Lee Giles

    Chen Liang, Xiao Yang, Neisarg Dave, Drew Wham, Bart Pursel, and C. Lee Giles. 2018. https://doi.org/10.18653/v1/W18-0533 Distractor generation for multiple choice questions using learning to rank . In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Buildin...

    2018 doi

  19. [27]

    Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.165 C ommon G en: A constrained text generation challenge for generative commonsense reasoning . In Findings of the Ass...

    2020 doi

  20. [28]

    Le, Barret Zoph, Jason Wei, and Adam Roberts

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. https://arxiv.org/abs/2301.13688 The flan collection: Designing data and methods for effective instruction tuning . Preprint, arXiv:2301.13688

    2023 arXiv

  21. [29]

    Subhankar Maity, Aniket Deroy, and Sudeshna Sarkar. 2024. https://arxiv.org/abs/2401.07098 A novel multi-stage prompting approach for language agnostic mcq generation using gpt . Preprint, arXiv:2401.07098

    2024 arXiv

  22. [30]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 a . https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Languag...

    2018 doi

  23. [31]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 b . Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP

    2018

  24. [32]

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. https://doi.org/10.18653/v1/N16-1098 A corpus and cloze evaluation for deeper understanding of commonsense stories . In Proceedings of the 2...

    2016 doi

  25. [33]

    Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. 2016. https://arxiv.org/abs/1602.06023 Abstractive text summarization using sequence-to-sequence rnns and beyond . Preprint, arXiv:1602.06023

    2016 arXiv

  26. [34]

    Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern ...

    2021 doi

  27. [35]

    Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. https://aclanthology.org/W12-3018/ Annotated G igaword . In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction ( AKBC - WEKEX ) , pages 95--100, Montr \'...

    2012

  28. [36]

    Jeroen Offerijns, Suzan Verberne, and Tessa Verhoef. 2020. https://arxiv.org/abs/2010.09598 Better distractions: Transformer-based distractor generation and multiple choice question filtering . Preprint, arXiv:2010.09598

    2020 arXiv

  29. [37]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  30. [38]

    Juan Miguel Pino and Maxine Esk \'e nazi. 2009. https://api.semanticscholar.org/CorpusID:1400908 Semi-automatic generation of cloze question distractors effect of students' l1 . In Slate

    2009

  31. [39]

    Yevgeniy Puzikov and Iryna Gurevych. 2018. https://doi.org/10.18653/v1/W18-6557 E 2 E NLG challenge: Neural models vs. templates . In Proceedings of the 11th International Conference on Natural Language Generation, pages 463--471, Tilburg University, The Netherlands. Associati...

    2018 doi

  32. [40]

    Zhaopeng Qiu, Xian Wu, and Wei Fan. 2020. https://doi.org/10.18653/v1/2020.coling-main.189 Automatic distractor generation for multiple choice questions in standard tests . In Proceedings of the 28th International Conference on Computational Linguistics, pages 2096--2106, Barc...

    2020 doi

  33. [41]

    Fanyi Qu, Hao Sun, and Yunfang Wu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.47 Unsupervised distractor generation via large language model distilling and counterfactual contrastive decoding . In Findings of the Association for Computational Linguistics: ACL 2024, pa...

    2024 doi

  34. [42]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  35. [43]

    Vatsal Raina, Adian Liusie, and Mark J. F. Gales. 2023. https://api.semanticscholar.org/CorpusID:265050481 Assessing distractors in multiple-choice tests . In EVAL4NLP

    2023

  36. [44]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://arxiv.org/abs/1806.03822 Know what you don't know: Unanswerable questions for squad . Preprint, arXiv:1806.03822

    2018 arXiv

  37. [45]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  38. [46]

    Siyu Ren and Kenny Q. Zhu. 2020. https://arxiv.org/abs/2004.09853 Knowledge-driven distractor generation for cloze-style multiple choice questions . Preprint, arXiv:2004.09853

    2020 arXiv

  39. [47]

    Burges, and Erin Renshaw

    Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. https://aclanthology.org/D13-1020/ MCT est: A challenge dataset for the open-domain machine comprehension of text . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, p...

    2013

  40. [48]

    Pengju Shuai, Zixi Wei, Sishun Liu, Xiaofei Xu, and Li Li. 2021. https://doi.org/10.1109/IJCNN52387.2021.9533341 Topic enhanced multi-head co-attention: Generating distractors for reading comprehension . In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1--8

    2021

  41. [49]

    Katherine Stasaski and Marti A. Hearst. 2017. https://doi.org/10.18653/v1/W17-5034 Multiple choice question generation utilizing an ontology . In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications , pages 303--312, Copenhagen, Denma...

    2017 doi

  42. [50]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  43. [51]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust and...

    2024 arXiv

  44. [52]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. https://arxiv.org/abs/2109.01652 Finetuned language models are zero-shot learners . Preprint, arXiv:2109.01652

    2022 arXiv

  45. [53]

    Han Cheng Yu, Yu An Shih, Kin Man Law, KaiYu Hsieh, Yu Chen Cheng, Hsin Chih Ho, Zih An Lin, Wen-Chuan Hsu, and Yao-Chung Fan. 2024. https://doi.org/10.18653/v1/2024.findings-acl.655 Enhancing distractor generation for multiple-choice questions with retrieval augmented pretrai...

    2024 doi

  46. [54]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 a . https://arxiv.org/abs/1905.07830 Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830

    2019 arXiv

  47. [55]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 b . https://api.semanticscholar.org/CorpusID:159041722 Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics

    2019

  48. [56]

    Rui Zhang and Joel Tetreault. 2019. https://doi.org/10.18653/v1/P19-1043 This email could save your life: Introducing the task of email subject line generation . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 446--456, Florenc...

    2019 doi

  49. [57]

    Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. https://api.semanticscholar.org/CorpusID:53116244 Record: Bridging the gap between human and machine commonsense reading comprehension . ArXiv, abs/1810.12885

    2018 arXiv

  50. [58]

    Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. https://api.semanticscholar.org/CorpusID:368182 Character-level convolutional networks for text classification . In Neural Information Processing Systems

    2015

  51. [59]

    Ziyin Zhang, Zhaokun Jiang, Lizhen Xu, Hongkun Hao, and Rui Wang. 2024. https://arxiv.org/abs/2405.11966 Multiple-choice questions are efficient and robust llm evaluators . Preprint, arXiv:2405.11966

    2024 arXiv

  52. [60]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

  53. [61]

    Xiaorui Zhou, Senlin Luo, and Yunfang Wu. 2019. https://arxiv.org/abs/1911.08648 Co-attention hierarchical network: Generating coherent long distractors for reading comprehension . Preprint, arXiv:1911.08648

    2019 arXiv

  54. [62]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  55. [63]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.