Pith. sign in

REVIEW 3 major objections 7 minor 27 references

Multiple-Choice Question Generation Using Large Language Models: Methodology and Educator Insights

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that, in a 21-educator comparative trial, GPT-3.5 generated the most effective multiple-choice questions on four of five evaluation metrics under a knowledge-injected prompt.

desk verdict A small, honest comparative study whose abstract overclaims pairwise superiority from an omnibus test; the data lean the way they say, but the statistics don't fully carry the headline. read the letter →

arxiv 2506.04851 v1 pith:ZMFQTGAP submitted 2025-06-05 cs.CL cs.AI

Giorgio Biancini , Alessio Ferrato , Carla Limongelli This is my paper
classification cs.CLcs.AI
keywords largelanguagemodelsmultiple-choicequestiongenerationGPT-3.5Llama2Mistralknowledgeinjectionpromptengineeringeducatorevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a single, carefully worded prompt can turn any source text into exam-ready multiple-choice questions, and that among three widely used language models, GPT-3.5 does that best. In the experiment, 21 high school and university teachers rated seven questions from each of GPT-3.5, Llama 2, and Mistral, using the same knowledge-injected prompt and five quality criteria. GPT-3.5 consistently received the highest ratings on all five criteria, and the differences were statistically significant for clarity, coherence, compliance with the source text, and distractor selection; only learning utility lacked a significant difference. The paper also reports that most participating educators would be willing to use such a generator, while a substantial group stays neutral. If the result holds, teachers can automate the time-consuming drafting of quizzes while keeping control of the source material.

What carries the argument

The central mechanism is a knowledge-injected prompt template. The prompt instructs the model to act as an infallible assistant, to create a high-school-level quiz strictly from the provided text, to format each item as a stem, options a through d, a correct-answer letter, and an exact source passage, and to make all options distinct while specifying years for dates. By feeding the source paragraph directly into the prompt, the design bypasses the model's internal memory, giving educators control over the test's content and reducing hallucinations. The evaluation machinery is the five-criteria rubric (compliance, clarity, distractor selection, coherence of key selection, and learning utility) rated by the 21 educators, with a non-parametric repeated-measures test used to check whether the score differences across models are significant.

What would settle it

Run the same prompt and rating procedure on several unrelated sources, such as a biology textbook chapter and a computer science manual, randomly selecting seven questions per model each time; the claim of GPT-3.5's general superiority would be falsified if it does not consistently rank first on the four metrics where it currently shows significant advantages.

Watch

Extended reading notes

Core claim

The paper's central claim is that GPT-3.5 produces the most effective MCQs under the proposed prompt-based method, outperforming Llama 2 and Mistral on every metric and doing so significantly on four of the five. The approach the authors stress is not fine-tuning but a knowledge-injected prompt: the model is told to act as an assistant that makes no mistakes, given a source paragraph, and is required to emit the question, four distinct options, a marked correct answer, and an exact source quotation. The teachers' ratings place GPT-3.5 first, Llama 2 second, and Mistral third, with significant differences on compliance, clarity, coherence of key selection, and distractor selection. The study also found that Llama 2 and Mistral sometimes reported placeholders like 'Line 5 of the text' as the source, which hurt their compliance and coherence scores; GPT-3.5 did not show this behavior. The authors position their work as an extension of earlier automated quiz-generation research, adding a larger educator sample and a prompt that does not rely on the models' pre-trained knowledge.

Load-bearing premise

The study's ranking rests on a single source text about the causes of World War II and on one random draw of seven questions per model, so the claim that GPT-3.5 is generally the best MCQ generator would fail if other topics or other draws produce a different ordering.

Editorial extensions

If this is right

  • A single source-grounded prompt is enough to get usable MCQs from LLMs without fine-tuning, which lowers the barrier for teachers to produce quizzes from their own materials.
  • Educators can keep control of the test's source text, because the prompt forces the model to answer from the injected paragraph rather than from memory, making the output traceable.
  • GPT-3.5 is the recommended model among the three tested for immediate use in this pipeline, while Llama 2 and Mistral need source-grounding improvements before they match it.
  • The statistically significant gaps on compliance and coherence mean open-weight models currently fail at the step that makes AI-generated questions verifiable in the classroom: citing the evidence.
  • Adoption is not guaranteed by quality alone: only eleven of the 21 educators said they would use such a system, while eight stayed neutral, pointing to trust as a separate hurdle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the topic dependence the authors concede means the GPT-3.5 ranking is a provisional result; the same protocol run on a biology or computer-science text could reorder the models, and a multi-topic test would be the natural follow-up.
  • Editorial inference: the non-significant learning-utility result suggests perceived usefulness may track the teacher's pedagogical goal and how the questions will be used, not just model output; a study that asks educators to name their intended use before rating could separate these effects.
  • Editorial inference: because knowledge injection already supplies the source text, the pipeline is a short step away from retrieval-augmented generation, where a verified knowledge base replaces a hand-picked paragraph, making the technique scalable to whole courses.
  • Editorial inference: the source-citation failures of Llama 2 and Mistral are the clearest thing to fix; a post-processing step that checks the cited line against the source would likely close most of the quality gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper compares three LLMs (Llama 2, Mistral, and GPT-3.5) for generating multiple-choice questions (MCQs) from a provided source text, using a knowledge-injection prompt that supplies the text to the model. Twenty-one educators rated the generated MCQs on five criteria from the literature (compliance, clarity, distractor selection, coherence, and learning utility) via a Likert scale. The authors report Friedman tests showing statistically significant differences among the three models on four of five metrics, with GPT-3.5 showing the highest sample means, and they conclude that GPT-3.5 generates the most effective MCQs. The paper also reports educators' mixed willingness to adopt such systems and discusses limitations including small sample size, a single topic, and lack of comparison with human-generated MCQs.

Significance. If the comparative claim is supported, the paper offers a practical, prompt-based methodology for using LLMs in MCQ generation, a task of genuine interest to the educational-technology community. The design choice to inject the source text into the prompt rather than relying on parametric knowledge is a sound practical mitigation for hallucination and gives educators control over the test's content. The use of a repeated-measures design with educators as evaluators is appropriate for the research question, and the paper transparently reports its limitations. However, the paper's headline claim—that GPT-3.5 is the most effective generator—rests on a statistical inference that is not fully established by the reported tests, and the lack of controlled generation parameters for GPT-3.5 raises comparability questions. These issues are correctable but are load-bearing for the paper's main contribution.

major comments (3)
  1. [Section 4.2, Table 1] The claim that 'GPT-3.5 consistently outperformed the other two LLMs' is not supported by the Friedman test as reported. The Friedman test's alternative hypothesis is that at least one model differs from the others; rejecting it says nothing about which pairs differ. Since no post-hoc pairwise tests (e.g., Wilcoxon signed-rank tests with a multiplicity correction) are reported, the direction of the effect is inferred solely from descriptive means and boxplots. The abstract's statement that GPT-3.5 generates the most effective MCQs is therefore stronger than the statistical evidence warrants. Please add pairwise comparisons with an appropriate correction, or adjust the conclusion to state that GPT-3.5 had higher sample means, with significant omnibus differences on four metrics.
  2. [Section 4.1.3] The description of GPT-3.5 generation is ambiguous and potentially inconsistent. The text says 'For generations using GPT-3.5, we used ChatGPT 4 and provided a new prompt for every paragraph while omitting the chat history from the account settings.' If GPT-3.5 was accessed through the ChatGPT chat interface, its sampling parameters (temperature, top-p, etc.) are not controlled and likely differ from the explicit temperature=0.5, top-k=30, and max_tokens=2048 settings used for Llama 2 and Mistral via HuggingFace. If instead GPT-4 was used, the model label is wrong. Either way, the comparison is not controlled across models, which weakens the validity of the observed performance differences. Please clarify the exact model, interface, and any accessible generation parameters, and discuss the consequence for the comparison.
  3. [Section 4.1.3 and Section 5] The generalizable conclusion that GPT-3.5 is the most effective MCG generator is based on a single Wikipedia article (Causes of World War II) and seven randomly selected MCQs per model. The paper acknowledges this in the limitations, but the abstract still states the result without hedging. Given the acknowledged topic dependence of LLM performance, the conclusion should be restricted to the tested topic or clearly framed as a pilot study, with the topic-dependence as an explicit boundary condition.
minor comments (7)
  1. [Abstract] The sentence 'but there are still space for new applications' contains a subject-verb agreement error; it should read 'there is still space'.
  2. [Section 4.1.4] Typo: 'the text sourrce' should be 'the text source'.
  3. [Table 2, GPT-3.5 first question] The correct-answer option 'b' corresponds to option 'b) The attacks on Malaysia, Singapore and Hong Kong', but the source text states that Canada declared war on Japan citing specifically the attacks on Malaya, Singapore and Hong Kong. The option text says 'Malaysia' while the source says 'Malaya'; this may be intentional but should be checked for consistency.
  4. [Table 3, Llama 2 second question] The option 'd) He wanted to secure a strategic alliance with the Soviet Union' is marked correct, but the source text states that Hitler hoped defeating the Soviet Union would bring Britain to the negotiating table. The marked answer appears incorrect relative to the source; this may affect the fidelity of the 'compliance' metric.
  5. [Table 4, Mistral first question] Option 'b' lacks a closing parenthesis: 'b December 7, 1941' should be 'b) December 7, 1941'.
  6. [Table 1] The table layout is confusing: the headings for the three models are interleaved with the metric label in a way that makes it easy to misread which mean/median belongs to which metric. A clearer structure (e.g., separate subtables or explicit model columns) would improve readability.
  7. [References] Several reference keys contain malformed labels or spacing (e.g., '[RR W22]', '[SL W+24]', '[WM+23]'), and the bibliography should be formatted consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the comparison rests on external educator ratings, not on fitted parameters or self-citation chains.

full rationale

The paper's central claim—that GPT-3.5 generates more effective MCQs—is an empirical comparative finding obtained by having 21 educators rate questions produced by three LLMs on five criteria drawn from an external survey (Rao and Saha [CS20]). The prompt used for generation is fixed and disclosed, the questions are randomly sampled per model, and the ratings are external human judgments; no parameter is fitted to the outcome and then presented as a prediction, and no derived quantity is defined in terms of another quantity it is supposed to explain. The few self-citations ([FLMS22a], [FLMS22b], [DAGMS17], [SGM19], [CBPR21]) appear only in speculative future-work remarks about personalization and cultural-heritage recommender systems; they are not load-bearing for the MCQ-generation comparison. Concerns that the omnibus Friedman test does not by itself establish pairwise superiority, or that the single World War II topic limits generality, are statistical and external-validity issues, not circularity: the conclusion is not forced by construction or by an unverified self-citation chain. Accordingly, no circular step can be quoted with the required specificity, and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No numerical parameters are fitted to the rating data. The hand-chosen generation settings (temperature, top-k, max tokens) are listed as free parameters because they are not swept and could affect the model ranking. The central claim also rests on the validity of the five evaluation criteria, the representativeness of the single-topic sample, and the assumption that a significant Friedman test licenses the pairwise conclusion.

free parameters (3)
  • Temperature (Llama 2 and Mistral) = 0.5
    Hand-chosen generation temperature; no sweep is reported, so the ranking could depend on this value.
  • Top-k sampling (Llama 2 and Mistral) = 30
    Hand-chosen top-k value; not optimized or varied across the study.
  • Maximum new tokens (Llama 2 and Mistral) = 2048
    Generation cap; chosen by hand, unlikely to affect short MCQs but part of the fixed setup.
assumptions (4)
  • domain assumption The five evaluation criteria from [CS20] are valid and sufficient measures of MCQ quality.
    The study relies entirely on these Likert-scale criteria; no inter-rater reliability or external benchmark validates them.
  • domain assumption A single Wikipedia article on the causes of World War II is representative of the domains where MCQs are needed.
    The paper itself notes topic bias in Section 5; the ranking may not generalize to other subjects.
  • domain assumption The random selection of seven MCQs per model yields a representative sample of each model's output for the given prompt.
    Generation is stochastic and unseeded; a single draw of seven questions may not represent the full distribution.
  • domain assumption A significant Friedman test on a metric implies GPT-3.5 is the best model on that metric.
    The Friedman test only detects whether at least one model differs; the paper infers pairwise superiority from means without post-hoc tests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multiple-Choice Question Generation Using Large Language Models: Methodology and Educator Insights." pith.science (2026). https://pith.science/paper/ZMFQTGAP

@misc{pith2026250604851,
  author       = {Pith},
  title        = {Pith review of: Multiple-Choice Question Generation Using Large Language Models: Methodology and Educator Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMFQTGAP}},
  note         = {Machine review of arXiv:2506.04851}
}
read the original abstract

Integrating Artificial Intelligence (AI) in educational settings has brought new learning approaches, transforming the practices of both students and educators. Among the various technologies driving this transformation, Large Language Models (LLMs) have emerged as powerful tools for creating educational materials and question answering, but there are still space for new applications. Educators commonly use Multiple-Choice Questions (MCQs) to assess student knowledge, but manually generating these questions is resource-intensive and requires significant time and cognitive effort. In our opinion, LLMs offer a promising solution to these challenges. This paper presents a novel comparative analysis of three widely known LLMs - Llama 2, Mistral, and GPT-3.5 - to explore their potential for creating informative and challenging MCQs. In our approach, we do not rely on the knowledge of the LLM, but we inject the knowledge into the prompt to contrast the hallucinations, giving the educators control over the test's source text, too. Our experiment involving 21 educators shows that GPT-3.5 generates the most effective MCQs across several known metrics. Additionally, it shows that there is still some reluctance to adopt AI in the educational field. This study sheds light on the potential of LLMs to generate MCQs and improve the educational experience, providing valuable insights for the future.

Figures

Figures reproduced from arXiv: 2506.04851 by the authors.

Figure 1
Figure 1. Distribution of the responses for each LLM and metric. The small white triangles indicate [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Distribution of the responses regarding the possible adoption of automatic MCQ gener [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 26 canonical work pages

  1. [1]

    A taxonomy for learning, teaching, and assessing: A revision of bloom's taxonomy of educational objectives: complete edition, 2001

    Lorin W Anderson and David R Krathwohl. A taxonomy for learning, teaching, and assessing: A revision of bloom's taxonomy of educational objectives: complete edition, 2001

    work page 2001

  2. [2]

    Generating questions and multiple-choice answers using semantic analysis of texts

    Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, and Teruko Mitamura. Generating questions and multiple-choice answers using semantic analysis of texts. In Yuji Matsumoto and Rashmi Prasad, editors, Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pages 11...

    work page 2016

  3. [3]

    Personality-based recommendation in e-commerce

    Ciro Bologna, Anna Chiara De Rosa, Alfonso De Vivo, Matteo Gaeta, Giuseppe Sansonetti, and Valeria Viserta. Personality-based recommendation in e-commerce. In CEUR Workshop Proceedings , volume 997, Aachen, Germany, 2013. CEUR-WS.org

    work page 2013

  4. [4]

    Scalable educational question generation with pre-trained language models

    Sahan Bulathwela, Hamze Muse, and Emine Yilmaz. Scalable educational question generation with pre-trained language models. In Ning Wang, Genaro Rebolledo-Mendez, Noboru Matsuda, Olga C. Santos, and Vania Dimitrova, editors, Artificial Intelligence in Education , pages 327--339, Cham, 2023. Springer Nature Switzerland

    work page 2023

  5. [5]

    Toward personalized xai: A case study in intelligent tutoring systems

    Cristina Conati, Oswald Barral, Vanessa Putnam, and Lea Rieger. Toward personalized xai: A case study in intelligent tutoring systems. Artificial intelligence , 298:103503, 2021

    work page 2021

  6. [6]

    A preliminary inquiry into using corpus word frequency data in the automatic generation of english language cloze tests

    David Coniam. A preliminary inquiry into using corpus word frequency data in the automatic generation of english language cloze tests. CALICO Journal , 14(2/4):15--33, 1997

    work page 1997

  7. [7]

    Automatic multiple choice question generation from text: A survey

    Dhawaleswar Rao Ch and Sujan Kumar Saha. Automatic multiple choice question generation from text: A survey. IEEE Transactions on Learning Technologies , 13(1):14--25, 2020

    work page 2020

  8. [8]

    A social cultural recommender based on linked open data

    Alessio De Angelis, Fabio Gasparetti, Alessandro Micarelli, and Giuseppe Sansonetti. A social cultural recommender based on linked open data. In Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization , UMAP '17, pages 329--332, New York, NY, USA, 2017. ACM

    work page 2017

Show all 27 references
  1. [9]

    Sabina Elkins, Ekaterina Kochmar, Jackie C. K. Cheung, and Iulian Serban. How useful are educational questions generated by large language models?, 2023

    2023

  2. [10]

    The meta4rs proposal: Museum emotion and tracking analysis for recommender systems

    Alessio Ferrato, Carla Limongelli, Mauro Mezzini, and Giuseppe Sansonetti. The meta4rs proposal: Museum emotion and tracking analysis for recommender systems. In Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization , UMAP '22 Adjunct,...

    2022

  3. [11]

    Using deep learning for collecting data about museum visitor behavior

    Alessio Ferrato, Carla Limongelli, Mauro Mezzini, and Giuseppe Sansonetti. Using deep learning for collecting data about museum visitor behavior. Applied Sciences , 12(2), 2022

    2022

  4. [12]

    Optimising moodle quizzes for online assessments

    Sithara HPW Gamage, Jennifer R Ayres, Monica B Behrend, and Elizabeth J Smith. Optimising moodle quizzes for online assessments. International journal of STEM education , 6:1--14, 2019

    2019

  5. [13]

    Shaping the future of education: exploring the potential and consequences of ai and chatgpt in educational settings

    Simone Grassini. Shaping the future of education: exploring the potential and consequences of ai and chatgpt in educational settings. Education Sciences , 13(7):692, 2023

    2023

  6. [14]

    A novel approach to generate distractors for multiple choice questions

    Archana Praveen Kumar, Ashalatha Nayak, Manjula Shenoy K., Shashank Goyal, and Chaitanya. A novel approach to generate distractors for multiple choice questions. Expert Systems with Applications , 225:120022, 2023

    2023

  7. [15]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  8. [16]

    Multiple choice question generation using bert xl net, 2023

    S Adi Lakshmi, Rajesh Saturi, Anupriya Bharti, Meghana Avvari, and Battu Bhavana. Multiple choice question generation using bert xl net, 2023

    2023

  9. [17]

    Automatic computer science domain multiple-choice questions generation based on informative sentences

    Farah Maheen, Muhammad Asif, Haseeb Ahmad, Shahbaz Ahmad, Fahad Alturise, Othman Asiry, and Yazeed Yasin Ghadi. Automatic computer science domain multiple-choice questions generation based on informative sentences. PeerJ Computer Science , 8:e1010, 2022

    2022

  10. [18]

    A computer-aided environment for generating multiple-choice test items

    Ruslan Miktov, Le An Ha, and Nikiforos Karamanis. A computer-aided environment for generating multiple-choice test items. Natural Language Engineering , 12(2):177–194, 2006

    2006

  11. [19]

    Knowledge injection to counter large language model (llm) hallucination

    Ariana Martino, Michael Iannelli, and Coleen Truong. Knowledge injection to counter large language model (llm) hallucination. In Catia Pesquita, Hala Skaf-Molli, Vasilis Efthymiou, Sabrina Kirrane, Axel Ngonga, Diego Collarana, Renato Cerqueira, Mehwish Alam, Cassia Trojahn, a...

    2023

  12. [20]

    A system for generating multiple choice questions: With a novel approach for sentence selection

    Mukta Majumder and Sujan Kumar Saha. A system for generating multiple choice questions: With a novel approach for sentence selection. In Hsin-Hsi Chen, Yuen-Hsien Tseng, Yuji Matsumoto, and Lung Hsiang Wong, editors, Proceedings of the 2nd Workshop on Natural Language Processi...

    2015

  13. [21]

    Evalquiz – llm-based automated generation of self-assessment quizzes in software engineering education

    Niklas Meißner, Sandro Speth, Julian Kieslinger, and Steffen Becker. Evalquiz – llm-based automated generation of self-assessment quizzes in software engineering education. In Software Engineering im Unterricht der Hochschulen 2024 , pages 53--64. Gesellschaft für Informatik e...

    2024

  14. [22]

    Leveraging large language models for multiple choice question answering, 2022

    Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering, 2022

    2022

  15. [23]

    Cross-domain recommendation for enhancing cultural heritage experience

    Giuseppe Sansonetti, Fabio Gasparetti, and Alessandro Micarelli. Cross-domain recommendation for enhancing cultural heritage experience. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization , pages 413--415, New York, NY, USA, 2019. ACM

    2019

  16. [24]

    Harnessing multi-role capabilities of large language models for open-domain question answering

    Hongda Sun, Yuxuan Liu, Chengwei Wu, Haiyu Yan, Cheng Tai, Xin Gao, Shuo Shang, and Rui Yan. Harnessing multi-role capabilities of large language models for open-domain question answering. In Proceedings of the ACM on Web Conference 2024 , WWW '24, page 4372–4382, New York, NY...

    2024

  17. [25]

    Chatgpt: Challenges, opportunities, and implications for teacher education

    Jeromie Whalen, Chrystalla Mouza, et al. Chatgpt: Challenges, opportunities, and implications for teacher education. Contemporary Issues in Technology and Teacher Education , 23(1):1--23, 2023

    2023

  18. [26]

    Siren's song in the ai ocean: A survey on hallucination in large language models, 2023

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models, 2023

    2023

  19. [27]

    Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts

    JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , pages 1--21, New York, NY, USA, 2023. Ass...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.