Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction

Suin Kim; Yohan Jo; Yooseop Lee

REVIEW 3 major objections 5 minor 13 references

Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pairwise ranker and preference optimization train a model to generate distractors students actually choose, improving how well multiple-choice questions distinguish strong from weak students.

desk verdict A promising DPO-based distractor generation pipeline whose headline results are partly self-confirming because the same ranker labels the training data and evaluates the outputs; the human study is too small to independently support the DI claim. read the letter →

arxiv 2501.13125 v3 pith:GGXBZLEH submitted 2025-01-21 cs.CL cs.AIcs.LG

Yooseop Lee , Suin Kim , Yohan Jo This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords distractorgenerationmultiple-choicequestionspairwiserankingdirectpreferenceoptimizationstudentchoicepredictionitemdiscriminationindexmisconceptionmodelingeducationalNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that distractor quality in multiple-choice questions can be improved by training a generator to prefer the wrong answers students are most likely to pick. The authors build a pairwise ranker that compares two distractors and reasons about student misconceptions, then use that ranker to label a synthetic dataset of distractor preferences, and finally train a distractor generator with direct preference optimization on those labels. On computer science questions (Python, databases, machine learning), the resulting generators beat GPT-3.5 and GPT-4o and other baselines on plausibility, and their distractors produce a higher item discrimination index, meaning the questions do a better job of separating high from low performers. The paper also reports that the ranker reaches 67.5% ranking accuracy, close to the 71.7% of human experts on the same task.

What carries the argument

The load-bearing mechanism is the pairing of a pairwise ranker with direct preference optimization. The ranker takes a question, its correct answer, and two distractors, produces reasoning about the knowledge tested and why each distractor might confuse students, and outputs which distractor students would choose more often; it is trained with supervised fine-tuning on GPT-4o-generated reasoning followed by DPO. Using this ranker, the authors build a student choice dataset in which each question's original distractors plus three GPT-4o-generated distractors are ordered by plausibility, then train the generator with SFT and DPO where top-ranked distractors are chosen and bottom-ranked ones rejected. This lets a 7-billion-parameter model distill the preference signal into generation without an explicit reward model at inference time.

What would settle it

Give the pairwise ranker's top-ranked and bottom-ranked distractors to a large cohort of hundreds of students in a controlled test; if selection counts invert the ranker's order, or if distractors from the DPO generator are chosen no more often than baseline distractors, the central claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that a distractor generator trained with direct preference optimization on pairs ranked by a dedicated pairwise ranker produces distractors that are more plausible to students than those from larger general-purpose language models, and that these distractors improve the discrimination index of the resulting multiple-choice question. The pipeline treats plausibility as a learned, pairwise preference: the ranker is trained on real student selection rates plus reasoning generated by a large model, the student choice dataset preserves rankings among original and newly synthesized distractors, and the generator is optimized to output distractors from the top of that ranking. On evaluation, the DPO-trained generator wins more pairwise plausibility comparisons than it loses against GPT-3.5-turbo, GPT-4o, a kNN approach, and a retrieval plus chain-of-thought baseline, and its distractors score highest on human selection counts and discrimination index among all compared models.

Load-bearing premise

The whole pipeline assumes the pairwise ranker's judgments on distractors it has never seen match what real students would actually choose, and the generator is both trained and scored with that same ranker; only a 15-student human trial directly checks the assumption.

Editorial extensions

If this is right

MCQ items assembled from DPO-ranked distractors separate top and bottom performers better than items from GPT-4o, with a discrimination index of 0.212 versus 0.119 for GPT-4o.
The pairwise ranker's accuracy of 67.5% lands close to the 71.7% of human experts on the same 60-question task, indicating the ranking task is learnable and can be automated.
The training pipeline does not require new human preference labels for each question; the ranker synthesizes them, and DPO then improves generation over SFT across subjects and in out-of-domain transfer sets.
On newly generated CS questions and on English exam items, DPO continues to improve plausibility over SFT, suggesting the method transfers beyond the original platform data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the ranker produces interpretable reasoning about misconceptions, those explanations could be reused as student feedback or item-writing guidance, a use the paper does not develop.
The ranking signal could be turned into a dial: sampling distractors at different plausibility thresholds would let instructors target item difficulty to a specific class rather than always maximizing plausibility.
Regenerating the student choice dataset in other subjects or languages would test whether the misconception categories the ranker identifies are domain-general or specific to computer science.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper builds a clean, well-documented pipeline: train a pairwise ranker to reason about student misconceptions, use it to rank augmented distractors into a synthetic student-choice dataset, then train a generator with DPO on chosen/rejected pairs. The reasoning-augmented ranker, the synthetic data construction for CS subjects, and the move beyond math are real extensions of Scarlatos et al. (2024). The ablations are careful, the generalization experiments to English and GPT-generated CS questions are a plus, and the code and data subset are released. This is honest, competent work.

The soft spot is the evaluation loop. The ranker created the training preferences in §3.3 and is the primary plausibility metric in §4.3. That would be acceptable if the ranker were independently validated on the novel, model-generated distractors it is asked to score, but it is only validated on human-authored distractors with known selection rates. The ranker's 67.5% accuracy on human distractors says little about its accuracy on GPT-4o-generated or model-generated distractors, which may have different error patterns. The stress-test note is right that the authors already possess human selection counts for the four model-generated distractors per question from their 15-student study, and they never compare ranker predictions against those counts. That comparison is the missing piece that would break the circularity.

The human study is underpowered. Fifteen students split into top/bottom 27% groups means roughly four students per group, and the reported DI difference (0.212 vs. 0.194) is within sampling noise. No confidence intervals are reported anywhere. The expert clarity/answerability ratings are a useful sanity check, but they do not validate plausibility.

I do not think the central idea is wrong. The flaws are in the evidence, not the concept. The paper deserves a serious referee and could become a solid contribution if the authors add the ranker-versus-human comparison on novel distractors, report intervals, and either enlarge the human study or soften the DI claim. I would bring this to a reading group as a case study in evaluation circularity, and I would cite it with a caveat if I worked in this area.

Referee Report

3 major / 5 minor

Summary. The paper presents a pipeline for generating plausible multiple-choice distractors. A pairwise ranker is trained on distractor pairs with known student selection rates; the same ranker is used to label new GPT-4o-generated distractors, forming a student choice dataset; a distractor generator is trained via SFT and DPO on this dataset. The authors report ranker accuracy of 67.5% versus 71.7% for human experts, win/lose plausibility counts against several baselines, and a discrimination index of 0.212 from a 15-student human study. The main claim is that the DPO-trained generator produces more plausible distractors and questions with higher DI.

Significance. If the central claim held, the pipeline would be a practical contribution to automated assessment, with interpretable reasoning. The paper deserves credit for releasing code and a data subset, running ablations on data augmentation and pairing methods, reporting generalization experiments on public datasets, and including expert evaluations of clarity and answerability. However, the evaluation loop—the same ranker that creates the training preferences also scores the final outputs—makes the automated plausibility results self-confirming unless the ranker transfers to novel distractors, and the human sample is too small to rescue the DI claim. The contribution is potentially useful but the evidence as presented is insufficient.

major comments (3)

[Sections 3.3-4.3, Table 4] The evaluation of the distractor generator is circular. The pairwise ranker is used in §3.3 to rank the GPT-4o-augmented distractors and thereby construct the chosen/rejected pairs for DPO, and the same 'DPO, Comb.' ranker is used in §4.3 to score the generator's outputs in Table 4. Since the ranker's judgments on these synthetic distractors are never checked against real student selections, the win/lose counts partly measure how well the generator reproduces the ranker's own preferences. Concretely, the authors already possess human selection counts for the four model-generated distractors per question from the 40-question study; computing the ranker's accuracy on exactly those distractors would break the circularity. Please add this validation or otherwise supply out-of-sample evidence for the ranker on novel distractors.
[Section 5.1, Table 3] The ranker's reported accuracy of 67.5% (Table 3) is measured only on pairs of original human-authored distractors with known selection rates. The student choice dataset built in §3.3 introduces GPT-4o-generated distractors and pairs them with original ones, with all such rankings assigned by the same ranker. Nothing in the paper demonstrates that the ranker's ability transfers to this new distribution; if it does not, the DPO training labels are systematically mislabeled. A direct test is to evaluate the ranker on the pairs that include at least one synthetic distractor using the human data from the 40-question test, and to report accuracy separately for original-original, original-synthetic, and synthetic-synthetic pairs.
[Section 5.2, Table 5, Appendix B.4] The item-discrimination claim rests on 15 students. Under the Top 50%/Low 50% split used in Table 5, the high and low groups contain either 7 or 8 students, and under the 27% rule stated in Appendix B.4 they contain about 4; in either case the DPO versus SFT difference in DI (0.212 vs 0.194) is within sampling noise, and no confidence intervals or significance tests are reported. The paper also contradicts itself on the group definitions: Table 5 says Top 50%/Low 50%, while Appendix B.4 says the top and bottom 27% are used. Please provide the variance of DI across students, exact group sizes, and a significance test, and reconcile the two definitions.

minor comments (5)

[Section 5.1, human evaluation paragraph] The human accuracy of 71.7% on 60 items is reported without inter-annotator agreement or a significance test against the model's 70%, so the 'comparable to human experts' claim is not quantified.
[Section 4.2, baseline list] The word 'distractros' appears in the description of the GPT baselines; it should be 'distractors'.
[Table 4] The note 'averaged over two evaluations' obscures run-to-run variability; please report both sets of counts or a variance measure.
[Appendix B.4] The DI computation treats each distractor as a separate true/false item; clarify how this maps onto the standard DI formula and whether 'None of the above' selections are scored as correct for that item.
[Limitations] The Limitations section does not acknowledge that the automated plausibility metric is the same model that generated the training preferences; this should be addressed in the final version.

Circularity Check

2 steps flagged · score 6.0 of 10

Automated plausibility evaluation is a closed loop: the same pairwise ranker that creates the DPO training preferences in §3.3–3.4 is used as the plausibility metric in §4.3, so the headline win/lose and DI claims rest substantially on a self-confirming proxy.

fitted input called prediction [Section 3.3 (Student Choice Dataset)]
"These new distractors, along with the original ones, are scored using the pairwise ranker. At this stage, the relative rankings of the original distractors are preserved, while rankings between the original and new distractors, as well as among the new distractors, are determined by our pairwise ranker."

The pairwise ranker was trained only on human-authored distractors with observed student selection rates. For the GPT-4o-added distractors introduced in §3.3, no student choices exist; the ranker's outputs are the only labels. These ranker-generated rankings are then treated as ground-truth preferences for SFT and DPO in §3.4. Any systematic bias in the ranker's extrapolation to novel distractors is therefore baked into the generator as training signal, not corrected by student data. Calling this dataset a 'student choice dataset' does not change the fact that the new-distractor ranks are model predictions, not observed choices.
other [Section 4.3 (Plausibility metric) vs Section 3.4 (DPO objective)]
"We compare the plausibility of distractors generated by our model, GPT models, a kNN approach (Feng et al., 2024), a CoT prompting approach (Hang et al., 2024), and human experts (from the base MCQ dataset) as measured by our pairwise ranker ('DPO, Comb.' in Table 3)."

This metric is the same model that produced the chosen/rejected DPO pairs in §3.4: 'we construct all possible pairs between the top-n distractors and the bottom-n distractors, labeling the distractor from the top-n as chosen and the one from the bottom-n as rejected.' The generator is therefore trained to maximize preference according to M_Rank, and Table 4's win/lose counts are a direct readout of that same preference function. The automated plausibility comparison is thus partly self-confirming: it measures fit to the ranker, not to student selection behavior, unless the ranker is independently shown to transfer to novel, model-generated distractors.

full rationale

The automated evaluation loop is the main circular element. The pairwise ranker is legitimately trained on observed selection rates for original distractors, so the ranker itself has independent grounding. However, §3.3 extends that ranker to GPT-4o-generated distractors with no student selection data, and §3.4 uses those ranker outputs as DPO preferences. Section 4.3 then evaluates the generator with the same ranker, so the win/lose results in Table 4 largely measure how well the generator optimizes its own training reward. This does not make the whole paper vacuous: the human study with 15 students and the expert clarity/answerability ratings provide external signal, and the ranker's 67.5% accuracy on held-out human distractors shows it captures real patterns. But the human study is too small (15 students, 40 questions, DI groups of roughly 4 students) to validate the novel-distractor predictions, and the authors do not report a ranker-versus-human comparison on the model-generated distractors they already collected. The 'higher DI' and 'more plausible than baselines' claims therefore rest substantially on a self-confirming proxy. Score 6 reflects partial circularity: one central evaluation reduces to the training signal, while independent but underpowered checks prevent a higher score.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

All free parameters are experimental design choices rather than theoretical constants; none is derived from first principles. The three axioms are empirical assumptions about the proprietary dataset, GPT-4o as teacher, and the ranker as oracle for unseen distractors. The third is the most consequential because it creates the circular evaluation loop. The paper introduces no new physical or mathematical entities.

free parameters (5)

Number of GPT-4o augmented distractors per question = 3
Used to build the student choice dataset in Section 3.3; the choice is not varied or justified.
DPO chosen-rejected pairing = top-n versus bottom-n distractors
Preference pairs for generator training are defined by this split in Section 3.4; an alternative sliding-window pairing gave no significant difference in Appendix B.2.
Ranker positional-bias mitigation = temperature 0.5, up to 10 attempts
Used in Section 4.2 to reduce AB/BA order bias; chosen by the authors without systematic tuning.
SFT and DPO training hyperparameters = various learning rates and epochs
Listed in Appendices A.2 and B.2; selected to avoid overfitting, but no hyperparameter search is reported.
DI group split = top and bottom 27% of 15 students
Used to compute the headline DI of 0.212 in Section 5.2 and Appendix B.4; groups of about 4 students make the estimate fragile.

assumptions (3)

domain assumption Selection rates from the proprietary Korean learning platform are reliable indicators of distractor plausibility for the target student population.
Invoked in Section 3.1: the selection rate information is considered reliable because each question was solved by hundreds of students from diverse sectors. No demographic or distributional validation is provided.
domain assumption GPT-4o can generate valid, diverse new distractors and can produce reasoning that accurately explains why students choose a distractor.
GPT-4o generates the augmented distractors in Section 3.3 and all SFT reasoning for the ranker in Section 3.2. The paper does not independently verify that GPT-4o rationales match real student cognitive processes.
ad hoc to paper The pairwise ranker's scores on synthetic distractors, never answered by real students, are a valid proxy for true student selection rates.
Section 3.3 uses the ranker to rank original and new distractors, and Section 4.3 uses the same ranker as the primary plausibility metric for the generator. This assumption is load-bearing and creates the circular evaluation loop.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction." pith.science (2026). https://pith.science/paper/GGXBZLEH

@misc{pith2026250113125,
  author       = {Pith},
  title        = {Pith review of: Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGXBZLEH}},
  note         = {Machine review of arXiv:2501.13125}
}

read the original abstract

In designing multiple-choice questions (MCQs) in education, creating plausible distractors is crucial for identifying students' misconceptions and gaps in knowledge and accurately assessing their understanding. However, prior studies on distractor generation have not paid sufficient attention to enhancing the difficulty of distractors, resulting in reduced effectiveness of MCQs. This study presents a pipeline for training a model to generate distractors that are more likely to be selected by students. First, we train a pairwise ranker to reason about students' misconceptions and assess the relative plausibility of two distractors. Using this model, we create a dataset of pairwise distractor ranks and then train a distractor generator via Direct Preference Optimization (DPO) to generate more plausible distractors. Experiments on computer science subjects (Python, DB, MLDL) demonstrate that our pairwise ranker effectively identifies students' potential misunderstandings and achieves ranking accuracy comparable to human experts. Furthermore, our distractor generator outperforms several baselines in generating plausible distractors and produces questions with a higher item discrimination index (DI).

Figures

Figures reproduced from arXiv: 2501.13125 by the authors.

**Figure 2.** Training pipeline for the distractor generation. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Human evaluation on our pairwise ranker. The [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Plausibility factors in our pairwise ranker’s [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

[1]

Read the MCQ carefully and think about the relevant misconceptions or mistakes related to the question from your perspective as a teacher

work page
[2]

Judge how plausible and confusing the distractor would be from a student’s perspective

work page
[3]

" [B]:

Choose one distractor based on Evaluation Criteria. Output your choice as a single token, either A or B. [Question] {question} [Answer] {answer} [Distractor A] {distractor} [Distractor B] {distractor} Evaluation Form (A or B ONLY): - Choice: Table 19: Instruction prompt (G-Eval) for pairwise ranker. 19 Pairwise Ranker Prompt (Discussion) <Prompt - Student...

work page 2024
[4]

Choose the incorrect statement

Direct preference optimization: Your language model is secretly a reward model. Advances in Neu- ral Information Processing Systems, 36. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International...

arXiv 2024
[9]

Incorrect knowledge

Distractors: - The distractor should be well-formatted so that it fits naturally when presented together with the question and answer. - If the distractor type is "Incorrect knowledge", the distractor must be an actually incorrect statement; if the distractor type is "Correct knowledge", the distractor must be an actually correct statement. [Question] {qu...

work page 2024
[10]

type" as

Check whether the question asks for a ‘correct’ or ‘incorrect’ option. If the question asks for a correct option, label "type" as "asking correct option." If the question asks for an incorrect option, label "type" as "asking incorrect option."

work page
[11]

Insert the given option into the question and analyze whether it is the correct choice

work page
[12]

valid." If it is not the correct answer, label it as

Based on the analysis, if the option is the correct answer to the question, label it as "valid." If it is not the correct answer, label it as "invalid." [Question] {question} [Option] {distractor} Output according to the following JSON format: {{ "type": "asking correct option" or "asking incorrect option", "analysis": "your analysis in one sentence", "va...

work page

Show all 13 references

[13]

Incorrect knowledge

Distractor type: - Analyze whether the question asks for a ‘correct’ or ‘incorrect’ option. - If the question asks for a correct option, the distractor type should be "Incorrect knowledge"; if it asks for an incorrect option, the distractor type should be "Correct knowledge"
[14]

Incorrect knowledge

Distractors: - The distractor should be well-formatted so that it fits naturally when presented together with the question and answer. - If the distractor type is "Incorrect knowledge", the distractor must be an actually incorrect statement; if the distractor type is "Correct ...
[2017]

An- nals of PIMS-Shaheed Zulfiqar Ali Bhutto Medical University, 13(4):310–315

Difficulty index, discrimination index and distractor efficiency in multiple choice questions. An- nals of PIMS-Shaheed Zulfiqar Ali Bhutto Medical University, 13(4):310–315. Hyeongdon Moon, Yoonseok Yang, Hangyeol Yu, Se- unghyun Lee, Myeongho Jeong, Juneyoung Park, Jamin Shi...

2022
[2020]

Preprint, arXiv:2010.09598

Better distractions: Transformer-based distrac- tor generation and multiple choice question filtering. Preprint, arXiv:2010.09598. Zhaopeng Qiu, Xian Wu, and Wei Fan. 2020. Automatic distractor generation for multiple choice questions in standard tests. In Proceedings of the 2...

2010 arXiv
[2024]

IEEE Access, 12:102261–102273

Mcqgen: A large language model-driven mcq generator for personalized learning. IEEE Access, 12:102261–102273. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models...

2021 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Read the MCQ carefully and think about the relevant misconceptions or mistakes related to the question from your perspective as a teacher

work page

[2] [2]

Judge how plausible and confusing the distractor would be from a student’s perspective

work page

[3] [3]

" [B]:

Choose one distractor based on Evaluation Criteria. Output your choice as a single token, either A or B. [Question] {question} [Answer] {answer} [Distractor A] {distractor} [Distractor B] {distractor} Evaluation Form (A or B ONLY): - Choice: Table 19: Instruction prompt (G-Eval) for pairwise ranker. 19 Pairwise Ranker Prompt (Discussion) <Prompt - Student...

work page 2024

[4] [4]

Choose the incorrect statement

Direct preference optimization: Your language model is secretly a reward model. Advances in Neu- ral Information Processing Systems, 36. Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International...

arXiv 2024

[5] [9]

Incorrect knowledge

Distractors: - The distractor should be well-formatted so that it fits naturally when presented together with the question and answer. - If the distractor type is "Incorrect knowledge", the distractor must be an actually incorrect statement; if the distractor type is "Correct knowledge", the distractor must be an actually correct statement. [Question] {qu...

work page 2024

[6] [10]

type" as

Check whether the question asks for a ‘correct’ or ‘incorrect’ option. If the question asks for a correct option, label "type" as "asking correct option." If the question asks for an incorrect option, label "type" as "asking incorrect option."

work page

[7] [11]

Insert the given option into the question and analyze whether it is the correct choice

work page

[8] [12]

valid." If it is not the correct answer, label it as

Based on the analysis, if the option is the correct answer to the question, label it as "valid." If it is not the correct answer, label it as "invalid." [Question] {question} [Option] {distractor} Output according to the following JSON format: {{ "type": "asking correct option" or "asking incorrect option", "analysis": "your analysis in one sentence", "va...

work page