{"id":"d47f323a-8b7b-4bc7-9785-9d49882f1886","arxiv_id":"2501.13125","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A distractor generator trained via pairwise ranking and direct preference optimization produces wrong options that students select more often than options from GPT-3.5, GPT-4o, and other baselines, and yields a higher item discrimination index in a small human test.","lead":"This paper builds a model that writes wrong answer choices for multiple-choice questions, aiming for choices that real students would actually select. The method trains a ranker on student answer data and then uses that ranker to guide a generator, which could make computer science tests more effective at exposing misconceptions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ranker is only validated on human-authored distractors, never on the novel distractors it is used to label and evaluate; the 15-student human study is too underpowered to independently confirm the DI claim. Re-analyzing the existing 40-question human data would directly test this.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the pipeline assumes the pairwise ranker's judgments on novel, never-answered distractors are a valid proxy for real student selection rates. The ranker is trained and evaluated on original distractors with known selection rates, but it is then used to label GPT-4o-generated distractors and to evaluate the generator's outputs. This is a genuine circularity, and the paper's only independent evidence, a 15-student human study, is too small to resolve it. I add two concrete observations. First, the paper already has the data needed to test the proxy assumption: the 40-question human evaluation includes selection counts for model-generated distractors, yet the authors do not report ranker accuracy on those distractors. Second, the DI comparison is statistically fragile: with 15 students and a 27% split, the upper and lower groups have 4 students each, so per-item DI values are coarse and the DPO/SFT difference of 0.018 is likely within noise. The paper deserves credit for a transparent pipeline, a public code/dataset subset, real human evaluation, and ablations showing DPO helps over SFT. But the central claim, that DPO training improves true student-selection plausibility, remains conditional on validating the ranker on novel distractors. The reader's CONDITIONAL verdict is appropriate; no verdict change is needed, but the proposed re-analysis of existing human data should be a precondition for acceptance.","tokens_in":23169,"tokens_out":6922,"duration_ms":75306,"concrete_test":"Re-analyze the existing 40-question human evaluation data: for each question, use the observed selection counts of the four model-generated distractors as ground truth and have the 'DPO, Comb.' pairwise ranker rank all six pairs among them. Compute rank accuracy against these observed student choices, with a permutation or bootstrap test for whether it exceeds chance. If accuracy on these novel distractors is not significantly above 50%, the ranker is not a valid proxy for student behavior on generated distractors, undermining both the DPO label construction in §3.3 and the automated plausibility comparisons in Table 4. Also report student-level bootstrap confidence intervals for the DI values in Table 5; if the DPO vs. SFT DI interval includes zero, the headline DI claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the pairwise ranker being a valid proxy for real student choices on the synthetic distractors it ranks. The ranker's 67.5% accuracy is measured only on original human-authored distractors with known selection rates, not on the GPT-4o-generated distractors added in §3.3 or the model-generated distractors evaluated in §4.3. Because the same DPO-trained ranker both constructs the chosen/rejected pairs for DPO and scores the generator's outputs in Table 4, the automated win/lose results are circular unless the ranker transfers to novel distractors. The paper's independent check, 15 students answering 40 questions, is too underpowered: with 27% extremes of 15, DI is based on groups of 4 students, and the reported DPO vs. SFT DI difference (0.212 vs. 0.194) is within sampling noise. The authors already possess human selection counts for the four model-generated distractors per question from this study, but they never compare ranker predictions against these counts. Without that comparison, the automated plausibility metric remains self-confirming and the 'higher DI' claim is not statistically supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a pipeline for generating plausible multiple-choice distractors. A pairwise ranker is trained on distractor pairs with known student selection rates; the same ranker is used to label new GPT-4o-generated distractors, forming a student choice dataset; a distractor generator is trained via SFT and DPO on this dataset. The authors report ranker accuracy of 67.5% versus 71.7% for human experts, win/lose plausibility counts against several baselines, and a discrimination index of 0.212 from a 15-student human study. The main claim is that the DPO-trained generator produces more plausible distractors and questions with higher DI.","tokens_in":23416,"tokens_out":6278,"duration_ms":58870,"significance":"If the central claim held, the pipeline would be a practical contribution to automated assessment, with interpretable reasoning. The paper deserves credit for releasing code and a data subset, running ablations on data augmentation and pairing methods, reporting generalization experiments on public datasets, and including expert evaluations of clarity and answerability. However, the evaluation loop—the same ranker that creates the training preferences also scores the final outputs—makes the automated plausibility results self-confirming unless the ranker transfers to novel distractors, and the human sample is too small to rescue the DI claim. The contribution is potentially useful but the evidence as presented is insufficient.","major_comments":[{"comment":"The evaluation of the distractor generator is circular. The pairwise ranker is used in §3.3 to rank the GPT-4o-augmented distractors and thereby construct the chosen/rejected pairs for DPO, and the same 'DPO, Comb.' ranker is used in §4.3 to score the generator's outputs in Table 4. Since the ranker's judgments on these synthetic distractors are never checked against real student selections, the win/lose counts partly measure how well the generator reproduces the ranker's own preferences. Concretely, the authors already possess human selection counts for the four model-generated distractors per question from the 40-question study; computing the ranker's accuracy on exactly those distractors would break the circularity. Please add this validation or otherwise supply out-of-sample evidence for the ranker on novel distractors.","section":"Sections 3.3-4.3, Table 4"},{"comment":"The ranker's reported accuracy of 67.5% (Table 3) is measured only on pairs of original human-authored distractors with known selection rates. The student choice dataset built in §3.3 introduces GPT-4o-generated distractors and pairs them with original ones, with all such rankings assigned by the same ranker. Nothing in the paper demonstrates that the ranker's ability transfers to this new distribution; if it does not, the DPO training labels are systematically mislabeled. A direct test is to evaluate the ranker on the pairs that include at least one synthetic distractor using the human data from the 40-question test, and to report accuracy separately for original-original, original-synthetic, and synthetic-synthetic pairs.","section":"Section 5.1, Table 3"},{"comment":"The item-discrimination claim rests on 15 students. Under the Top 50%/Low 50% split used in Table 5, the high and low groups contain either 7 or 8 students, and under the 27% rule stated in Appendix B.4 they contain about 4; in either case the DPO versus SFT difference in DI (0.212 vs 0.194) is within sampling noise, and no confidence intervals or significance tests are reported. The paper also contradicts itself on the group definitions: Table 5 says Top 50%/Low 50%, while Appendix B.4 says the top and bottom 27% are used. Please provide the variance of DI across students, exact group sizes, and a significance test, and reconcile the two definitions.","section":"Section 5.2, Table 5, Appendix B.4"}],"minor_comments":[{"comment":"The human accuracy of 71.7% on 60 items is reported without inter-annotator agreement or a significance test against the model's 70%, so the 'comparable to human experts' claim is not quantified.","section":"Section 5.1, human evaluation paragraph"},{"comment":"The word 'distractros' appears in the description of the GPT baselines; it should be 'distractors'.","section":"Section 4.2, baseline list"},{"comment":"The note 'averaged over two evaluations' obscures run-to-run variability; please report both sets of counts or a variance measure.","section":"Table 4"},{"comment":"The DI computation treats each distractor as a separate true/false item; clarify how this maps onto the standard DI formula and whether 'None of the above' selections are scored as correct for that item.","section":"Appendix B.4"},{"comment":"The Limitations section does not acknowledge that the automated plausibility metric is the same model that generated the training preferences; this should be addressed in the final version.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The reader's concern about circularity is well-founded and is the main barrier. The authors can address it with a re-analysis of their existing human data, which is why I recommend major revision rather than reject. I would also ask the editor to ensure the human-subject statistics are reported properly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds a clean, well-documented pipeline: train a pairwise ranker to reason about student misconceptions, use it to rank augmented distractors into a synthetic student-choice dataset, then train a generator with DPO on chosen/rejected pairs. The reasoning-augmented ranker, the synthetic data construction for CS subjects, and the move beyond math are real extensions of Scarlatos et al. (2024). The ablations are careful, the generalization experiments to English and GPT-generated CS questions are a plus, and the code and data subset are released. This is honest, competent work.\n\nThe soft spot is the evaluation loop. The ranker created the training preferences in §3.3 and is the primary plausibility metric in §4.3. That would be acceptable if the ranker were independently validated on the novel, model-generated distractors it is asked to score, but it is only validated on human-authored distractors with known selection rates. The ranker's 67.5% accuracy on human distractors says little about its accuracy on GPT-4o-generated or model-generated distractors, which may have different error patterns. The stress-test note is right that the authors already possess human selection counts for the four model-generated distractors per question from their 15-student study, and they never compare ranker predictions against those counts. That comparison is the missing piece that would break the circularity.\n\nThe human study is underpowered. Fifteen students split into top/bottom 27% groups means roughly four students per group, and the reported DI difference (0.212 vs. 0.194) is within sampling noise. No confidence intervals are reported anywhere. The expert clarity/answerability ratings are a useful sanity check, but they do not validate plausibility.\n\nI do not think the central idea is wrong. The flaws are in the evidence, not the concept. The paper deserves a serious referee and could become a solid contribution if the authors add the ranker-versus-human comparison on novel distractors, report intervals, and either enlarge the human study or soften the DI claim. I would bring this to a reading group as a case study in evaluation circularity, and I would cite it with a caveat if I worked in this area.","headline":"A promising DPO-based distractor generation pipeline whose headline results are partly self-confirming because the same ranker labels the training data and evaluates the outputs; the human study is too small to independently support the DI claim.","tokens_in":23960,"tokens_out":1520,"would_cite":true,"duration_ms":16286,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pairwise ranker and preference optimization train a model to generate distractors students actually choose, improving how well multiple-choice questions distinguish strong from weak students.","keywords":["distractor generation","multiple-choice questions","pairwise ranking","direct preference optimization","student choice prediction","item discrimination index","misconception modeling","educational NLP"],"falsifier":"Give the pairwise ranker's top-ranked and bottom-ranked distractors to a large cohort of hundreds of students in a controlled test; if selection counts invert the ranker's order, or if distractors from the DPO generator are chosen no more often than baseline distractors, the central claim would be falsified.","tokens_in":22915,"feed_emoji":"🎓","tokens_out":6285,"duration_ms":59639,"temperature":0.7,"pith_summary":"This paper claims that distractor quality in multiple-choice questions can be improved by training a generator to prefer the wrong answers students are most likely to pick. The authors build a pairwise ranker that compares two distractors and reasons about student misconceptions, then use that ranker to label a synthetic dataset of distractor preferences, and finally train a distractor generator with direct preference optimization on those labels. On computer science questions (Python, databases, machine learning), the resulting generators beat GPT-3.5 and GPT-4o and other baselines on plausibility, and their distractors produce a higher item discrimination index, meaning the questions do a better job of separating high from low performers. The paper also reports that the ranker reaches 67.5% ranking accuracy, close to the 71.7% of human experts on the same task.","feed_headline":"AI trained on student choices writes more tempting wrong answers","feed_subtitle":"A pairwise ranker plus preference optimization lifts the discrimination index above GPT-4o baselines.","key_machinery":"The load-bearing mechanism is the pairing of a pairwise ranker with direct preference optimization. The ranker takes a question, its correct answer, and two distractors, produces reasoning about the knowledge tested and why each distractor might confuse students, and outputs which distractor students would choose more often; it is trained with supervised fine-tuning on GPT-4o-generated reasoning followed by DPO. Using this ranker, the authors build a student choice dataset in which each question's original distractors plus three GPT-4o-generated distractors are ordered by plausibility, then train the generator with SFT and DPO where top-ranked distractors are chosen and bottom-ranked ones rejected. This lets a 7-billion-parameter model distill the preference signal into generation without an explicit reward model at inference time.","core_discovery":"The central claim is that a distractor generator trained with direct preference optimization on pairs ranked by a dedicated pairwise ranker produces distractors that are more plausible to students than those from larger general-purpose language models, and that these distractors improve the discrimination index of the resulting multiple-choice question. The pipeline treats plausibility as a learned, pairwise preference: the ranker is trained on real student selection rates plus reasoning generated by a large model, the student choice dataset preserves rankings among original and newly synthesized distractors, and the generator is optimized to output distractors from the top of that ranking. On evaluation, the DPO-trained generator wins more pairwise plausibility comparisons than it loses against GPT-3.5-turbo, GPT-4o, a kNN approach, and a retrieval plus chain-of-thought baseline, and its distractors score highest on human selection counts and discrimination index among all compared models.","pith_inferences":["Because the ranker produces interpretable reasoning about misconceptions, those explanations could be reused as student feedback or item-writing guidance, a use the paper does not develop.","The ranking signal could be turned into a dial: sampling distractors at different plausibility thresholds would let instructors target item difficulty to a specific class rather than always maximizing plausibility.","Regenerating the student choice dataset in other subjects or languages would test whether the misconception categories the ranker identifies are domain-general or specific to computer science."],"forward_implications":["MCQ items assembled from DPO-ranked distractors separate top and bottom performers better than items from GPT-4o, with a discrimination index of 0.212 versus 0.119 for GPT-4o.","The pairwise ranker's accuracy of 67.5% lands close to the 71.7% of human experts on the same 60-question task, indicating the ranking task is learnable and can be automated.","The training pipeline does not require new human preference labels for each question; the ranker synthesizes them, and DPO then improves generation over SFT across subjects and in out-of-domain transfer sets.","On newly generated CS questions and on English exam items, DPO continues to improve plausibility over SFT, suggesting the method transfers beyond the original platform data."],"supporting_citations":[{"why":"Supplies the overgenerate-and-rank DPO baseline and the closest prior pipeline, whose ranker is reproduced on the authors' data as a comparison.","marker":"Scarlatos et al. (2024)"},{"why":"Provides the direct preference optimization method used to train the distractor generator.","marker":"Rafailov et al. (2024)"},{"why":"Provides the kNN retrieval baseline for distractor generation with in-context examples.","marker":"Feng et al. (2024)"},{"why":"Provides the retrieval-augmented and chain-of-thought MCQ generation baseline used for plausibility comparison.","marker":"Hang et al. (2024)"},{"why":"Supplies the G-Eval prompt adapted as one of the pairwise ranker baselines.","marker":"Liu et al. (2023)"},{"why":"Provides the acceptable discrimination index range used to interpret the DI results.","marker":"Kumar et al. (2021)"},{"why":"Supplies the discrimination index formula and upper/lower group cutoff used in the human evaluation.","marker":"Mahjabeen et al. (2017)"},{"why":"Supports the item analysis methodology used to evaluate distractor efficiency and discrimination power.","marker":"Rezigalla et al. (2024)"}],"fun_headline_variants":["Wrong answers that students actually pick: AI learns from choices","DPO-trained AI crafts distractors that fool students better","Student choice data yields more plausible MCQ distractors","AI ranks distractors by student preference for better MCQs","Learning from student picks makes AI write tempting wrong answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes the pairwise ranker's judgments on distractors it has never seen match what real students would actually choose, and the generator is both trained and scored with that same ranker; only a 15-student human trial directly checks the assumption.","fun_headline_variants_meta":{"raw":{"variants":["Wrong answers that students actually pick: AI learns from choices","DPO-trained AI crafts distractors that fool students better","Student choice data yields more plausible MCQ distractors","AI ranks distractors by student preference for better MCQs","Learning from student picks makes AI write tempting wrong answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1411,"prompt_tokens":899,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":515,"tokens_out":512,"duration_ms":5023,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:36:09.087440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the pairwise ranker's top-ranked and bottom-ranked distractors to a large cohort of hundreds of students in a controlled test; if selection counts invert the ranker's order, or if distractors from the DPO generator are chosen no more often than baseline distractors, the central claim would be falsified.","supporting_citations":[],"review_version":1}