Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Applying IRT to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessments

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Person-fit statistics from item response theory separate chatbot answers from human answers on multiple-choice assessments.

desk verdict Group-level separation is real and clean; the pollution analysis is confounded by re-estimated item difficulties. read the letter →

arxiv 2412.02713 v2 pith:Y5WCARAC submitted 2024-11-28 cs.AI

Alona Strugatski , Giora Alexandron This is my paper · ORCID
classification cs.AI
keywords generativeAIcheatingdetectionmultiple-choiceassessmentsitemresponsetheoryperson-fitstatisticschatbotseducationalmeasurement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that item response theory's person-fit statistics can separate chatbot-generated answers from human answers on multiple-choice tests. Working from two authentic instruments—a high-school chemistry formative test and a quantitative section of a high-stakes entrance exam—the authors show that premium versions of ChatGPT, Claude, and Gemini produce response patterns that are statistically far more aberrant than those of human test-takers. The same statistics also reveal that the three chatbots reason differently from one another. A third finding is that the separation shrinks as the share of AI responses in a dataset grows, so the method is most credible when AI cheating is still a minority behavior.

What carries the argument

The machinery is person-fit statistics built on the Guttman pattern: an ideal response vector in which a test-taker answers the $r$ easiest items correctly and the rest incorrectly. $G$ counts item-pair deviations from this pattern, $G^*$ normalizes $G$ to the range $[0,1]$, $U_3$ compares the observed vector with the reversed Guttman pattern, and $ZU_3$ standardizes $U_3$ to a unit-normal distribution. Because IRT orders items by estimated difficulty, a chatbot that solves hard items while missing easy ones inflates these statistics, marking it as aberrant relative to the human calibration sample.

What would settle it

Take a real class answer set and inject cheated answer sheets constructed by giving a chatbot only a random half of the items and letting a human answer the rest. If the PFS distributions of these blended sheets overlap those of honest students, the method fails on realistic partial cheating. A sharper version: run the same pipeline at a 10% pollution level on a third high-stakes instrument; if $ZU_3$'s separation already vanishes there, the operating range is even narrower than the paper reports.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that person-fit statistics computed under item response theory—specifically $G$, $G^*$, $U_3$, and $ZU_3$—assign significantly higher aberrant-response scores to chatbot answer sheets than to human answer sheets. In every combination of two instruments and three chatbots, the difference was significant at $p < 0.00001$, with cleanly separated density distributions for the $G$ statistic shown. It further shows the three chatbots are not interchangeable: Kruskal-Wallis tests found differences among them, though which chatbot stands out depends on the instrument. The $ZU_3$ measure loses statistical significance at a 25% pollution level, while $G$ remains significant on some comparisons, indicating that the detectability of AI responses decreases as their prevalence increases.

Load-bearing premise

The results assume that a student cheating with GenAI behaves like 20 full answer sheets produced by one chatbot prompted to output only the chosen options; real cheating that mixes human answers with AI answers, or uses varied prompts and model versions, may not produce the same misfit.

Editorial extensions

If this is right

  • Instructors using MCQ platforms could flag answer sheets whose person-fit values fall above a threshold calibrated on the class's human response distribution.
  • The approach needs no text analysis and works on already machine-graded data, making it cheap to deploy at scale.
  • Because chatbots differ from each other, detectors may need per-model calibration rather than a single AI profile.
  • Detection degrades as AI responses approach 25% of the dataset, so the method is a deterrent against minority cheating, not a cure for systematic AI-generated test-taking.
  • The same IRT framework could in principle extend beyond MCQs to any binary-scored assessment, since the statistics depend only on item difficulty order and correctness scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's clean separation partly reflects that each chatbot produced 20 full answer sheets with no human items mixed in; real cheating that blends human and AI answers would likely blur the PFS distributions, and the paper's data do not test that blend.
  • A testable extension: prompt the same models to explain their reasoning or to answer only a subset of items, to see whether person-fit separation survives more realistic cheating behavior.
  • If PFS-based screening were deployed, students who are simply weak, unusual, or have learning disabilities would risk being flagged; the paper itself names learning disabilities as a limitation, so a detector would need to rule out legitimate sources of misfit before accusing anyone.
  • The finding that chatbots differ in their reasoning profiles hints that IRT misfit could be used not only for detection but also for tracing which model generated a set of answers, for instance in forensic analysis of leaked answer sheets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes using item response theory (IRT) person-fit statistics (G, G*, U3, ZU3) to distinguish between human responses and responses generated by three chatbots (ChatGPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) on two multiple-choice instruments: a high-school chemistry formative assessment (n=931) and a psychometric quantitative test (n>4800). For each chatbot, 20 full answer sheets are generated with a fixed prompt. The paper reports highly significant Wilcoxon rank-sum differences between human and AI person-fit distributions (RQ1), significant chatbot differences via Kruskal-Wallis tests (RQ2), and a diminishing gap between human and AI PFS as the proportion of AI responses increases from 5% to 25% (RQ3). The authors conclude that PFS provide a psychometrically grounded basis for detecting AI cheating in MCQ assessments.

Significance. If the reported effects are genuine, the paper addresses a nearly unexplored and practically important problem in educational assessment. The RQ1 result is strengthened by the use of two authentic large-scale datasets, three leading chatbot models, and a standard published implementation (PerFit), with very low p-values and visually separated densities. The paper also connects the work to established psychometric theory and clearly states its limitations. However, the practical contribution depends on whether group-level differences translate into individual-level detection and whether the RQ3 trend is not an artifact of recalibration. If the calibration confound is resolved, these results would be a useful foundation for AI-cheating screening in MCQ-based settings.

major comments (4)
  1. [Section 3.3, Figure 3] The conclusion that the PFS gap diminishes with increasing pollution is confounded by re-estimating item difficulties within each pollution condition. As stated in Section 2.3, the nonparametric PFS are computed directly from the response matrix, and in the PerFit implementation item difficulties (proportion correct) are estimated from the same data. From 5% to 25% pollution, the calibration sample shrinks from 380 to 60 humans and gains 20 AI responses. Because AI response patterns differ systematically, the item-difficulty ordering drifts toward items that AI tends to answer correctly, which mechanically changes G, G*, U3, and ZU3 for both groups. The observed shrinkage may therefore be an artifact of calibration contamination rather than evidence that AI patterns become normal. Please re-run RQ3 with item difficulties fixed from the full human-only samples (or with parametric IRT and fixed item parameters) and show whether the diminishing-gap trend persists.
  2. [Section 3.1, Table 1] The manuscript reports only group-level distributional comparisons; it never evaluates per-test-taker detection performance, which is needed to support the abstract's claim that the method 'effectively highlights the differences' and can 'distinguish' human from GenAI responses. No ROC/AUC analysis, no threshold, no sensitivity/specificity, and no effect sizes (e.g., Cliff's delta or rank-biserial correlation) are provided. With AI samples of only 15-20 responses per model, the extremely small p-values do not quantify how well an individual score can be classified. Please add individual-level metrics such as AUC or misclassification rates, or explicitly restrict the claim to group-level differences.
  3. [Section 2.2.3] The simulation treats 20 full answer sheets generated by one model with a single fixed prompt as representative of real GenAI cheating. Actual cheating is likely partial: students may answer some items themselves, mix answers from different models or runs, verify outputs, or use different prompts. The paper does not test such mixed patterns, so the strong separation in Figure 1 may not transfer to realistic cheating scenarios. At minimum, the authors should acknowledge this as a boundary of the central claim and, ideally, add a partial-cheating condition.
  4. [Sections 3.1-3.3] The reported construction of the '5% pollution' datasets is inconsistent. Section 3.3 says the 5% condition combines 20 AI responses with 380 human responses (20/400 = 5%), while Section 3.2 describes RQ2 datasets with 931 or 980 humans plus 45 or 60 AI responses, which are approximately 4.6% and 5.8% pollution, respectively. Section 3.1 does not state how many humans were used in RQ1. Please specify the exact sample sizes and whether item difficulties for RQ1 were estimated on the full human sample or on the pollution-condition subsample; this is needed to reproduce the results and to interpret RQ3.
minor comments (6)
  1. [Table 1] 'Signed-rank test' should be 'rank-sum test' (Mann-Whitney) for independent human and AI groups; Section 2.2.1 uses the correct term.
  2. [Section 2.3] Please state explicitly that PerFit computes nonparametric item difficulties as sample proportions from the same response matrix; this detail is essential for interpreting RQ3.
  3. [Section 2.1] There is a duplicated 'of of' in the sentence introducing the research questions.
  4. [Reference [29]] The reference title is garbled ('itelVergelijkbaarheid van individuele testprestaties,Comparability of individual test performance') and should be cleaned up.
  5. [Figure 3] Some Z values are rendered as 'Z=,-4.76' and the caption should name the model and instrument for each panel; as printed, the panels are not fully self-explanatory.
  6. [Appendix A] In the U3 subplot the axis labels appear corrupted (e.g., '0.6 0.8' with a stray '.'); replace with clean plots.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ3's diminishing PFS gap is partly by construction because PFS are recomputed on each polluted matrix, so AI responses help define the Guttman reference against which they are judged.

  1. fitted input called prediction [Section 2.3 (Person-fit statistics) and Section 3.3 (RQ3)]
    "A non parametric PFS are calculated directly from the data set on 𝑁 examinees, and theirs scored responses to 𝐽 items, without relaying on estimating parameters for IRT model [13]. ... To create these datasets, we maintained the 20 responses collected from each GenAI agent and combined them with the appropriate number of human responses to achieve the required level of pollution (380 learners’ responses for 5%, 180 for 10%, and 60 for 25%)."

    The nonparametric PFS (G, G*, U3, ZU3) require an item-difficulty ordering of the response matrix; in the PerFit implementation used by the paper, this ordering is obtained from the same response matrix. RQ3 constructs each pollution level by recombining the fixed 20 AI responses with 380, 180, or 60 human responses and recomputes the statistics on that new matrix. The 'expected' Guttman pattern is therefore re-estimated from a mixture whose AI share is exactly the manipulated variable. As AI share grows, the item ordering drifts toward items AI tends to answer correctly, making AI responses more pattern-consistent and humans less so; the diminishing gap is partly forced by the statistic's construction rather than an independent empirical law.

full rationale

The paper's central RQ1 and RQ2 results are self-contained: the four PFS are standard statistics (Guttman; van der Flier) implemented in the public PerFit package, and they are applied to real human response data with 20 responses per chatbot. No parameter is fitted to the conclusion, and the self-citations [2,4] for metric choice are corroborated by the external Karabatsos [13] review, so they are not load-bearing. The only concerning step is RQ3: because nonparametric PFS are 'calculated directly from the data set,' the item-difficulty/Guttman reference is re-estimated on each artificially polluted matrix. Since the pollution level is defined by the composition of that same matrix, the finding that the human-AI PFS gap shrinks as pollution rises is partly a definitional consequence of the statistic's norm-referenced construction, not an independent measurement of the effect of cheating prevalence. This is partial circularity: the RQ3-based claim that the method is 'sensitive to the amount of AI cheating' is partly built into the re-calibration. RQ1 and RQ2 remain valid, but the RQ3 conclusion should be re-tested with human-only calibration or interpreted with this caveat.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities or ad hoc constants are introduced. The method uses four established person-fit statistics implemented in the PerFit R package; item difficulties are estimated from the data within the standard procedure. The load-bearing assumptions are about the validity of the Guttman/IRT model for these instruments and about the synthetic chatbot sample standing in for real cheating.

free parameters (1)
  • Item difficulty ordering = Estimated by PerFit from each combined dataset
    G, G*, U3, and ZU3 all require items sorted by difficulty; the ordering is derived from the same data used in the comparison, so the central result inherits uncertainty from this estimation, but no free constant is chosen by the authors.
assumptions (4)
  • domain assumption MCQ response data are unidimensional enough for Guttman-style person-fit statistics to be meaningful.
    Guttman pattern and item ordering in Section 2.3 assume a single dominant ability dimension; violations can make all examinees look aberrant.
  • domain assumption AI responses are generated by a different response process than humans, so deviations from human IRT patterns indicate AI provenance rather than just noise.
    Central to the method; stated in the abstract and Section 1. It is tested empirically, but not independently validated.
  • ad hoc to paper The 20 responses per chatbot with a fixed prompt represent realistic cheating behavior.
    Section 2.2.3; true cheating may be partial, mixed with human reasoning, or use different prompting.
  • domain assumption Human test-taker data are genuine and not already contaminated by AI responses.
    Section 2.2.2; if contamination exists, group differences are underestimated, not inflated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Applying IRT to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessments." pith.science (2026). https://pith.science/paper/Y5WCARAC

@misc{pith2026241202713,
  author       = {Pith},
  title        = {Pith review of: Applying IRT to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5WCARAC}},
  note         = {Machine review of arXiv:2412.02713}
}
read the original abstract

Generative AI is transforming the educational landscape, raising significant concerns about cheating. Despite the widespread use of multiple-choice questions in assessments, the detection of AI cheating in MCQ-based tests has been almost unexplored, in contrast to the focus on detecting AI-cheating on text-rich student outputs. In this paper, we propose a method based on the application of Item Response Theory to address this gap. Our approach operates on the assumption that artificial and human intelligence exhibit different response patterns, with AI cheating manifesting as deviations from the expected patterns of human responses. These deviations are modeled using Person-Fit Statistics. We demonstrate that this method effectively highlights the differences between human responses and those generated by premium versions of leading chatbots (ChatGPT, Claude, and Gemini), but that it is also sensitive to the amount of AI cheating in the data. Furthermore, we show that the chatbots differ in their reasoning profiles. Our work provides both a theoretical foundation and empirical evidence for the application of IRT to identify AI cheating in MCQ-based assessments.

Figures

Figures reproduced from arXiv: 2412.02713 by the authors.

Figure 1
Figure 1. Density plots of 𝐺 for human learners (blue) and conversational chatbots (red). between these two groups, showing its value in assessing the response source. Additional figures displaying the density distributions for 𝐺 ∗ , 𝑈 3, and 𝑍𝑈 3 can be found in appendix A. Thus, with respect to RQ1, we conclude that the response patterns of conversational chatbots differ significantly from those of human learners, as measur… view at source ↗
Figure 2
Figure 2. Results and density plots of PFS for conversational chatbots. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. PFS mean with ±1-STD range for ChatGPT with varying pollution rates for the chemistry and psychometric instruments. chemistry and psychometric instrument under the three pollution conditions. As can be seen, the difference between these PFS values for the chatbot ‘students’ and the human ones diminishes as the level of pollution increases. At pollution levels of 5% and 10%, the Wilcoxon results revealed a significan… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Density plots PFS for human learners (blue) and conversational chatbots (red). All calculated using 5% of conversational [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 30 canonical work pages

  1. [1]

    Noor Akbari. 2024. The AI Cheating Crisis: Education Needs Its Anti-Doping Movement . https://www.edweek.org/technology/opinion-the-ai- cheating-crisis-education-needs-its-anti-doping-movement/2024/02

    work page 2024

  2. [2]

    Giora Alexandron, Aviram Berg, and José A Ruipérez-Valiente. 2023. A General Purpose Anomaly-Based Method for Detecting Cheaters in Online Courses. IEEE Transactions on Learning Technologies

    work page 2023

  3. [3]

    Giora Alexandron, José A Ruipérez-Valiente, Zhongzhou Chen, Pedro J Muñoz-Merino, and David E Pritchard. 2017. Copying@Scale: Using harvesting accounts for collecting correct answers in a MOOC. Computers & Education 108 (2017), 96–114

    work page 2017

  4. [4]

    Ruipérez-Valiente, and David E

    Giora Alexandron, José A. Ruipérez-Valiente, and David E. Pritchard. 2019. Towards a General Purpose Anomaly Detection Method to Identify Cheaters in Massive Open Online Courses. In Proceedings of the 12th International Conference on Educational Data Mining . 480–483

    work page 2019

  5. [5]

    Giora Alexandron, Mary Ellen Wiltrout, Aviram Berg, and José A Ruipérez-Valiente. 2020. Assessment that matters: Balancing reliability and learner-centered pedagogy in MOOC assessment. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge . 512–517

    work page 2020

  6. [6]

    Beatriz Borges et al. 2024. Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants. arXiv:2408.11841 (2024)

    work page Pith review arXiv 2024

  7. [7]

    Debby RE Cotton, Peter A Cotton, and J Reuben Shipway. 2024. Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in education and teaching international 61, 2 (2024), 228–239

    work page 2024

  8. [8]

    Rafael Jaime De Ayala. 2013. The theory and practice of item response theory . Guilford Publications. Manuscript submitted to ACM 10 Strugatski & Alexandron

    work page 2013

Show all 32 references
  1. [9]

    Dat Duong and Benjamin D Solomon. 2024. Analysis of large-language model versus human performance for genetics questions. European Journal of Human Genetics 32, 4 (2024), 466–468

    2024

  2. [10]

    John M Felt, Ruben Castaneda, Jitske Tiemensma, and Sarah Depaoli. 2017. Using person fit statistics to detect outliers in survey research. Frontiers in psychology 8 (2017), 863

    2017

  3. [11]

    Jing Guo and Fritz Drasgow. 2010. Identifying Cheating on Unproctored Internet Tests: The Z-test and the likelihood ratio test. International Journal of Selection and Assessment 18, 4 (2010), 351–364

    2010

  4. [12]

    Louis Guttman. 1944. A basis for scaling qualitative data. American sociological review 9, 2 (1944), 139–150

    1944

  5. [13]

    George Karabatsos. 2003. Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education 16, 4 (2003), 277–298

    2003

  6. [14]

    Daniel Martin Katz and et al. 2024. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A 382, 2270 (2024), 20230254

    2024

  7. [15]

    Mohammad Khalil and Erkan Er. 2023. Will ChatGPT Get You Caught? Rethinking of Plagiarism Detection. In International Conference on Human-Computer Interaction. Springer, 475–487

    2023

  8. [16]

    Donald L McCabe et al. 2001. Cheating in academic institutions: A decade of research. Ethics &Behavior 11, 3 (2001), 219–232

    2001

  9. [17]

    Meijer and Klass Sijtsma

    Rob R. Meijer and Klass Sijtsma. 1995. Detection of Aberrant Item Score Patterns: A Review of Recent Developments. Applied Measurement in Education 8, 3 (1995), 261–272

    1995

  10. [18]

    Rob R Meijer and Klaas Sijtsma. 2001. Methodology review: Evaluating person fit. Applied psychological measurement 25, 2 (2001), 107–135

    2001

  11. [19]

    Philip Newton and Maira Xiromeriti. 2024. ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review. Assessment & Evaluation in Higher Education 49, 6 (2024), 781–798

    2024

  12. [20]

    Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2023. Ai psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science (2023), 17456916231214460

    2023

  13. [21]

    Arianna Prothero. 2024. New Data Reveal How Many Students Are Using AI to Cheat . https://www.edweek.org/technology/new-data-reveal-how- many-students-are-using-ai-to-cheat/2024/04

    2024

  14. [22]

    Asaf Salman and Giora Alexandron. 2024. The digital fingerprint of learner behavior: Empirical evidence for individuality in learning using deep learning. Computers and Education: Artificial Intelligence 7 (2024), 100322. https://doi.org/10.1016/j.caeai.2024.100322

    2024

  15. [23]

    Avinash C. Singhal. 1982. FACTORS IN STUDENTS’ DISHONESTY. Psychological Reports 51, 3 (1982), 775–780

    1982

  16. [24]

    Benjamin Sorenson and Kenneth Hanson. 2024. Identifying Generative Artificial Intelligence Chatbot Use on Multiple-Choice, General Chemistry Exams Using Rasch Analysis. Journal of Chemical Education 101, 8 (2024), 3216–3223

    2024

  17. [25]

    Teo Susnjak and Timothy R McIntosh. 2024. ChatGPT: The end of online exam integrity? Education Sciences 14, 6 (2024), 656

    2024

  18. [26]

    Jorge N Tendeiro and Rob R Meijer. 2012. A CUSUM to detect person misfit: A discussion and some alternatives for existing procedures. Applied Psychological Measurement 36, 5 (2012), 420–442

    2012

  19. [27]

    Tendeiro, Rob R

    Jorge N. Tendeiro, Rob R. Meijer, and A. Susan M. Niessen. 2016. PerFit: An R Package for Person-Fit Analysis in IRT.J. of Stat. Software 74, 5 (2016)

    2016

  20. [28]

    Henk Van der Flier. 1977. Environmental factors and deviant response patterns. Basic problems in cross cultural psychology, Swets & Seitlinger (1977)

    1977

  21. [29]

    Henk Van der Flier. 1980. itelVergelijkbaarheid van individuele testprestaties,Comparability of individual test performance . Swets en Zeitlinge

    1980

  22. [30]

    Karen D Wang, Eric Burkholder, Carl Wieman, Shima Salehi, and Nick Haber. 2024. Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving. In Frontiers in Education, Vol. 8. Frontiers Media SA, 1330486

    2024

  23. [31]

    Field M Watts, Amber J Dood, Ginger V Shultz, and Jon-Marc G Rodriguez. 2023. Comparing student and generative artificial intelligence chatbot responses to organic chemistry writing-to-learn assignments. Journal of Chemical Education 100, 10 (2023), 3806–3817

    2023

  24. [32]

    Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. 2024. Practical and ethical challenges of large language models in education: A systematic scoping review. British J of Educational Technology...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.