Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Does Multiple Choice Have a Future in the Age of Generative AI? A Posttest-only RCT

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing open-response practice with multiple-choice practice yields equivalent posttest learning outcomes while saving roughly a third of instruction time.

desk verdict The time-saving result is real, but the 'as effective' claim rests on a null p-value and a composite posttest score that never tests the paper's own transfer claim. read the letter →

arxiv 2412.10267 v1 pith:2UOJBIPI submitted 2024-12-13 cs.HC cs.AI

Danielle R. Thomas , Conrad Borchers , Sanjit Kakarla , Jionghao Lin , Shambhavi Bhushan , Boyuan Guo , Erin Gatz , Kenneth R. Koedinger This is my paper · ORCID
classification cs.HCcs.AI
keywords multiple-choicequestionsopen-responseposttest-onlyrandomizedcontrolledtrialtutortraininglearning-by-doinggenerativeAIassessmentGPT-4learningefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether multiple-choice questions (MCQs) can serve as learning tasks, not just assessment tools, now that generative AI makes open-response grading scalable. In a posttest-only randomized trial with 234 tutors and 790 lesson completions across six scenario-based advocacy lessons, the authors compared three learning-by-doing conditions: MCQ only, open-response only, and both. They found no statistically significant difference in posttest performance across conditions ($F(2,717)=0.27$, $p=.765$), but the MCQ-only condition was significantly faster, taking about 29% less time than open-response-only and 35% less than the combined condition. The authors conclude that MCQs are as effective and more efficient than open-response tasks for learning when practice time is limited, and that GPT-4o and GPT-4-turbo can grade open responses with sufficient proficiency for low-stakes assessment. This matters because, if true, large-scale training and homework could shift toward MCQs without sacrificing learning outcomes.

What carries the argument

The load-bearing mechanism is a posttest-only randomized design in which each of six scenario-based lessons embeds one of three learning-by-doing conditions: MCQ only, open-response only, or both, followed by identical instruction and a common posttest that mixes both question types. The comparison of posttest accuracy across conditions, via ANOVA with lesson, condition, and scenario order as between-subjects factors, combined with completion times from lesson log data, is what produces the equivalence-and-efficiency result. The auxiliary machinery is a prompt-engineering pipeline for LLM autograding, using few-shot examples, chain-of-thought rationale requests, temperature 0, and JSON output, which is what makes scalable open-response grading plausible.

What would settle it

Re-running the same ANOVA on the released lesson log data with tutor as a random effect or with clustered standard errors would settle it: if the condition effect becomes statistically significant, or the 29-35% time savings shrink to non-significance, the core claims of equivalent learning and time efficiency would not withstand the correction.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that practice format does not reliably change learning: posttest scores were statistically indistinguishable across MCQ-only, open-response-only, and combined conditions ($F(2,717)=0.27$, $p=.765$), despite a significant condition-by-lesson interaction ($F(10,717)=2.20$, $p=.012$) that the authors attribute mostly to random variability. The MCQ-only condition completed instruction in $M = 3.83$ minutes, versus $M = 5.38$ minutes for open-response-only and $M = 5.87$ minutes for both, yielding 29% and 35% time savings. The authors interpret this as evidence that MCQ practice transfers to open-ended posttest performance at least as well as open-response practice, and they explicitly note that this result is inconsistent with the ICAP framework's general prediction that constructive tasks (open response) should beat active tasks (MCQ with feedback). In the auxiliary LLM evaluation, GPT-4o and GPT-4-turbo, prompted with few-shot examples and chain-of-thought, achieved 71-91% accuracy on predict responses and 71-87% accuracy on explain responses, with notable exceptions such as an AUC of 0.17 for one lesson, indicating that LLM grading is usable but not yet universally reliable.

Load-bearing premise

The analysis treats each of the 790 lesson completions as an independent observation, even though the same tutor often completed several lessons, meaning that if a tutor's skill or motivation links their lessons together, the reported error bars and p-values could be misestimated.

Editorial extensions

If this is right

  • If MCQs are as effective and faster, homework and tutor-training platforms could substitute MCQ practice for open-response practice without expecting learning losses, freeing learner time for other content.
  • The transfer of MCQ practice to open-ended posttest performance supports using MCQs as learning tasks even when the final assessment is open response.
  • The null result challenges the general ICAP prediction that constructive open-response tasks outperform active MCQ tasks, at least for advocacy content, and invites theory refinement.
  • The significant condition-by-lesson interaction suggests the equivalence may not be uniform across content; two lessons showed significant pairwise contrasts, so content-treatment interactions deserve direct study.
  • GPT-4o and GPT-4-turbo autograding, with accuracy mostly in the 71-91% range, could support low-stakes open-response assessment at scale, though the low-AUC cases mark where human review remains necessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's null result is consistent with the view that MCQ selection and open-response construction both require the learner to retrieve and evaluate the same underlying rule; what differs is the response mode, not the memory retrieval, which would explain the equivalent transfer.
  • Because the time savings came from the learning-by-doing phase, and follow-up instruction was actually shortest in the combined condition, a natural testable extension is to vary the ratio of MCQ to open-response items within a fixed practice budget to locate the efficiency frontier.
  • The 3 of 18 significant pairwise contrasts, versus roughly 1 in 20 expected by chance, could be probed by pre-registering lesson-specific hypotheses about which advocacy skills rely on constructed justification.
  • The public release of log data would allow a reanalysis with tutor-level random effects, which is exactly the precision check the design needs to rule out clustering artifacts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper reports a posttest-only randomized experiment comparing three learning-by-doing conditions---multiple-choice questions only, open-response questions only, and a combination of both---embedded in six scenario-based tutor-training lessons on advocacy. With 234 tutors contributing 790 lesson completions, the authors find no significant main effect of condition on a composite posttest score (F(2,717)=0.27, p=.765), a significant condition-by-lesson interaction (F(10,717)=2.20, p=.012), and significantly shorter instruction time in the MCQ-only condition (about 29% less than open-response only and 35% less than both). The paper also evaluates GPT-4o and GPT-4-turbo as automated graders of open responses, reporting accuracy between 57% and 91% depending on lesson and question type, and argues that these models are adequate for low-stakes assessment. The authors conclude that MCQs are as effective as, and more efficient than, open-response tasks for learning when practice time is limited.

Significance. If the equivalence claim were properly supported, this would be a practically valuable result for scalable tutor training and for the broader debate about whether MCQ practice transfers to open-ended performance. The randomized design, the use of six content lessons in a less-structured domain, the time-on-task measurement, the reporting of human inter-rater reliability, and the stated commitment to sharing data, rubrics, and prompts are genuine strengths. The LLM-grading comparison is also relevant to the growing use of generative AI in low-stakes assessment. However, the central 'as effective' claim currently rests on a non-significant p-value rather than on an equivalence test or an effect-size bound, and the missing subscale analysis leaves the transfer-to-open-ended claim unsupported. These are load-bearing gaps that can be addressed with additional analyses.

major comments (4)
  1. [4.1, 5.1] The central claim that MCQs are 'as effective' as open-response practice rests on a null ANOVA result (F(2,717)=0.27, p=.765), but no equivalence test, effect-size estimate, or confidence interval for the condition differences is reported. A non-significant p-value cannot support the assertion of equivalent outcomes; report a TOST procedure or a confidence interval for the mean differences and specify the smallest effect size the design can rule out. The significant condition-by-lesson interaction (F(10,717)=2.20, p=.012) also needs to be addressed before a general equivalence claim is made.
  2. [3.4.1, 4.1, 6] The posttest score aggregates two multiple-choice and two open-response items, yet the paper reports only this composite and then concludes in Section 6 that 'MCQ practice transfers to open-ended performance.' That conclusion requires an analysis of the open-response subscale separately (and ideally the MCQ subscale as well); otherwise a format-specific trade-off (MCQ practice helping MCQ posttest items but not open-ended items) could produce the same null composite. Please add subscale analyses by condition and lesson, with the same equivalence/effect-size evidence requested for the composite.
  3. [3.1, 3.4] The 790 lesson completions come from only 234 tutors, with many tutors completing multiple lessons, but the ANOVA in Section 3.4 treats all observations as independent ('all factors being between subjects'). If tutor-level ability or motivation is correlated across lessons, the standard errors and p-values for both posttest performance and completion time are misestimated. Report the distribution of lessons per tutor, the intraclass correlation, and a mixed-effects model with a random tutor intercept (or cluster-robust standard errors) to verify the null result and the time differences.
  4. [4.3, 5.3] The claim that GPT models 'demonstrate proficiency' for low-stakes assessment is not well supported by Tables 6 and 7, which include AUC values as low as 0.17 (GPT-4-turbo, predict, Helping Students Manage Inequity) and 0.43-0.45 in several other lesson-by-question-type cells. Moreover, the human rubrics were developed by the same research team and the prompts were iteratively refined on the same responses, so the reported accuracy is a development-sample estimate. Please report the range of AUC/F1 as evidence of variability and clarify whether any holdout or cross-validation procedure was used.
minor comments (6)
  1. [Abstract, Section 6] The sample size is reported inconsistently: the abstract and Section 3.1 say 234 tutors, while Section 6 says n=235.
  2. [Figure 3 caption] The caption states 'no overall significant differences in instruction time prior to posttest were found between the conditions,' which contradicts the significant main effect of condition reported in Section 4.2 (F(2,716)=12.56, p<.001).
  3. [4.2] The text reports F(10,716)=13.46, p=.199 for the condition-by-lesson interaction on time; an F of 13.46 with those degrees of freedom would have a far smaller p-value, so this appears to be a typo (possibly F=1.346).
  4. [5.2] The first sentence is garbled: it says there was no significant interaction between instruction time and condition on learning outcomes and then says there was a significant interaction between condition and lesson; please rewrite to match Section 4.2.
  5. [4.3] The sentence saying GPT-4-turbo 'demonstrated proficiency across all lessons with poorer performance relative to the other lessons' for Helping Students Manage Inequity is self-contradictory and should be revised.
  6. [4.2] The observation that the Both condition 'took students less time than the sum of the open and MCQ conditions' is trivially true and not an informative result; consider removing or reframing it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: central claims rest on randomized posttest comparison and logged time data, not on fitted inputs or self-citation chains.

full rationale

This paper is an empirical randomized comparison, not a derivation, so the equation-level circularity patterns do not apply. The central learning-outcome claim rests on a three-condition posttest-only RCT with random assignment; the reported null ANOVA (F(2,717)=0.27, p=.765) is an independent statistical result, and the efficiency claim is a direct calculation from logged completion times (29% and 35% reductions). The posttest composite includes both MCQ and open-response items, and the paper does not report a separate open-ended subscale analysis; that is a substantive validity gap for the transfer claim, but it is an omitted analysis rather than a reduction of the conclusion to its inputs. The LLM grading component was developed iteratively on the same responses for which accuracy is reported, which is a potential overfitting concern, but it is secondary to the main learning comparison and does not enter the central claim; moreover, the main posttest analysis uses human graders. The self-citation to the authors' prior pre-post study [35] is used only as external support that the assessments can detect learning gains and is not the basis for the condition comparison. Hence no circular step meeting the quoted-reduction standard is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard statistical assumptions plus domain assumptions inherited from the group's prior lesson-development work. The main unmodeled risk is the repeated-measures structure of the data. No new entities are postulated.

free parameters (2)
  • LLM prompt exemplars and rubric wording = iteratively refined on study data
    Few-shot examples and scoring rubric text were selected and adjusted based on model outputs on the same responses later used in evaluation, creating a tuning loop that can inflate agreement metrics.
  • Posttest outcome aggregation = equal-weight mean accuracy over 2 MCQ + 2 open-response items
    The posttest score is a simple mean treating all items as equally informative; no item weighting, IRT, or reliability correction is applied, and the measure is assumed to be a valid learning indicator.
assumptions (5)
  • standard math ANOVA assumptions of normality and homogeneity of variance on log-transformed time and on posttest accuracy.
    The authors report checking residual Q-Q plots for log-time; normality of binary-derived accuracy scores is assumed without explicit diagnostics.
  • domain assumption Random assignment to condition is sufficient to achieve baseline equivalence despite the absence of a pretest.
    Section 3.4 states a posttest-only design was chosen to avoid testing effects; the causal interpretation assumes randomization balances unobserved tutor characteristics across conditions.
  • ad hoc to paper The 790 lesson completions are independent observations, with no within-tutor correlation for tutors who completed multiple lessons.
    Section 3.4 describes ANOVA with all factors between subjects; if tutor-level random effects exist, standard errors are likely underestimated. No clustering adjustment is reported.
  • domain assumption The human annotation rubrics are a valid ground truth for open-response correctness and advocacy competence.
    Section 3.2 describes binary coding by two experienced researchers; inter-rater reliability varies (kappa 0.55 to 0.93), so the ground truth itself is noisy in some lessons.
  • domain assumption The advocacy competencies and scenario-based lesson structure from prior work are valid learning targets.
    The lessons are based on prior research by the same group (refs [6,35]); the current paper does not independently validate the construct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Multiple Choice Have a Future in the Age of Generative AI? A Posttest-only RCT." pith.science (2026). https://pith.science/paper/2UOJBIPI

@misc{pith2026241210267,
  author       = {Pith},
  title        = {Pith review of: Does Multiple Choice Have a Future in the Age of Generative AI? A Posttest-only RCT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UOJBIPI}},
  note         = {Machine review of arXiv:2412.10267}
}
read the original abstract

The role of multiple-choice questions (MCQs) as effective learning tools has been debated in past research. While MCQs are widely used due to their ease in grading, open response questions are increasingly used for instruction, given advances in large language models (LLMs) for automated grading. This study evaluates MCQs effectiveness relative to open-response questions, both individually and in combination, on learning. These activities are embedded within six tutor lessons on advocacy. Using a posttest-only randomized control design, we compare the performance of 234 tutors (790 lesson completions) across three conditions: MCQ only, open response only, and a combination of both. We find no significant learning differences across conditions at posttest, but tutors in the MCQ condition took significantly less time to complete instruction. These findings suggest that MCQs are as effective, and more efficient, than open response tasks for learning when practice time is limited. To further enhance efficiency, we autograded open responses using GPT-4o and GPT-4-turbo. GPT models demonstrate proficiency for purposes of low-stakes assessment, though further research is needed for broader use. This study contributes a dataset of lesson log data, human annotation rubrics, and LLM prompts to promote transparency and reproducibility.

Figures

Figures reproduced from arXiv: 2412.10267 by the authors.

Figure 1
Figure 1. Instructional design sequence of the lessons illustrating the three learning-by-doing conditions, then the follow-up instruction [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Average posttest scores compared across learning-by-doing conditions: [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Average instruction time prior to posttest compared across learning-by-doing conditions: [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 29 canonical work pages

  1. [1]

    Vincent A Aleven and Kenneth R Koedinger. 2002. An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. Cognitive science 26, 2 (2002), 147–179

    work page 2002

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  3. [3]

    Andrew C Butler. 2018. Multiple-choice testing in education: Are the best practices for assessment also good for learning? Journal of Applied Research in Memory and Cognition 7, 3 (2018), 323–331

    2018

  4. [4]

    Dan Carpenter, Wookhee Min, Seung Lee, Gamze Ozogul, Xiaoying Zheng, and James Lester. 2024. Assessing Student Explanations with Large Language Models Using Fine-Tuning and Few-Shot Learning. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Ekaterina Kochmar, Marie Bexte, Jill Burstein, Andrea ...

    work page 2024

  5. [5]

    Loredana Caruccio, Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugam Sundaramurthy, and Genoveffa Tortora. 2024. Claude 2.0 Large Language Model: tackling a real-world classification problem with a new Iterative Prompt Engineering approach. Intelligent Systems with Applications 21 (2024)

    work page 2024

  6. [6]

    Pallavi Chhabra, Danielle Chine, Adetunji Adeniran, Shivang Gupta, and Kenneth Koedinger. 2022. An evaluation of perceptions regarding mentor competencies for technology-based personalized learning. In Society for Information Technology & Teacher Education International Conference . Association for the Advancement of Computing in Education (AACE), San Die...

    work page 2022

  7. [7]

    Michelene TH Chi and Ruth Wylie. 2014. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational psychologist 49, 4 (2014), 219–243

    work page 2014

  8. [8]

    Danielle R Chine, Pallavi Chhabra, Adetunji Adeniran, Shivang Gupta, and Kenneth R Koedinger. 2022. Development of scenario-based mentor lessons: an iterative design process for training at scale. In Proceedings of the Ninth ACM Conference on Learning@ Scale . Association for Computing Manuscript submitted to ACM 16 Thomas et al. Machinery, New York, NY, ...

    work page 2022

Show all 43 references
  1. [9]

    Chine, Pallavi Chhabra, Adetunji Adeniran, Joseph Kopko, Cindy Tipper, Shivang Gupta, and Kenneth R

    Danielle R. Chine, Pallavi Chhabra, Adetunji Adeniran, Joseph Kopko, Cindy Tipper, Shivang Gupta, and Kenneth R. Koedinger. 2022. Scenario-based training and on-the-job support for equitable mentoring. InProceedings of The Learning Ideas Conference 2022 . Springer, Cham, Switz...

    2022

  2. [10]

    Aubrey Condor, Max Litster, and Zachary Pardos. 2021. Automatic Short Answer Grading with SBERT on Out-of-Sample Questions. International Educational Data Mining Society (2021)

    2021

  3. [11]

    Thomas D Cook and Donald T Campbell. 2007. Experimental and quasi-experimental designs for generalized causal inference . Figures

    2007

  4. [12]

    Alison Cook-Sather. 2020. Student voice across contexts: Fostering student agency in today’s schools. Theory into practice 59, 2 (2020), 182–191

    2020

  5. [13]

    Andreea Dutulescu, Stefan Ruseti, Denis Iorga, Mihai Dascalu, and Danielle S McNamara. 2024. Beyond the Obvious Multi-choice Options: Introducing a Toolkit for Distractor Generation Enhanced with NLI Filtering. In International Conference on Artificial Intelligence in Educatio...

    2024

  6. [14]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed

  7. [15]

    Graham Gibbs. 1988. Learning by doing: A guide to teaching and learning methods. Further Education Unit (1988)

    1988

  8. [16]

    Ashish Gurung, Kirk Vanacore, Andrew A Mcreynolds, Korinn S Ostrow, Eamon Worden, Adam C Sales, and Neil T Heffernan. 2024. Multiple Choice vs. Fill-In Problems: The Trade-off Between Scalability and Learning. In Proceedings of the 14th Learning Analytics and Knowledge Confere...

    2024

  9. [17]

    Jonathan Guryan, Jens Ludwig, Monica P Bhatt, Philip J Cook, Jonathan Davis, Kenneth Dodge, George Farkas, Roland G Fryer Jr, Susan Mayer, Harold Pollack, et al. 2021. Not Too Late: Improving Academic Outcomes among Adolescents (Working Paper 28531). National Bureau of Economi...

    2021

  10. [18]

    Thomas M Haladyna. 2004. Developing and validating multiple-choice test items . Routledge

    2004

  11. [19]

    Zifei FeiFei Han, Jionghao Lin, Ashish Gurung, Danielle Thomas, Eason Chen, Conrad Borchers, Shivang Gupta, and Ken Koedinger. 2024. Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation. InAI for Education: Bridging Innovation and Responsibility at t...

    2024

  12. [20]

    Owen Henkel, Libby Hills, Adam Boxer, Bill Roberts, and Zach Levonian. 2024. Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability To Mark Short Answer Questions in K-12 Education. In Proceedings of the Eleventh ACM Conference on Learning@ Scale ...

    2024

  13. [21]

    Dollaya Hirunyasiri, Danielle R Thomas, Jionghao Lin, Kenneth R Koedinger, and Vincent Aleven. 2023. Comparative Analysis of GPT-4 and Human Graders in Evaluating Human Tutors Giving Praise to Students.. In Human-AI Math Tutoring@ AIED. 37–48

    2023

  14. [22]

    Sanjit Kakarla, Danielle Thomas, Jionghao Lin, Shivang Gupta, and Kenneth R Koedinger. 2024. Using large language models to assess tutors’ performance in reacting to students making math errors. arXiv preprint arXiv:2401.03238 (2024)

    2024 arXiv

  15. [23]

    Kenneth R Koedinger, Jihee Kim, Julianna Zhuxin Jia, Elizabeth A McLaughlin, and Norman L Bier. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale . Association ...

    2015

  16. [24]

    MA Kraft and G Falken. 2021. A blueprint for scaling tutoring across public schools (EdWorkingPaper No. 21–335). Annenberg Institute at Brown University

    2021

  17. [25]

    Gyeong-Geon Lee, Ehsan Latif, Xuansheng Wu, Ninghao Liu, and Xiaoming Zhai. 2024. Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence 6 (2024), 100213

    2024

  18. [26]

    Russell V. Lenth. 2024. emmeans: Estimated Marginal Means, aka Least-Squares Means . https://CRAN.R-project.org/package=emmeans R package version 1.10.2

    2024

  19. [27]

    Thomas, Wei Tan, Ngoc Dang Nguyen, and Kenneth R

    Jionghao Lin, Eason Chen, Zifei Han, Ashish Gurung, Danielle R. Thomas, Wei Tan, Ngoc Dang Nguyen, and Kenneth R. Koedinger. 2024. How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses. In Proceedings of the 17th International Confer...

    2024

  20. [28]

    Jionghao Lin, Zifei Han, Danielle R Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, and Kenneth R Koedinger. 2024. How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses. International Journal of Artificial Intelligence in Education (2024), 1–27

    2024

  21. [29]

    Jionghao Lin, Danielle R Thomas, Feifei Han, Shivang Gupta, Wei Tan, Ngoc Dang Nguyen, and Kenneth R Koedinger. 2023. Using Large Language Models to Provide Explanatory Feedback to Human Tutors. In Human-AI Math Tutoring@ AIED. 12–32

    2023

  22. [30]

    Andre Nickow, Philip Oreopoulos, and Vincent Quan. 2020. The impressive effects of tutoring on prek-12 learning: A systematic review and meta-analysis of the experimental evidence. (2020)

    2020

  23. [31]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774

    2024 arXiv

  24. [32]

    John F Pane, Beth Ann Griffin, Daniel F McCaffrey, and Rita Karam. 2014. Effectiveness of cognitive tutor algebra I at scale. Educational Evaluation and Policy Analysis 36, 2 (2014), 127–144

    2014

  25. [33]

    Napol Rachatasumrit, Paulo F Carvalho, Sophie Li, and Kenneth R Koedinger. 2023. Content matters: A computational investigation into the effectiveness of retrieval practice and worked examples. In International Conference on Artificial Intelligence in Education . Springer, Tok...

    2023

  26. [34]

    Carly D Robinson and Susanna Loeb. 2021. High-impact tutoring: State of the research and priorities for future learning. National Student Support Accelerator 21, 284 (2021), 1–53

    2021

  27. [35]

    Danielle Thomas, Xinyu Yang, Shivang Gupta, Adetunji Adeniran, Elizabeth Mclaughlin, and Kenneth Koedinger. 2023. When the tutor becomes the student: Design and evaluation of efficient scenario-based lessons for tutors. In LAK23: 13th International Learning Analytics and Knowl...

    2023

  28. [36]

    Danielle R Thomas, Jionghao Lin, Shambhavi Bhushan, Ralph Abboud, Erin Gatz, Shivang Gupta, and Kenneth R Koedinger. 2024. Learning and AI Evaluation of Tutors Responding to Students Engaging in Negative Self-Talk. InProceedings of the Eleventh ACM Conference on Learning@ Scal...

    2024

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  30. [38]

    William MK Trochim and James P Donnelly. 2001. Research methods knowledge base . Vol. 2. Atomic dog publishing Cincinnati, OH

    2001

  31. [39]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations

    2022

  32. [40]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837

    2022

  33. [41]

    Joy Yun, Yann Hicke, Mariah Olson, and Dorottya Demszky. 2024. Enhancing Tutoring Effectiveness Through Automated Feedback: Preliminary Findings from a Pilot Randomized Controlled Trial on SAT Tutoring. In Proceedings of the Eleventh ACM Conference on Learning@ Scale . 422–426

    2024

  34. [42]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren’s song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023). A Digital Appendix All anal...

    2023 arXiv

  35. [2024]

    Computational Linguistics (2024), 1–79

    Bias and fairness in large language models: A survey. Computational Linguistics (2024), 1–79

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.