Pith. sign in

REVIEW 4 major objections 6 minor 42 references

An AI-built course alone carried three learners through a vendor-scored certification exam.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:55 UTC pith:XAYJ27XU

load-bearing objection A coherent, unusually transparent design paper that deserves a referee, but its 3/3 certification pass is a weak existence proof and the abstract overstates what the three signals validate. the 4 major comments →

arxiv 2607.14044 v1 pith:XAYJ27XU submitted 2026-07-15 cs.AI

AI-accelerated End-to-End Framework for Rapid Professional Upskilling

Tam Nguyen , Hung Nguyen , Robert Ogburn This is my paper
classification cs.AI
keywords rapid upskillinggenerative AImulti-agent systemsintelligent tutoring systemsassessment developmentprofessional certificationknowledge baseAI-accelerated pipeline
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the bottleneck in workforce upskilling is production speed, not willingness: 59 of every 100 workers will need reskilling by 2030, while the average time to close an enterprise skills gap grew from about 3 days in 2014 to 36 days in 2018. It presents an end-to-end pipeline covering knowledge acquisition, content development, content review and verification, AI-tutor coaching, and assessment development, with AI doing the volume work and humans keeping the high-judgment roles. The central claim is that this pipeline can produce a certification-grade program for a frontier topic—multi-agent AI systems—inside the window in which the knowledge is still current, yielding a roughly 3,000-page prerequisite-ordered knowledge base, a 530-question assessment bank, 16 tutoring protocols, and three mock exam forms. The evidence is three convergence signals: three learners who studied only the knowledge base passed the NVIDIA Certified Professional in Agentic AI exam; an independent accountancy-standards body approved a program built on the outputs for continuing-education credit; and the same knowledge base supported a downstream risk analysis producing 1,267 risk items. The paper states plainly that these are observed signals, not controlled comparisons, and that n=3 establishes that the substrate can carry a learner, not the rate at which it will.

Core claim

The discovery the paper claims is that a single five-stage pipeline can compress the entire production chain of a certification-grade training program—from raw knowledge to verified chapters, misconception-keyed assessments, and protocolized tutoring—into the few months a frontier topic stays current. The strongest behavioral evidence is that all three learners who prepared with only the framework's knowledge base passed the NVIDIA Certified Professional in Agentic AI exam, which is administered and scored by an external vendor the authors do not control. Two further external checks converge with that result: an independent continuing-education accreditor approved a program built on the fram

What carries the argument

The central mechanism is the paired efficiency design: production efficiency from AI drafting, condensation, and cross-referencing, and learning efficiency from prerequisite-ordered structure, spaced retrieval, and misconception-targeted feedback. The load-bearing artifact is the knowledge base, an approximately 3,000-page four-level dependency hierarchy in which content is linked by strict prerequisite chains, so each learner step needs only already-mastered context; this same substrate is what the three exam passers studied exclusively and what later fed the risk analysis. Around it sit the verification layer—automated hallucination and faithfulness checks with numeric targets that can fai

Load-bearing premise

The load-bearing premise is that the three certification passes were caused by studying the framework's knowledge base—that these were ordinary adult learners with no prior agentic-AI expertise, hidden coaching, or selection that would have produced the passes anyway; the paper reports no demographic, experience, or time-to-competency data for them.

What would settle it

Give the same knowledge base to a larger, non-preselected cohort—say 30 learners—with prior agentic-AI experience, study time, and coaching recorded, and compare their pass rate and time-to-competency against a matched group using conventional materials. If prior experience predicts passing or the pass rate collapses, the causal claim fails. A cheaper check: audit the promised immutable audit trail for SME sign-off across the ~3,000-page knowledge base; the paper states Layer 2 sign-off is 'fully specified but not yet fully evidenced in execution records,' so a finding that most chapters carry

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If a knowledge base alone can carry learners through a vendor-scored certification, then for new frontier certifications the critical production constraint shifts from content creation to verification and assessment design.
  • The stage-level methods are documented for replication, so the pipeline can be re-pointed at a new topic by replacing the blueprint and the authoritative references.
  • Because any unaccelerated stage re-imposes its bottleneck on the whole chain, the framework's end-to-end coverage, not any single stage, is what makes the compressed timeline possible.
  • The same knowledge base serving both exam prep and downstream risk analysis suggests the substrate is reusable beyond training, as a structured reference for expert analysis.
  • The mixed verification results (only 5 of 10 chapters met the pedagogical-progression bar at the reported pass, and the assessment bank stood at 63% of target) imply the quality gates are doing real filtering rather than ratifying the pipeline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 3-for-3 result replicates with the 14 learners in progress, certification pass rate could become a practical benchmark for comparing knowledge-base-only instructional materials, something the field currently lacks.
  • A controlled comparison—matched learners using conventional vendor materials versus the framework's knowledge base, with prior experience and study time recorded—would test the causal claim that the paper's n=3 cannot.
  • The two-regime framing suggests the framework will help most in domains where certification resources barely exist; its gains should shrink in mature topics with abundant textbooks, instructors, and item banks.
  • The unresolved items (SME sign-off not fully evidenced, pedagogical-progression remediation still tracked) mean an audit of the verification layer could either strengthen or overturn the 'verified content' premise; that audit is a plausible next step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes CrewScaler, a five-stage AI-accelerated pipeline for professional upskilling: knowledge acquisition, content development, content review and verification, AI-tutor coaching, and assessment development. It reports design details such as a prerequisite-ordered knowledge hierarchy, a one-new-element readability rule, 16 tutoring protocols, and a misconception-keyed item bank. The claimed validation consists of three 'external signals': NASBA CPE approval of a program built on the framework, 3/3 passes on the NVIDIA Certified Professional in Agentic AI (NCP-AAI) exam by learners who used only the knowledge base, and a downstream risk-analysis dataset of 1,267 items generated from the same knowledge base. The authors explicitly state that the paper reports design and validation signals, not controlled comparisons.

Significance. If the framework's production and learning-efficiency claims hold, the paper would offer a replicable, end-to-end template for rapidly building certification-grade training in frontier technical domains, an area with real labor-market relevance. The paper has notable strengths: it chooses a vendor-scored external exam rather than self-graded outcomes; it reports criterion-referenced QC failures openly; and it documents stage-level methods in enough detail to reproduce. Those strengths make the paper worth engaging. However, the current evidence base is too thin to support the causal interpretation the abstract and Section IV-A give to the certification result, and one of the three 'external' signals is produced by the authors' own pipeline.

major comments (4)
  1. [Section IV-A] The claim that 'studying the knowledge base alone can carry a learner' through the NCP-AAI exam is an existence claim that requires at least one learner with verified low baseline proficiency, exclusive use of the KB, and a passing score. The paper provides no demographics, prior agentic-AI expertise, education, or time-to-competency for the three passers, nor the exam's overall pass rate. Without a baseline, 3/3 is fully compatible with the learners already being able to pass. Please add per-learner background (or state that such data cannot be disclosed) and soften the causal wording; otherwise the result is a use-case anecdote, not validation.
  2. [Section IV-B / IV-C] The 'capability outcome' is not external: the 1,267-item risk dataset is generated by the authors' own KDEG/threat-modeling pipeline [28], and [28] is under review with no reliability or validity data reported here. 'Surface validation' before ~500 federal employees is not a scientific check. NASBA approval is a standards-based program-design review and does not measure learning outcomes. Thus the abstract's 'three strong external signals validates the framework' overstates the evidence: only the vendor-scored certification is outcome-based, and it is the one with the weakest documentation.
  3. [Section III-C] The reported QC numbers directly bear on the certification claim. Only 5/10 chapters met the 70% pedagogical-progression bar, the assessment bank was at 63% of target, and SME sign-off was 'not yet fully evidenced.' The paper says the assessment gap was closed later, but it never states whether the three certification learners studied the remediated or pre-remediation KB. State the timeline and QC status of the exact material used, and discuss how 5/10 chapters passing the pedagogical bar at the time is compatible with the claim of certification-grade content.
  4. [General / Section IV] The paper does not disentangle which components of the framework (prerequisite hierarchy, one-new-element rule, 16 tutoring protocols, 30/50/20 difficulty split, etc.) contributed to the 3/3 outcome. Many parameters and design choices are introduced, but no ablation or component-level evidence is provided. The observed pass outcome can therefore at best validate the pipeline as a whole, and only weakly at that. The text should explicitly acknowledge this attribution limit in the validation discussion.
minor comments (6)
  1. [Abstract] 'Significantly short amount of time' is undefined; no time-to-competency data appear in Section IV-A. Either provide the numbers or remove the phrase.
  2. [Section III-B] Typo: 'human currated' should be 'human-curated'.
  3. [Section IV-B] Typo: 'being peered reviewed' should be 'being peer-reviewed.' Also, 'surface validation' is vague; specify the procedure or remove the claim.
  4. [Section VI] Typo in the conclusion: 'fand' should be 'and'.
  5. [Section IV-A] Capitalization of 'Nvidia' vs 'NVIDIA' is inconsistent; standardize to the vendor's official style.
  6. [References] Reference [28] is central to the capability-outcome signal but is listed as 'manuscript under review' with no stable identifier. Include an arXiv ID or a supplementary data appendix so the claim can be checked by reviewers.

Circularity Check

1 steps flagged

Certification and NASBA signals are external, but the capability outcome reduces to the authors' own knowledge base processed by their own unreviewed pipeline.

specific steps
  1. self citation load bearing [Abstract; Section IV-B; reference [28]]
    "The knowledge base also served as the direct input to a systematic risk analysis of a baseline multi-agent AI system [28]. In that analysis, threat-modeling agents worked through the roughly 3,000-page knowledge base chapter by chapter, and produced 1,267 risk items across 81 categories and 14 domains."

    This capability outcome is advertised as one of three strong external validation signals, but the 1,267 item count is produced by the authors' own threat-modeling agents reading the authors' own knowledge base. The output is a deterministic function of the input, so it cannot independently confirm the knowledge base's completeness; it is a self-consistency check. The only documentation is [28], an unreviewed manuscript under the same first author, and the paper's own Section III-C says SME sign-off is 'not yet fully evidenced'. Thus this pillar of the triangulation reduces to self-citation plus internal reuse rather than an external benchmark.

full rationale

The paper's strongest claim—that three learners passed the external NVIDIA NCP-AAI exam after studying only the knowledge base—is not circular: the exam is administered and scored by a vendor the authors do not control, and the paper is explicit that it does not extrapolate the 3/3 rate. The lack of learner-baseline data and the absence of a comparison pass rate are validity/selection threats, not definitional circularity. Similarly, NASBA CPE approval is an independent third-party review and is not used to define any predicted quantity. The one substantially circular element is the capability outcome (1,267 risk items): it is generated from the same knowledge base by the same authors' threat-modeling pipeline and cited to [28], a manuscript under review by the same first author. Presenting this as a third external signal makes the paper's triangulation partially self-referential; however, because the certification outcome remains genuinely external and the paper makes no fitted prediction that is forced by construction, the overall circularity is mild.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 2 invented entities

The central claim rests on a set of domain assumptions about what external signals mean (certification passes, NASBA approval, risk-item derivation), the validity of LLM-based verification, and the accuracy of cited labor-market statistics. The framework's instructional design depends on hand-selected constants (difficulty splits, review ratios, QC thresholds) that are asserted, not optimized.

free parameters (6)
  • Assessment difficulty split 30/50/20 = 30% easy / 50% medium / 20% hard
    Chosen by hand in Section III-E to steer LLM-generated item pools; not fitted to learner data.
  • Spaced-review ratio 70/20/10 = 70% current, 20% prior, 10% foundational questions
    Set by design in Section III-B to operationalize spaced retrieval; no empirical tuning.
  • Pedagogical-progression QC threshold = 70%
    Section III-C/IV reports 5 of 10 chapters failed to meet this hand-set bar; threshold is asserted, not calibrated.
  • One-new-element rule = one new complexity per section
    Design constraint imposed during readability pass (Section III-B) to keep learner cognitive load bounded.
  • Learning-objective count per chapter = 3 to 8 SMART objectives
    Fixed template constraint in Section III-B; chosen by design rather than derived from data.
  • Six-pass revision count = six passes
    Process count in Section III-B; no evidence given for the number.
axioms (6)
  • domain assumption The NVIDIA NCP-AAI exam is a valid and reliable measure of agentic-AI professional competency.
    Section IV-A treats passing the vendor-scored exam as evidence of competency; the paper provides no psychometric data for the exam.
  • domain assumption NASBA CPE approval of a program is an indicator of educational-program quality.
    Section IV-C: NASBA review checks program design/delivery against CPE standards, not learner outcomes; the paper equates this with validation.
  • domain assumption Adapted RAGAS automated checks reliably detect hallucination and content defects.
    Section III-C: Layer 1 verification uses an adapted RAGAS framework with numeric targets; the paper itself reports mixed results, so the axiom is load-bearing and imperfectly satisfied.
  • domain assumption Deriving a 1,267-item risk dataset from the knowledge base demonstrates knowledge-base completeness and quality.
    Section IV-B: threat-modeling agents consume the 3,000-page knowledge base; the output's quality is taken as a signal of input quality, which is partly circular.
  • domain assumption Cited labor-market statistics (WEF 59/100, IBM 3-to-36-day gap) are accurate.
    Section I motivates the problem with these numbers; the framework's significance depends on them.
  • ad hoc to paper The 'two-regime framing' (AI as leveler vs multiplier) accurately categorizes the cited studies.
    Introduced in Section I as an interpretation; not tested independently and used to justify the framework's human-in-the-loop design.
invented entities (2)
  • Knowledge Domain Exploring Guide (KDEG) no independent evidence
    purpose: Four-step procedure for knowledge acquisition: domain framing, job-task analysis, blueprinting, evidence mapping (Section III-A).
    KDEG is a new named construct; its effectiveness is only evidenced through the framework's own outputs (the knowledge base), with no independent test.
  • Two-regime framing (AI as leveler vs multiplier) no independent evidence
    purpose: Interpretive lens to explain AI benefit heterogeneity and justify human high-judgment roles (Section I).
    It is a post-hoc categorization of published experiments; no new data test it.

pith-pipeline@v1.3.0-alltime-deepseek · 8837 in / 17775 out tokens · 151520 ms · 2026-08-02T02:55:19.986575+00:00 · methodology

0 comments
read the original abstract

By 2030, 59 of every 100 workers will need reskilling or upskilling, yet the average time to close an enterprise skills gap grew from roughly 3 days in 2014 to 36 days in 2018. Most current frameworks accelerate single stages of upskilling programs and generally lack industry validation. We present an end-to-end framework that applies AI acceleration across five stages of knowledge acquisition, content development, content review and verification, teaching, and assessment development; with a strong focus on both production and learning efficiency. Three strong external signals validates the framework: the US National Association of State Boards of Accountancy reviewed and approved an upskilling program built on the framework for continuing-professional-education credits; 3 learners followed the program and passed the NVIDIA Certified Professional in Agentic AI exam in a significantly short amount of time, with 14 more in progress; the program's knowledge base supports complex downstream analysis such as the production of a robust 1,267 risk item dataset for managing multi-agent AI system risks.

Figures

Figures reproduced from arXiv: 2607.14044 by Hung Nguyen, Robert Ogburn, Tam Nguyen.

Figure 1
Figure 1. Figure 1: The five-stage pipeline. AI accelerates production within each [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 1 linked inside Pith

  1. [1]

    The future of jobs report 2025,

    World Economic Forum, “The future of jobs report 2025,” World Economic Forum, Geneva, Switzerland, Insight Report, Jan. 2025. [Online]. Available: https://www.weforum.org/publications/the-future- of-jobs-report-2025/

    2025

  2. [2]

    Skills aren’t soft or hard — they’re durable or perishable,

    M. J. Daniel, “Skills aren’t soft or hard — they’re durable or perishable,” Chief Learning Officer, Oct. 2020. [Online]. Available: https://www.chieflearningofficer.com/2020/10/29/skills-arent- soft-or-hard-theyre-durable-or-perishable/

    2020

  3. [3]

    The enterprise guide to closing the skills gap: Strategies for building and maintaining a skilled workforce,

    A. LaPrade, J. Mertens, T. Moore, and A. Wright, “The enterprise guide to closing the skills gap: Strategies for building and maintaining a skilled workforce,” IBM Institute for Business Value, Tech. Rep., Sep. 2019. [Online]. Available: https://www.ibm.com/thought- leadership/institute-business-value/en-us/report/closing-skills-gap

    2019

  4. [4]

    Generative AI at work,

    E. Brynjolfsson, D. Li, and L. R. Raymond, “Generative AI at work,” The Quarterly Journal of Economics, vol. 140, no. 2, pp. 889–942, 2025

    2025

  5. [5]

    Experimental evidence on the productivity effects of generative artificial intelligence,

    S. Noy and W. Zhang, “Experimental evidence on the productivity effects of generative artificial intelligence,”Science, vol. 381, no. 6654, pp. 187–192, Jul. 2023

    2023

  6. [6]

    Navigating the jagged technological frontier: Field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality,

    F. Dell’Acqua, E. McFowland III, E. Mollick, H. Lifshitz, K. C. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani, “Navigating the jagged technological frontier: Field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality,”Organization Science, vol. 37, no. 2, pp. 403–423, Mar. 2026

    2026

  7. [7]

    Retraining and reskilling workers in the age of automation,

    P. Illanes, S. Lund, M. Mourshed, S. Rutherford, and M. Tyreman, “Retraining and reskilling workers in the age of automation,” McKinsey Global Institute, Tech. Rep., Jan. 2018. [Online]. Available: https://www.mckinsey.com/featured-insights/future- of-work/retraining-and-reskilling-workers-in-the-age-of-automation

    2018

  8. [8]

    M. S. Knowles, E. F. Holton, III, and R. A. Swanson,The Adult Learner: The Definitive Classic in Adult Education and Human Resource Development, 8th ed. London, U.K.: Routledge, 2015

    2015

  9. [9]

    Guidelines for designing AI technologies to support adult learning,

    J. M. Reddiget al., “Guidelines for designing AI technologies to support adult learning,” inProc. ACM Designing Interactive Syst. Conf. (DIS). Singapore: ACM, 2026, pp. 2474–2496

    2026

  10. [10]

    PlanGlow: Personalized study planning with an explainable and controllable LLM-driven system,

    J. Chun, Y . Zhao, H. Chen, and M. Xia, “PlanGlow: Personalized study planning with an explainable and controllable LLM-driven system,” in Proc. 12th ACM Conf. Learning @ Scale (L@S). Palermo, Italy: ACM, 2025, pp. 116–127

    2025

  11. [11]

    Short-term gains, long-term gaps: The impact of GenAI and search technologies on retention,

    M. Akgun and S. Toker, “Short-term gains, long-term gaps: The impact of GenAI and search technologies on retention,” inProc. AIED 2025 Posters and Late Breaking Results, ser. Communications in Computer and Information Science, vol. 2591. Springer Nature Switzerland, 2025, pp. 44–52

    2025

  12. [12]

    Instructional agents: Reducing teaching faculty workload through multi-agent instructional design,

    H. Yao, W. Xu, J. Turnau, N. Kellam, and H. Wei, “Instructional agents: Reducing teaching faculty workload through multi-agent instructional design,” inProc. 19th Conf. Eur. Chapter Assoc. Comput. Linguistics (EACL). Rabat, Morocco: Association for Computational Linguistics, Mar. 2026, pp. 4087–4109

    2026

  13. [13]

    Enabling multi-agent systems as learning designers: Applying learning sciences to AI instructional design,

    J. Wang, R. Xiao, X. Hou, and J. Stamper, “Enabling multi-agent systems as learning designers: Applying learning sciences to AI instructional design,” 2025. [Online]. Available: https://arxiv.org/abs/2508.16659

    Pith/arXiv arXiv 2025

  14. [14]

    ARCHED: A human-centered framework for transparent, responsible, and collaborative AI assisted instructional design,

    H. Liet al., “ARCHED: A human-centered framework for transparent, responsible, and collaborative AI assisted instructional design,” inProc. iRAISE Workshop at AAAI, ser. Proceedings of Machine Learning Research, vol. 273. PMLR, 2025, pp. 94–104. [Online]. Available: https://proceedings.mlr.press/v273/li25a.html

    2025

  15. [15]

    RAG-PRISM: A personalized, rapid, and immersive skill mastery framework with adaptive retrieval-augmented tutoring,

    G. Raulet al., “RAG-PRISM: A personalized, rapid, and immersive skill mastery framework with adaptive retrieval-augmented tutoring,” in Proc. IEEE Frontiers in Educ. Conf. (FIE). Nashville, TN, USA: IEEE, Nov. 2025, pp. 1–9

    2025

  16. [16]

    Tutor CoPilot: A human-AI approach for scaling real-time expertise,

    R. E. Wang, A. T. Ribeiro, C. D. Robinson, S. Loeb, and D. Demszky, “Tutor CoPilot: A human-AI approach for scaling real-time expertise,” 2024, arXiv preprint; also available as EdWorkingPaper No. 24-1054, Annenberg Institute at Brown University

    2024

  17. [17]

    Teacher- authored prompts for configuring student-AI dialogue: K12 classroom implementation,

    A. Liu, M. Sun, L. Esbenshade, V . Tian, Z. Zhang, and K. He, “Teacher- authored prompts for configuring student-AI dialogue: K12 classroom implementation,” 2026, arXiv preprint; journal DOI to be assigned

    2026

  18. [18]

    A classroom study of LLM-generated feed- back intervention in introductory programming,

    H. Heickal and A. Lan, “A classroom study of LLM-generated feed- back intervention in introductory programming,” 2026, arXiv preprint; accepted at IRAISE 2026 (Festival of Learning)

    2026

  19. [19]

    AI technicians: Developing rapid occupational training methods for a competitive AI workforce,

    J. Savelkaet al., “AI technicians: Developing rapid occupational training methods for a competitive AI workforce,” inProc. 56th ACM Tech. Symp. Comput. Sci. Educ. (SIGCSE TS). Pittsburgh, PA, USA: ACM, 2025, pp. 1029–1035

    2025

  20. [20]

    A systematic review of research on large language models for computer programming education,

    M. Zhu, L. Xu, and B. Ericson, “A systematic review of research on large language models for computer programming education,” 2025, arXiv preprint

    2025

  21. [21]

    Quantifying cognitive bias induc- tion in LLM-generated content,

    A. Alessa, P. Somane, A. T. Lakshminarasimhan, J. Skirzynski, J. McAuley, and J. M. Echterhoff, “Quantifying cognitive bias induc- tion in LLM-generated content,” inProc. IJCNLP-AACL. Mumbai, India: The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, Dec. 2025, pp. 2890–2910

    2025

  22. [22]

    Fine-grained hallucination detection and editing for language models,

    A. Mishraet al., “Fine-grained hallucination detection and editing for language models,” inProc. 1st Conf. Language Modeling (COLM), 2024. [Online]. Available: https://openreview.net/forum?id=dJMTn3QOWO

    2024

  23. [23]

    VeriScore: Evaluating the factuality of verifiable claims in long-form text generation,

    Y . Song, Y . Kim, and M. Iyyer, “VeriScore: Evaluating the factuality of verifiable claims in long-form text generation,” inFindings Assoc. Comput. Linguistics: EMNLP. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 9447–9474

    2024

  24. [24]

    TutorGym: A testbed for evaluating AI agents as tutors and students,

    D. Weitekamp, M. N. Siddiqui, and C. J. MacLellan, “TutorGym: A testbed for evaluating AI agents as tutors and students,” inProceedings of the 26th International Conference on Proc. AIED 2025. Springer, 2025, pp. 361–376

    2025

  25. [25]

    Position: Evaluating generative AI systems is a social science measurement challenge,

    H. M. Wallachet al., “Position: Evaluating generative AI systems is a social science measurement challenge,” inProc. 42nd Int. Conf. Mach. Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 82 232–82 251. [Online]. Available: https://proceedings.mlr.press/v267/wallach25a.html

    2025

  26. [26]

    From course to skill: Evaluating large language model performance in curricular analytics,

    Z. Xu, X. Li, Y . Huan, V . Minaya, and R. Yu, “From course to skill: Evaluating large language model performance in curricular analytics,” in Proc. AIED 2025, ser. Lecture Notes in Computer Science, vol. 15882. Cham: Springer, 2025, pp. 203–211

    2025

  27. [27]

    Modernizing ground truth: Four shifts toward improving reliability and validity in AI in education,

    D. R. Thomas, C. Borchers, K. P. Vanacore, K. R. Koedinger, and R. F. Kizilcec, “Modernizing ground truth: Four shifts toward improving reliability and validity in AI in education,” inProc. 27th Int. Conf. Artif. Intell. Educ. (AIED), ser. Lecture Notes in Computer Science, vol. 16586. Springer, 2026, pp. 117–131

    2026

  28. [28]

    Security considerations for multi-agent AI systems: A scalable design-based framework and survey,

    T. N. Nguyenet al., “Security considerations for multi-agent AI systems: A scalable design-based framework and survey,” 2026, manuscript under review

    2026

  29. [29]

    Test-enhanced learning: Taking memory tests improves long-term retention,

    H. L. Roediger, III and J. D. Karpicke, “Test-enhanced learning: Taking memory tests improves long-term retention,”Psychological Science, vol. 17, no. 3, pp. 249–255, 2006

    2006

  30. [30]

    The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems,

    K. VanLehn, “The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems,”Educational Psychologist, vol. 46, no. 4, pp. 197–221, 2011

    2011

  31. [31]

    Training LLM- based tutors to improve student learning outcomes in dialogues,

    A. Scarlatos, N. Liu, J. Lee, R. Baraniuk, and A. Lan, “Training LLM- based tutors to improve student learning outcomes in dialogues,” inProc. AIED 2025, ser. Lecture Notes in Computer Science, vol. 15877. Cham: Springer, 2025, pp. 251–266

    2025

  32. [32]

    Productive failure in learning math,

    M. Kapur, “Productive failure in learning math,”Cognitive Science, vol. 38, no. 5, pp. 1008–1022, 2014

    2014

  33. [33]

    Dynamics of affective states during complex learning,

    S. D’Mello and A. Graesser, “Dynamics of affective states during complex learning,”Learning and Instruction, vol. 22, no. 2, pp. 145– 157, 2012

    2012

  34. [34]

    Knowledge tracing: Modeling the acquisition of procedural knowledge,

    A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,”User Modeling and User-Adapted Interaction, vol. 4, no. 4, pp. 253–278, 1994

    1994

  35. [35]

    Brief but impactful: How human tutoring interactions shape engagement in online learning,

    C. Borchers, A. Gurung, Q. Liu, D. R. Thomas, M. Khalil, and K. R. Koedinger, “Brief but impactful: How human tutoring interactions shape engagement in online learning,” inProc. 16th Int. Learning Analytics and Knowledge Conf. (LAK). Bergen, Norway: ACM, 2026

    2026

  36. [36]

    Assessing the quality of AI-generated exams: A large- scale field study,

    C. Isleyet al., “Assessing the quality of AI-generated exams: A large- scale field study,” inProc. AAAI Conf. Artif. Intell., vol. 40, no. 45, 2026, pp. 38 626–38 634

    2026

  37. [37]

    MCQG-SRefine: Multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feed- back,

    Z. Yaoet al., “MCQG-SRefine: Multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feed- back,” inProc. NAACL-HLT. Albuquerque, New Mexico: Association for Computational Linguistics, Apr. 2025, pp. 10 728–10 777

    2025

  38. [38]

    QG-SMS: En- hancing test item analysis via student modeling and simulation,

    B. Nguyen, T. Du, M. Yu, L. Angrave, and M. Jiang, “QG-SMS: En- hancing test item analysis via student modeling and simulation,” inProc. 63rd Annu. Meeting Assoc. Comput. Linguistics (ACL). Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 26 152–26 168

    2025

  39. [39]

    The iceberg index: Measuring skills-centered exposure in the AI economy,

    A. Chopraet al., “The iceberg index: Measuring skills-centered exposure in the AI economy,” 2025

    2025

  40. [40]

    Automation experiments and inequal- ity,

    S. G. Benzell and K. R. Myers, “Automation experiments and inequal- ity,” National Bureau of Economic Research, Working Paper 34668, Jan. 2026, also available as arXiv:2510.24923

    arXiv 2026

  41. [41]

    Some simple economics of AGI,

    C. Catalini, X. Hui, and J. Wu, “Some simple economics of AGI,”

  42. [2026]

    Available: https://arxiv.org/abs/2602.20946

    [Online]. Available: https://arxiv.org/abs/2602.20946

    arXiv