{"id":"e4d05af3-f0db-4a51-aa77-b45bfe62eb95","arxiv_id":"2411.17924","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AI2T lets non-programmers author model-tracing tutors in about 20 to 30 minutes by teaching an agent that reports how certain it is about unseen problem steps.","lead":"AI2T is an interactively teachable AI that learns to run intelligent tutoring systems from demonstrations and yes/no feedback, while estimating its own certainty during learning. A small user study suggests authors can use those certainty scores to decide when the trained tutor program is reliable enough to trust.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The self-aware stopping claim is the most load-bearing part of the paper, and it is not yet established: only 4/10 study-2 participants used the certainty scores, there was no no-certainty control, and the paper itself defers to a future randomized experiment.","rationale":"The paper's strongest evidence is the study 2 result: 8/10 participants achieved 100% holdout completeness in multicolumn addition and 5/10 in fraction arithmetic, with median authoring times near 22 minutes. That is real, hard-won evidence for the core efficiency claim, and I would not discount it. I part ways with the reader's choice of weakest assumption: the ideal-user simulation in Section 5 is explicitly a learning-mechanism comparison, not the source of the user-facing 20-30 minute claim; that claim comes directly from the user studies. The more load-bearing gap is the certainty-stopping heuristic, which the paper presents as the mechanism that makes AI2T 'self-aware' and 'trustable.' The direct evidence is only 4/10 self-selected users in a non-randomized study with no control arm, and the paper's own Section 6.4 calls for a randomized experiment. Moreover, the simulation's productive-monotonicity result is below 50% until about 60 training problems in multicolumn addition, whereas study 2 users stopped around 11 problems. Thus, if the certainty heuristic fails in that region, the claim that authors can know when training is sufficient loses its operational support. The absence of released code and data is a separate reproducibility concern, not the single load-bearing correctness issue. My recommendation is to keep the CONDITIONAL verdict: the central system may well work, but this particular component needs a controlled test before the paper's 'trustable' claim can be accepted as stated.","tokens_in":21258,"tokens_out":7042,"duration_ms":68090,"concrete_test":"Pre-register and run a between-subjects experiment in one domain, such as multicolumn addition, with at least 20 non-programmers per arm: both arms get identical training instructions and the AI2T interface, but the control arm has the certainty-score display disabled. Measure holdout model-tracing completeness, number of training problems, and time to self-reported stopping. If the certainty arm does not achieve higher median completeness and/or fewer incomplete stops than the control arm, the claim that authors can use certainty scores as a stopping heuristic is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Study 2 is the only direct evidence that authors can use STAND's instance certainty to decide when training is sufficient, but it is a non-randomized, small-N study in which 5/10 participants noticed the scores and only 4/10 reported using them. There is no control condition without scores, so the observed stopping behavior cannot be attributed to the certainty display, and the paper concedes in Section 6.4 that a future randomized experiment is needed. This gap is central to the paper's promise: the abstract and Section 3.5 claim that certainty lets authors know when the agent is complete, and the headline 20-30 minute authoring time is only meaningful if stopping when certainty is high reliably yields complete tutors. The simulation evidence does not close the gap. In Figure 11 and Section 5.1, STAND's productive monotonicity in multicolumn addition is below 50% until roughly 60 training problems, while study 2 participants stopped after a median of about 11 problems and 22 minutes. The quantitative support for certainty-based stopping is therefore weakest in exactly the regime where authors must apply it. The ideal-user simulation is a real limitation for the algorithm comparison in Section 5, but it is not the load-bearing issue for the user-facing claim, since the 20-30 minute result comes from the actual user studies.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AI2T, an interactive authoring-by-tutoring system for building model-tracing intelligent tutoring systems (ITSs). Authors train the agent by demonstrating step-by-step solutions and grading the agent's proposed actions; the system induces hierarchical task networks (HTNs) via a process-learning mechanism and learns preconditions with STAND, a version-space-based algorithm that also outputs instance-certainty scores. The authors report simulation experiments in multicolumn addition and fraction arithmetic comparing STAND with decision trees, random forests, and XGBoost on productive monotonicity, precision at high certainty, and holdout completeness, and two user studies (n=10 each) in which participants authored tutors. The paper claims that authors can achieve correct and complete model-tracing behavior in about 20-30 minutes per topic and can use STAND's certainty scores as a heuristic to decide when training is sufficient.","tokens_in":21454,"tokens_out":6252,"duration_ms":59173,"significance":"If the central claims hold, this is a substantial contribution to ITS authoring and interactive task learning: it demonstrates a concrete path for non-programmers to author flexible model-tracing tutors, and it provides a self-aware certainty signal that could address a long-standing gap in authoring-by-tutoring. The simulation comparison against strong baselines, the detailed interaction-design contributions (skill application window, behavior graph, unordered groups), and the honest reporting of study 1's failures and study 2's limitations are all strengths. The user-study results, with 8/10 participants reaching 100% completeness in multicolumn addition in study 2, are encouraging and clearly improve on prior work. However, the evidence for the certainty-based stopping heuristic is preliminary, and several load-bearing quantitative claims need strengthening before the paper's strongest conclusions can be accepted.","major_comments":[{"comment":"The claim that 'our user study shows that authors can use STAND's certainty heuristic to estimate when AI2T has been trained on enough diverse problems' is not supported by the study 2 evidence as reported. Only 5 of 10 participants noticed the certainty scores and 4 of 10 reported using them; there is no control condition without scores, no random assignment, and the reported use is self-reported. Section 6.4 itself concedes that a future randomized experiment is needed. Because the abstract and Section 3.5 make the certainty heuristic a central contribution, the manuscript should either substantially soften this claim or provide an additional quantitative analysis (e.g., comparing stopping outcomes for participants who did and did not use the scores) that directly supports it.","section":"Section 6.3.1 and Section 6.4 (also Abstract and Section 3.5)"},{"comment":"In multicolumn addition, STAND's productive monotonicity is below 50% for approximately the first 60 training problems, yet study 2 participants stopped after a median of 11 problems and 22 minutes. The certainty-based stopping heuristic is therefore being applied in exactly the regime where the simulation suggests certainty changes are not yet reliably aligned with holdout performance. The paper should report productive monotonicity at early training sizes (e.g., problems 5-20) for both domains and either show that early-problem monotonicity is adequate for the observed stopping behavior or explicitly restrict the certainty-heuristic claim to later training.","section":"Section 5.1, Figure 11, and Table 4"},{"comment":"The simulation experiments assume an ideal user who always selects the correct formula annotation and grades every proposed action. This assumption is load-bearing for the comparative data-efficiency claims in Tables 1 and 2, because the user studies demonstrate that real authors make uncaught training mistakes, and two study 2 participants ended below 50% completeness in fraction arithmetic. The paper should at minimum quantify the sensitivity of the STAND-vs-baseline comparisons to annotation and grading noise (e.g., a noise-injection experiment), and the conclusions should state that the strong data-efficiency numbers assume a perfect trainer.","section":"Section 5, automated training system paragraph"}],"minor_comments":[{"comment":"The heading 'Supporting Completene Correctness Feedback for Each Problem State' contains a typo; it should be 'Supporting Complete Correctness Feedback for Each Problem State.'","section":"Section 3.3"},{"comment":"The text says 'in the two domains that we had participants author in our user study (section 5)', but the user studies are described in Section 6, not Section 5.","section":"Section 5, first paragraph"},{"comment":"There are repeated spelling errors: 'heirarchical' should be 'hierarchical', and 'processs-learning' should be 'process-learning'.","section":"Sections 2 and 4"},{"comment":"The simulation results are averaged over 40 repetitions but no error bars or confidence intervals are shown; adding them would help the reader assess the stability of the reported differences, especially the early-training comparisons that are central to the certainty-stopping claim.","section":"Figures 11 and 12"},{"comment":"The hyperparameters for the comparison classifiers are only partially specified (e.g., random forest size is given but XGBoost details are not); please provide the exact settings and any hyperparameter tuning procedure for reproducibility.","section":"Section 5, classifier descriptions"},{"comment":"STAND is the core algorithmic contribution but is only summarized here and defined in the authors' prior preprint [57]; the paper would be more self-contained if it included a concise formal description of the version-space representation and the certainty computation, or a precise pointer to the relevant equations in [57].","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"For a journal venue, the contribution depends heavily on the authors' own prior STAND paper [57]; the editor may wish to ensure that manuscript is peer-reviewed and accessible, since the self-aware stopping claim is not fully reproducible from this paper alone. The user-study evidence for the headline certainty heuristic is thin, and the revision should bring the abstract and conclusions in line with the actual evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the process-learning mechanism and the study 2 results are real contributions, but the central claim that certainty scores let authors know when to stop is not yet established, and the paper is candid about that. It deserves peer review, not a desk reject.\n\nWhat is new: process-learning induces hierarchical task networks bottom-up from action sequences and is order-agnostic. The simulation comparison to decision trees, random forests, and XGBoost shows STAND+HTN reaches higher holdout completeness with fewer problems, with 40/40 repetitions reaching 100% completeness in multicolumn addition. In the user studies, 8/10 participants achieved 100% completeness in multicolumn addition and 5/10 in fractions, at a median of around 22 minutes. For a field where prior SimStudent/AL work plateaued below 100%, that is a meaningful result, and the paper reports the failures honestly.\n\nThe soft spots are real. The user studies are small, non-randomized, and the two studies differ in multiple factors, so study 1 versus study 2 is a loose pseudo-experiment. The stopping-heuristic evidence is the weakest link: only 5/10 participants noticed the certainty scores and 4/10 said they used them, there was no control condition, and the paper itself says a randomized experiment is needed. The simulation's productive monotonicity in multicolumn addition is below 50% until roughly 60 training problems, while study 2 participants stopped after a median of 11 problems, so the quantitative support is weakest exactly where the authors need it. That does not kill the paper, but it means the 20-30 minute authoring claim and the certainty-stopping claim should be read as preliminary. The ideal-user simulation assumption is a limitation for the algorithm comparison, not the main issue for the user-facing claim. Also, no code or data are provided, and the LLM comparison is asserted rather than evaluated; that part needs either evidence or removal.\n\nThe citation pattern is fine: STAND is prior work and is cited as such, and the paper is clear that the algorithm itself is not new. I would like to see the authors release artifacts and run a controlled stopping-rule study, but the current evidence is enough to justify a serious referee.","headline":"Process-learning and the study 2 results are worth taking seriously, but the certainty-based stopping claim is preliminary and needs a controlled study before it can carry the weight the abstract puts on it.","tokens_in":22019,"tokens_out":1851,"would_cite":true,"duration_ms":17375,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A non-programmer can author a complete model-tracing tutor in 20-30 minutes by demonstrating solutions and grading the agent's attempts, with certainty scores as the stopping signal.","keywords":["interactive task learning","authoring-by-tutoring","model-tracing tutors","hierarchical task networks","programming by demonstration","certainty estimation","intelligent tutoring systems","self-aware learning"],"falsifier":"Run the paper's exact simulation protocol but inject a single incorrect correctness label or one wrong formula annotation at an early training problem; if STAND plus HTN induction then fails to reach 100% holdout completeness in most repetitions, the completeness result depends on flawless teaching rather than on the learning mechanism itself.","tokens_in":20997,"feed_emoji":"🧑🏫","tokens_out":7433,"duration_ms":62294,"temperature":0.7,"pith_summary":"AI2T is an interactively teachable agent that lets non-programmers author intelligent tutoring systems by doing what tutors do: demonstrating step-by-step solutions and then grading the agent's own attempts. The paper's central claim is that from roughly 20-30 minutes of such interaction, AI2T can induce hierarchical production-rule programs that are model-tracing complete, meaning they permit every correct next action in a problem state and no incorrect ones. The second claim is that STAND, the agent's precondition-learning algorithm, produces certainty estimates that reliably track actual learning progress, so an author can use low certainty scores as a signal to keep training. If these claims hold, the most difficult part of building a model-tracing tutor, traditionally estimated at 200-300 programming hours per hour of instruction, becomes a short, programming-free teaching session.","feed_headline":"Non-programmers can author complete AI tutors in 20-30 minutes","feed_subtitle":"Demonstrate a solution, grade the agent's attempts, and let its certainty scores tell you when training is done.","key_machinery":"The load-bearing mechanism is STAND, a self-aware precondition induction algorithm that maintains an approximate version space: rather than fitting one classifier, it compresses and tracks the space of all good generalizations consistent with the author's positive and negative feedback, and it derives an instance certainty score from how much those generalizations disagree on a candidate action. Around STAND sits process-learning, a fourth learning mechanism that induces hierarchical task networks from the author's action sequences, so solution ordering starts sequential and is relaxed only when demonstrations show alternative orders, which removes the need for dozens of negative examples to teach ordering constraints.","core_discovery":"On the paper's own terms, AI2T succeeds where prior authoring-by-tutoring systems fell short: it induces correct and complete model-tracing behavior, not merely high-accuracy problem solving. The agent learns hierarchical task networks from demonstrated action sequences, and STAND learns the preconditions that gate when each skill may apply. In simulation, STAND combined with HTN induction reached 100% holdout model-tracing completeness in all 40 repetitions for multicolumn addition and 38 of 40 for fraction arithmetic, outperforming decision trees, random forests, and XGBoost. In user study 2, 8 of 10 untrained participants (six of them self-reported non-programmers) produced 100% complete multicolumn-addition tutors, and 5 of 10 produced 100% complete fraction tutors, with median authoring times around 22 minutes per topic. The paper also claims that STAND's instance certainty is a reliable heuristic: unlike random-forest or XGBoost prediction probabilities, which tracked holdout performance only at chance levels, STAND's certainty scores moved with actual learning gains.","pith_inferences":["Our inference: if STAND's certainty is really a measure of version-space ambiguity, then the same stopping heuristic could be lifted out of AI2T and applied to any interactive task learner that maintains a version space, giving those systems a self-assessment signal they currently lack.","Our inference: the simulation's ideal-user assumption is the fragile link; real authors make uncaught errors, and the paper's own data show that two study-2 participants fell below 50% completeness because of mistakes they never found, so error-detection support is the next bottleneck rather than raw learning efficiency.","Our inference: the paper's comparison against LLMs suggests a complementary architecture in which LLMs generate or parse interface primitives and natural-language explanations while AI2T supplies the trustworthy rule induction, a division of labor neither system achieves alone.","Our inference: a randomized experiment that turns the certainty display on and off for matched authors would directly test whether the display causes completeness, which the paper notes it has not yet run."],"forward_implications":["If the central claim is correct, non-programmers can author two representative model-tracing tutors (multicolumn addition and fraction arithmetic) in about 20-30 minutes per topic, compared with the 200-300 programming hours per hour of instruction traditionally estimated for such tutors.","If certainty scores track learning progress, then an author who waits until every proposed action reaches 100% certainty has a principled stopping rule for training, rather than guessing how many problems are enough.","Because AI2T induces inspectable hierarchical task networks rather than opaque statistical models, the resulting tutor behavior is checkable and does not carry the hallucination risk of LLM-based tutoring.","The same process-learning mechanism that organizes skills into unordered groups (and conditional or optional steps) makes the authoring interface simpler and reduces the number of states an author must grade.","If the tool's scope expands to more domains with adequate primitive libraries, the authoring-by-tutoring approach could replace hand-programming for a wider class of procedural skills."],"supporting_citations":[{"why":"This is the STAND paper; it supplies the precondition-induction algorithm and the instance-certainty measure that carry AI2T's self-awareness claim.","marker":"[57]"},{"why":"This earlier authoring-by-tutoring study defines model-tracing completeness and documents the interaction-design failures that AI2T's design is built to overcome.","marker":"[54]"},{"why":"SimStudent is the canonical prior authoring-by-tutoring system; AI2T claims to improve on its completeness and data efficiency.","marker":"[39]"},{"why":"The Apprentice Learner architecture is the immediate predecessor; prior studies with it fell short of 100% model-tracing completeness even in its creators' hands.","marker":"[37]"},{"why":"This work supplies the cost estimate of 200-300 developer hours per hour of instruction and the graph-based interface authoring approach used to prepare the tutor interfaces.","marker":"[1]"},{"why":"XGBoost is one of the baseline classifiers whose prediction probabilities are compared against STAND's certainty on productive monotonicity.","marker":"[10]"},{"why":"Random forests are the other ensemble baseline, and the paper argues their bagging drops edge cases needed for complete multicolumn addition.","marker":"[5]"},{"why":"The decision-tree implementation serves as the basic when-learning baseline that STAND and process-learning must beat in the simulation experiments.","marker":"[6]"},{"why":"Sierra is the prior hierarchical-task-network learner that required curated lesson sequences; AI2T's process-learning claims to be agnostic to lesson order.","marker":"[52]"}],"fun_headline_variants":["Teach an AI to tutor: 20 minutes, no coding needed","Self-aware AI tutor learns to grade from 20-min demos","Non-programmers build complete AI tutors in half an hour","20-minute demos teach AI to build accurate tutors","AI2T: self-aware tutor authoring, no coding, 20-30 min"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The strongest efficiency and completeness results assume an ideal author who always demonstrates correct actions, always selects the correct formula to explain a demonstration, and grades every proposed action accurately; the paper's own user studies show real authors make uncaught mistakes, so the 20-30 minute numbers depend on that flawless teaching behavior.","fun_headline_variants_meta":{"raw":{"variants":["Teach an AI to tutor: 20 minutes, no coding needed","Self-aware AI tutor learns to grade from 20-min demos","Non-programmers build complete AI tutors in half an hour","20-minute demos teach AI to build accurate tutors","AI2T: self-aware tutor authoring, no coding, 20-30 min"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001078,"raw_usage":{"total_tokens":4523,"prompt_tokens":972,"completion_tokens":3551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":3474}},"tokens_in":588,"tokens_out":3551,"duration_ms":23389,"temperature":1.0,"reasoning_tokens":3474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:41:30.973974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's exact simulation protocol but inject a single incorrect correctness label or one wrong formula annotation at an early training problem; if STAND plus HTN induction then fails to reach 100% holdout completeness in most repetitions, the completeness result depends on flawless teaching rather than on the learning mechanism itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This is the STAND paper; it supplies the precondition-induction algorithm and the instance-certainty measure that carry AI2T's self-awareness claim."},{"cited_title":"2020 in press","cited_arxiv_id":null,"evidence_quote":"This earlier authoring-by-tutoring study defines model-tracing completeness and documents the interaction-design failures that AI2T's design is built to overcome."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SimStudent is the canonical prior authoring-by-tutoring system; AI2T claims to improve on its completeness and data efficiency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Apprentice Learner architecture is the immediate predecessor; prior studies with it fell short of 100% model-tracing completeness even in its creators' hands."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work supplies the cost estimate of 200-300 developer hours per hour of instruction and the graph-based interface authoring approach used to prepare the tutor interfaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"XGBoost is one of the baseline classifiers whose prediction probabilities are compared against STAND's certainty on productive monotonicity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The decision-tree implementation serves as the basic when-learning baseline that STAND and process-learning must beat in the simulation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sierra is the prior hierarchical-task-network learner that required curated lesson sequences; AI2T's process-learning claims to be agnostic to lesson order."}],"review_version":1}