pith. sign in

arxiv: 2604.16738 · v1 · submitted 2026-04-17 · 💻 cs.HC

Teacher-Authored Prompts for Configuring Student-AI Dialogue: K-12 Classroom Implementation

Alex Liu , Min Sun , Lief Esbenshade , Victor Tian , Zachary Zhang , Kevin He This is my paper

Pith reviewed 2026-05-10 07:02 UTC · model grok-4.3

classification 💻 cs.HC
keywords teacher-authored promptsstudent-AI dialogueK-12 classroomDepth of Knowledgegenerative AI in educationpedagogical intentAI orchestrationclassroom deployment
0
0 comments X

The pith

Teacher-authored prompts align most student-AI dialogues with instructional goals while explicit finish lines and guardrails improve cognitive demand.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines a live K-12 deployment of a system where teachers create two prompt layers to shape how AI interacts with students during class. Most of the 1,479 recorded conversations stayed aligned with the teacher's intent, yet a sizable share fell short of the targeted depth of thinking, especially when higher-order reasoning was the goal. Adding clear end points to the prompt and instructions that block direct answers measurably narrowed those shortfalls. The work shows how everyday teacher authoring can turn generic AI tools into structured classroom activities rather than leaving outcomes to chance.

Core claim

Teacher-authored prompt layers translate pedagogical intent into structured student-AI dialogue. Across 77 settings and 16 implementing teachers, 71 percent of conversations were fully on-track and fewer than 1 percent were substantially off-track. However, 38 percent under-reached the teacher-targeted Depth of Knowledge level, rising near 50 percent at DOK 3. Explicit finish lines reduced the DOK gap by 0.22 levels, and no-direct-answers guardrails cut AI final-answer rates by 8.5 percentage points.

What carries the argument

Teacher-authored prompt layers, consisting of a teacher-to-AI setup prompt that acts as an instructional scaffold and a student-facing conversation starter that launches the dialogue while enforcing boundaries.

If this is right

  • Student-AI conversations can be kept largely aligned with teacher intent through authored prompts in live classrooms.
  • Including explicit finish lines in the setup prompt increases the cognitive demand level students reach during the interaction.
  • Adding no-direct-answers instructions reduces the rate at which the AI supplies final answers.
  • Gaps in higher-order thinking remain common even with these prompts, showing limits to prompt-only approaches.
  • Logs and coded analysis can track design-enactment gaps for AI tools used by multiple teachers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same prompt patterns work across other AI platforms, education tools could include templates that automatically suggest finish lines and guardrails based on a teacher's chosen learning goal.
  • The persistent shortfalls at higher DOK levels indicate that prompt instructions alone may need to be paired with model-level changes that favor reasoning steps over quick answers.
  • Similar layering of teacher control could be tested in non-dialogue AI uses, such as AI-assisted writing or simulation tools, to see whether the same alignment benefits appear.

Load-bearing premise

That platform logs, LLM coding validated by humans, and interviews with ten teachers accurately and without bias measure both instructional alignment and Depth of Knowledge levels across the observed conversations.

What would settle it

A follow-up study that records live classroom sessions, collects student work products, and compares independent human-coded Depth of Knowledge levels against the paper's log-based and LLM-coded results to check for systematic differences in the reported gaps.

Figures

Figures reproduced from arXiv: 2604.16738 by Alex Liu, Kevin He, Lief Esbenshade, Min Sun, Victor Tian, Zachary Zhang.

Figure 1
Figure 1. Figure 1: Teacher-Orchestrated Student-AI Dialogue at Classroom Scale and communicates expectations for participation. Students then engage in individualized conversations, with the AI adapting to student responses while remaining anchored to the teacher-authored instructional frame. Workflow is despicted in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of pilot conversations across subject areas. The variation likely reflects differences in teacher framing, assignment integration (graded vs. optional), timing within the school year, and student motivation, factors that characterize authentic classroom implementations. Participation also varied substantially across discussions (range: 1-98 conversations within given TASD activities), indicati… view at source ↗
Figure 3
Figure 3. Figure 3: Summary of teacher prompt authoring patterns across key dimensions (N=94). RQ2. How Teacher-Authored Prompts Shape Enacted Interactions After decomposing teacher-authored prompt content, we now examine how teacher-authored facilitation shaped what actually happened in student-AI conversations, addressing three distinct outcomes, in terms of task alignment (staying on the intended topic), rigor alignment (a… view at source ↗
Figure 5
Figure 5. Figure 5: Types of task deviation among conversations with alignment issues. Shortcut/answer-seeking and off-topic exchanges account for the majority of partial deviations, while substantial off-task behavior is rare [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of AI role assignments in teacher prompts [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of DOK gap categories across conversations (N = 1,362). Positive values indicate under-targeting. Extended Qualitative Findings Qualitative data from semi-structured interviews (N = 10) and teacher reflections contextualized the trace-based pat￾terns reported above. Across sources, teachers framed TASD as the introduction of a ‘third agent” that could support individualized instruction and fee… view at source ↗
Figure 8
Figure 8. Figure 8: Heatmap of target vs. demonstrated DOK levels. The diagonal represents perfect alignment; cells above the diagonal indicate under-targeting [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effect of teacher “no direct answers” guardrail on AI final answer rate. actionable next step, or when responses were overly ver￾bose, some students disengaged. This pattern got further exacerbated when AI responses were long with multiple follow-up questions. Teachers also described this as an enactment problem. When the interaction demanded sub￾stantial interpretation, it competed with increased the stu￾… view at source ↗
read the original abstract

GenAI has rapidly entered instructional and learning settings as a teaching assistant or AI tutor. However, less is known about how pedagogical intent connects to the learning generated within these systems, especially when student-facing AI dialogues are fine-tuned through teacher orchestration in live classrooms. This study examines a classroom deployment of a "Classroom Teaching Aide" (TASD) system, which enables teachers to author both a teacher-to-AI setup prompt (instructional scaffold) and a student-facing conversation starter to launch AI-mediated classroom discussions. We analyze a multi-subject pilot conducted in Spring 2025, involving 20 participating teachers (16 of whom implemented the system), across 39 classrooms and 77 TASD settings, yielding 1,479 student-AI conversations with 878 unique students. Using platform logs, LLM coding with human validation, and post-study teacher interviews (N=10), we characterize teacher authoring choices and link them to enacted student-AI interaction outcomes. In deployment, student-AI conversations were largely aligned with instructional intent: 71% were fully on-track, and fewer than 1% were substantially off-track. However, a persistent design-enactment gap emerged for cognitive demand: 38% of conversations under-reached the teacher-targeted DOK level, approaching 50% when targeting DOK 3. The study also shows that explicit finish lines in the prompt reduced the DOK gap by 0.22 levels (p < .001), and "no direct answers" guardrails reduced AI final-answer rates by 8.5 percentage points. These findings position teacher-authored prompt layers as critical orchestration levers that translate pedagogical intent into structured student-AI dialogue, underscoring both their promise for scalable classroom integration and the need for additional supports to reliably sustain higher-order reasoning during enactment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. This paper describes a pilot study of a 'Classroom Teaching Aide' (TASD) system in K-12 classrooms involving 20 teachers (16 implementers), 39 classrooms, and 77 TASD settings that generated 1,479 student-AI conversations. The authors claim that student-AI dialogues were largely aligned with teacher intent (71% fully on-track), but exhibited a design-enactment gap in cognitive demand with 38% under-reaching the targeted Depth of Knowledge (DOK) level. They report that including explicit finish lines in teacher-authored prompts reduced the DOK gap by 0.22 levels (p < .001) and that 'no direct answers' guardrails reduced AI final-answer rates by 8.5 percentage points. The study positions teacher-authored prompt layers as critical for orchestrating structured student-AI dialogue.

Significance. If the empirical measurements hold, the findings provide concrete evidence on effective prompt design strategies for integrating GenAI into K-12 instruction. The specific quantitative results on prompt elements like finish lines and guardrails offer actionable insights for educators and system designers aiming to maintain high cognitive demand in AI-mediated learning. This contributes to the growing literature on human-AI collaboration in education by demonstrating scalable orchestration mechanisms in authentic classroom settings.

major comments (3)
  1. [Methods (DOK Coding)] The protocol for coding Depth of Knowledge levels in student-AI conversation logs is not adequately described. There is no mention of inter-rater reliability metrics (such as percentage agreement or Cohen's kappa) between LLM-assisted coding and human validation, nor a detailed rubric for applying DOK levels to short dialogue turns rather than traditional student artifacts. This is critical because the headline result of a 0.22-level DOK gap reduction depends directly on the validity and consistency of these measurements.
  2. [Participant Recruitment and Data Collection] Details on how the 20 participating teachers were selected, criteria for the 16 who implemented the system, and any post-hoc data exclusions are absent. Without this, it is difficult to evaluate selection bias or the representativeness of the 1,479 conversations and N=10 interview sample for the reported alignment rates and statistical effects.
  3. [Results (DOK Gap Analysis)] The exact definition and thresholding of 'under-reach' (e.g., how much lower the enacted DOK must be to count as under-reaching) and the regression model used to obtain the 0.22 level reduction (p < .001) are not specified. If LLM coders introduce systematic biases (e.g., under-detecting DOK 3 in brief dialogues), this could artifactually inflate the reported gap and the effect of explicit finish lines.
minor comments (2)
  1. [Abstract] The date 'Spring 2025' appears in the abstract; if this is a future or hypothetical deployment, it should be clarified to avoid confusion about the study timeline.
  2. [Abstract] Some acronyms (e.g., TASD, DOK) are used without initial expansion in the abstract, which could hinder readability for readers unfamiliar with the terms.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive comments, which help strengthen the methodological transparency of our pilot study. We address each major point below, indicating where revisions will be made to the manuscript.

read point-by-point responses

  1. Referee: The protocol for coding Depth of Knowledge levels in student-AI conversation logs is not adequately described. There is no mention of inter-rater reliability metrics (such as percentage agreement or Cohen's kappa) between LLM-assisted coding and human validation, nor a detailed rubric for applying DOK levels to short dialogue turns rather than traditional student artifacts. This is critical because the headline result of a 0.22-level DOK gap reduction depends directly on the validity and consistency of these measurements.

    Authors: We agree that additional detail on the DOK coding protocol is needed. The revised manuscript will include the complete rubric adapted for short dialogue turns (with examples of DOK 1-4 applications), the exact human validation procedure, and inter-rater reliability statistics (Cohen's kappa and percentage agreement) between the LLM coder and two human raters. These additions will directly support the reported DOK gap findings. revision: yes

  2. Referee: Details on how the 20 participating teachers were selected, criteria for the 16 who implemented the system, and any post-hoc data exclusions are absent. Without this, it is difficult to evaluate selection bias or the representativeness of the 1,479 conversations and N=10 interview sample for the reported alignment rates and statistical effects.

    Authors: We will expand the Methods section to describe recruitment (via district partnerships and professional development networks), inclusion criteria for the 16 implementers (completion of a 2-hour training session and submission of at least one TASD setting), and any post-hoc exclusions (e.g., conversations with fewer than three turns or incomplete logs). This will allow readers to assess selection bias and generalizability. revision: yes

  3. Referee: The exact definition and thresholding of 'under-reach' (e.g., how much lower the enacted DOK must be to count as under-reaching) and the regression model used to obtain the 0.22 level reduction (p < .001) are not specified. If LLM coders introduce systematic biases (e.g., under-detecting DOK 3 in brief dialogues), this could artifactually inflate the reported gap and the effect of explicit finish lines.

    Authors: We will add precise definitions: under-reach is defined as enacted DOK at least one level below the teacher-targeted level. The 0.22-level reduction comes from a linear mixed-effects regression with fixed effects for prompt features (finish lines, guardrails) and random intercepts for classroom and teacher, controlling for subject and grade. We will also report human validation agreement rates and discuss potential LLM biases for brief dialogues as a limitation, with sensitivity analyses if feasible. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical pilot with data-driven results

full rationale

The paper is a straightforward empirical pilot study of a classroom deployment. All headline results (71% on-track alignment, 0.22-level DOK gap reduction with p<.001, 8.5 pp reduction in final-answer rates) are computed directly from platform logs of 1,479 conversations, LLM coding plus human validation, and N=10 interviews. No equations, fitted parameters renamed as predictions, self-citations used as load-bearing uniqueness theorems, or ansatzes appear in the reported chain. The derivation is therefore self-contained against external benchmarks and does not reduce to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on empirical observations from a classroom pilot; no free parameters are fitted to produce the headline percentages, and no new entities are postulated.

axioms (1)
  • domain assumption Depth of Knowledge (DOK) levels provide a valid and reliable way to classify cognitive demand in student-AI conversations
    The study treats DOK as the benchmark for whether conversations under-reached teacher intent.

pith-pipeline@v0.9.0 · 5646 in / 1416 out tokens · 53792 ms · 2026-05-10T07:02:30.471912+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 37 canonical work pages

  1. [1]

    Springer, 2023. Robyn M. Gillies. Promoting academically productive student dialogue during collaborative learning.International Journal of Educational Research, 97:200–209, 2019. Stian Haaklev, Jim Slotta, Niels Pinkwart, and Pierre Dillenbourg. Orchestration graphs: Enabling rich social pedagogical scenarios in moocs. InProceedings of the Fourth ACM Con...

    work page 2023

  2. [2]

    URL https://bera-journals.onlinelibrary

    doi: https://doi.org/10.1111/bjet.13372. URL https://bera-journals.onlinelibrary. wiley.com/doi/abs/10.1111/bjet.13372. Jessica Leek et al. Teacher time allocation in secondary classrooms: Evidence from a large-scale observational study.Teaching and Teacher Education, 139:104433, 2024. Ang´elique L ´etourneau, Marion Deslandes Martineau, Patrick Charland,...

    work page doi:10.1111/bjet.13372 2024

  3. [3]

    Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen

    Routledge, London, 2012. Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. Socraticlm: Exploring socratic personalized teaching with large language models.Advances in Neural Information Processing Systems, 37:85693–85721, 2024. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neub...

    work page arXiv 2012

  4. [4]

    Can you walk us through how you implemented the AI-supported in-class discussions?

    work page

  5. [5]

    How did you support students in engaging with the tool in your class setting?

    work page

  6. [6]

    When some students opted out, how did you facilitate discussion with and without AI at the same time? 1.2 Changes across tries

    work page

  7. [7]

    Between your first and later attempts, did you change any instructional strategies? What changed, and why?

    work page

  8. [8]

    What strategies worked well, and what did not work as well?

    work page

  9. [9]

    From your perspective, what types of AI support or feedback were most helpful for students’ learning during discussions? Prepared usingsagej.cls 20 Journal Title XX(X) 1.3 Feature utility and tool vision

    work page

  10. [10]

    Which features in the AI discussion tool were most helpful? Why?

    work page

  11. [11]

    Which features were least helpful or confusing?

    work page

  12. [12]

    If you could design an ideal version of this tool for in-class discussions, what would it look like?

    work page

  13. [13]

    What functionalities or supports would you add to better support classroom use? 1.4 Teacher and student perspectives

    work page

  14. [14]

    Were there aspects of the tool where your perspective differed from students’ perspectives (e.g., you liked something students disliked, or vice versa)?

    work page

  15. [15]

    Did the tool ever feel disruptive in the classroom, such as standing between students and teachers? In what situations?

    work page

  16. [16]

    What was the biggest challenge in using this feature in your classroom?

    work page

  17. [17]

    What use cases would you recommend for this feature?

    work page

  18. [18]

    What advice would you give to teachers who are new to AI? What advice would you give to classrooms that are new to AI? Section 2: AI Assessment and Feedback (20 minutes) 2.1 Implementation walkthrough

    work page

  19. [19]

    Can you walk us through how you implemented the AI-supported assessments and feedback?

    work page

  20. [20]

    What kinds of tasks or assessments did you apply the tool to? 2.2 Changes across tries

    work page

  21. [21]

    Did you make changes in your assessment or feedback strategies between attempts? What helped, and what did not?

    work page

  22. [22]

    What additional facilitation did you need in order to support students who adopted versus did not adopt AI for an assessment-related activity? 2.3 Workflow and tool vision

    work page

  23. [23]

    What aspects of the AI assessment workflow felt smooth?

    work page

  24. [24]

    Were any steps clunky or unintuitive?

    work page

  25. [25]

    How should the workflow be streamlined to improve the user experience? 2.4 Feature utility and future use cases

    work page

  26. [26]

    Which features in the assessment and feedback tool were most helpful?

    work page

  27. [27]

    Which features were least helpful or underwhelming?

    work page

  28. [28]

    What future classroom use cases do you envision for this kind of tool? Section 3: Student Growth Insights (10 minutes) 3.1 Usefulness and integration

    work page

  29. [29]

    Did you use the student growth insights or AI- generated summaries? If so, how?

    work page

  30. [30]

    Did any of the insights inform your instructional decisions or student support? Please describe an example

    work page

  31. [31]

    Were the insights easy to understand and apply? Why or why not? Section 4: Broader reflections on contextual fit (10 minutes) 4.1 Educational settings fit

    work page

  32. [32]

    In your opinion, what kinds of classroom settings or educational contexts benefit most from these AI features?

    work page

  33. [33]

    Are there settings where these tools might be less effective or would require significant adaptation? 4.2 Closing reflections

    work page

  34. [34]

    Is there anything else you would like to share about your experience using the platform or ideas for future development? Permission for public-facing educator stories (as applicable) If the study team plans to share educator use cases publicly (e.g., newsletters or social media), the interviewer asked:

    work page

  35. [35]

    If we make use cases publicly available, would you prefer to be credited or to remain anonymous?

    work page

  36. [36]

    With your review, edits, and consent, would that be okay?

    Based on todays interview, we may draft a brief educator testimony or feature story about your classroom use of the platform. With your review, edits, and consent, would that be okay?

    work page

  37. [37]

    no direct answers

    We sometimes feature educators who have substan- tially explored the platform or supported development as educational innovation partners. Would you be will- ing to be featured? We would send any draft content to you for review before publishing. For scholarly publications, teacher identities are kept anonymous and results are reported in aggregate, consi...

    work page