Pith. sign in

REVIEW 2 major objections 3 minor 21 references

BoilerSketch claims that a supervised, diagram-first GenAI workflow—chat plus sketches plus Mermaid diagrams under TA oversight—can serve as a credible first-response layer for routine CS1/early CS2 conceptual questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-05 00:10 UTC pith:BXA67ZXW

load-bearing objection A modest, honestly scoped practice paper on supervised diagram-first GenAI support; the evaluation only measures staff perceptions, so the 'bounded' claim rests on design rather than evidence, but it deserves a serious referee. the 2 major comments →

arxiv 2608.00844 v1 pith:BXA67ZXW submitted 2026-08-01 cs.CY cs.HC

BoilerSketch: A TA-Supervised, Diagram-First GenAI Practice for Structured Diagrams in CS1/Early CS2

Ethan Dickey , Vivan Tiwari , Anvit Sinha , Andres Bejarano This is my paper
classification cs.CY cs.HC
keywords computing educationgenerative artificial intelligencediagram-first tutoringMermaid diagramsteaching assistant supervisionprogram visualizationCS1/CS2human-in-the-loop
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

BoilerSketch tries to show that AI help in large introductory programming courses works better when it is diagram-first, deliberately narrow, and supervised. Instead of generating free-form images or executable code, the system forces the model to emit Mermaid diagram syntax, renders it server-side, and keeps teaching assistants accountable for the interaction. The evidence is a 45-minute expert evaluation with 21 instructional staff: two-thirds rated the tool at least moderately helpful for conceptual understanding and at least moderately useful for typical support tasks. The authors conclude that this configuration is a credible first-response layer for routine, diagram-amenable questions, while deeper misconception diagnosis remains with humans. The stakes are a concrete, replicable pattern for using GenAI in staffed CS support without surrendering instructional judgment.

Core claim

The paper claims that a deliberately narrow configuration—chat plus pen whiteboard, a prompt that forbids code and restricts visuals to Mermaid, deterministic server-side rendering, and TAs as accountable supervisors—can serve as a credible first-response layer for routine conceptual questions in CS1 and early CS2. The supporting evidence is an expert evaluation in which 14 of 21 instructional staff rated the system at least moderately helpful for conceptual understanding and the same fraction at least moderately useful for typical support tasks, with no respondent choosing the worst category. The authors position the result as validating a supervised, diagram-first practice rather than an a

What carries the argument

The load-bearing mechanism is a prompt-as-policy pipeline. A base system prompt defines the assistant as a concise, integrity-bound teaching assistant that never produces executable code or debug corrections; a separate visualization subprompt is invoked only when a diagram helps, emitting a single block of Mermaid syntax—a text-based diagram language rendered into images. The server validates and renders that block deterministically, so students see images rather than syntax. A dual-pane tablet interface lets students ask questions in chat or attach pen/sketch/photo context, and TAs monitor sessions with three intervention options: correct, extend, or escalate. This combination is what boun

Load-bearing premise

The central assumption is that 45 minutes of hands-on use by 21 teaching assistants from one course, answering representative questions, is enough for their judgments of 'helpful' and 'useful' to stand for real educational value in live student support.

What would settle it

A systematic audit of BoilerSketch-generated diagrams on a fixed bank of CS1/CS2 questions, checking whether each Mermaid diagram correctly represents the concept and whether a novice would be misled, would settle the 'credible first-response layer' claim: if a substantial share of diagrams are wrong or misleading, the central claim collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Courses can adopt the practice by treating the prompt as a reviewed policy artifact that encodes academic-integrity boundaries and by pairing it with deterministic diagram rendering.
  • Routine diagram-amenable questions—recursion traces, tree traversals, pointer relationships, and DP tables—can receive faster first-pass explanations while TAs keep final authority.
  • Structured diagram output is sufficient for instructional value; polished free-form images are not necessary.
  • Explicit escalation rules let staff correct, extend, or redirect to live help, preserving instructional authority.
  • The practice is transferable to smaller courses where one instructor can supervise a handful of sessions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pattern holds beyond programming, the same scope-and-escalation design could apply to other diagram-heavy disciplines, such as circuit analysis or data-flow modeling in systems courses, where novice-friendly visuals are also scarce.
  • The 45-minute expert session cannot reveal whether moderate ratings persist under real student load; a natural next test is a randomized deployment comparing diagram correctness, student learning, and TA workload against human-only support.
  • The Mermaid-only constraint likely does double duty: it makes outputs predictable, but it also caps the visual vocabulary, so advanced topics may fail precisely because their diagrams exceed Mermaid's expressiveness—a trade-off testable by logging topic-to-repair rates.
  • The paper measures perceived usefulness, not actual use; monitoring whether TAs actually delegate routine questions to the tool in live labs would be a stronger behavioral test of the first-response-layer claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. BoilerSketch is a TA-supervised, diagram-first GenAI practice for CS1/early CS2 support. The system combines a chat/whiteboard tablet interface, a prompt that constrains the model to conceptual explanation and Mermaid-structured diagrams, deterministic server-side rendering, and an explicit human-in-the-loop workflow in which TAs monitor and intervene. The paper reports a 45-minute expert evaluation with 21 instructional staff from Purdue's CS251. On post-use surveys, 14/21 (66.7%) rated the system at least moderately helpful for conceptual understanding, and 14/21 rated it at least moderately useful for typical support tasks. The authors conclude that BoilerSketch is a bounded, replicable instructional practice that can serve as a credible first-response layer for routine diagram-amenable questions, while explicitly disclaiming evidence on student learning, diagram correctness, and operational throughput.

Significance. If the design claims hold, the paper contributes a useful, transferable design pattern: prompt-as-policy guardrails, structured Mermaid output, and TA escalation rules together provide a bounded alternative to both unconstrained chatbots and non-conversational program visualizations. The paper is honest about its scope: it evaluates perceived usefulness and fit, not learning gains or correctness. It also grounds the design in relevant CER literature and provides concrete adoption guidance. However, the empirical support is limited to a single convenience sample of 21 staff in one session, and two load-bearing aspects of the practice—the actual behavior of the prompt guardrails and the functioning of TA supervision—are not directly evaluated. The significance is therefore real but conditional on future validation of those mechanisms.

major comments (2)
  1. [§III.D, §III.E, §VI.E] The paper's central claim that BoilerSketch is a 'bounded' practice rests on prompt guardrails preventing executable code and direct debugging, and on structured Mermaid diagrams being pedagogically usable. However, Section VI.E explicitly states that the evaluation did not measure diagram correctness systematically, and Section IV.F states the study was not designed to measure comprehensive correctness. No audit of model outputs for policy violations (e.g., leaked code or direct debugging) is reported. Without such evidence, 'bounded' is a design intention rather than an empirically supported property. I recommend adding a response audit: sample the session logs, classify outputs for code leaks, direct debugging, and Mermaid renderability/correctness, and report the rates. This would directly support or qualify the boundedness claim.
  2. [§III.F, §IV.D] The practice is defined as TA-supervised, with TAs monitoring sessions, correcting responses, and escalating when needed. Yet the evaluation asked participants to interact as students seeking help; it did not ask them to act as supervisors, nor did it observe real TA monitoring, intervention, or escalation decisions (Section IV.D). The staff ratings therefore reflect the learner-facing artifact, not the supervision workflow that is central to the proposed practice. A small simulation or staged deployment where participants play the TA role—deciding which responses to correct, extend, or escalate—would provide evidence on whether the human-in-the-loop model is feasible and effective in practice.
minor comments (3)
  1. [Abstract/Introduction] Typos: 'thenotional machine' should be 'the notional machine'; check spacing throughout. Also, 'very/moderately not helpful/useful' in Section IV.E is ambiguous; clarify the exact response scale labels.
  2. [§III.D] The paper mentions 'OpenAI's o1' but does not specify the exact model/version beyond the prototype; this matters for replicability. Please provide the model identifier and date as available.
  3. [References] Some reference entries have inconsistent spacing (e.g., 'V . Kann' in [3] and [24]). Also, the supplementary material link is not shown; ensure it is accessible.

Circularity Check

0 steps flagged

No significant circularity: the central claim is a direct survey result, not derived from prior work or fitted parameters.

full rationale

BoilerSketch's central claim—that instructional staff judged the system helpful and useful—is measured directly via a post-use survey (Tables II and III), not derived from the authors' prior work or from any fitted parameter. The paper explicitly scopes its evidence: Section IV.F states the study 'was not designed to measure ... comprehensive correctness across all topic types,' and Section VI.E concedes that 'the evaluation did not measure diagram correctness systematically across prompts or topics.' These are acknowledged limitations and missing support for the stronger 'bounded' characterization, but they are not circular reductions: the perceived-usefulness result does not presuppose diagram correctness, and the 'bounded' claim is presented as a design stance rather than as something the data establish. Self-citations to BoilerTAI [13] and AI-Lab [23] are used only to position the work within existing literatures; they do not carry the load of the evaluation or the design conclusions. No equation, definition, or statistical fit makes the claimed outcome equal to an input. The paper therefore exhibits no self-definitional, fitted-prediction, or self-citation-dependent circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No numerical free parameters are fitted; the ledger records domain assumptions about visual learning, LLM reliability, TA oversight, self-report validity, and prompt enforcement. These are load-bearing for the practice but not empirically established by the study.

axioms (5)
  • domain assumption Diagrams and program visualizations aid novice comprehension when integrated into pedagogy.
    Invoked in Section II-A via Sorva's notional machine and Naps et al.; foundational for the diagram-first premise, but not tested in this study.
  • domain assumption A large language model can generate valid and pedagogically useful Mermaid syntax under a constrained prompt.
    Assumed throughout Section III-E; the evaluation did not measure diagram correctness or rendering failure rates.
  • domain assumption TA supervision can reliably detect and correct errors or off-policy responses from the AI in a live setting.
    Central to the human-in-the-loop design (Section III-F); the 45-minute session did not exercise real-time monitoring or escalation.
  • ad hoc to paper Self-report by instructional staff is a valid proxy for instructional fit and future student benefit.
    The evaluation uses expert judgment as evidence of usefulness (Section IV-F), without linking staff ratings to student outcomes.
  • ad hoc to paper The base prompt's guardrails effectively prevent code generation and direct debugging in practice.
    The paper states the prompt forbids certain outputs (Section III-D), but no adversarial testing or content audit is reported.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of BoilerSketch: A TA-Supervised, Diagram-First GenAI Practice for Structured Diagrams in CS1/Early CS2." pith.science (2026). https://pith.science/paper/BXA67ZXW

@misc{pith2026260800844,
  author       = {Pith},
  title        = {Pith review of: BoilerSketch: A TA-Supervised, Diagram-First GenAI Practice for Structured Diagrams in CS1/Early CS2},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXA67ZXW}},
  note         = {Machine review of arXiv:2608.00844}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This innovative practice full paper presents BoilerSketch, a TA-supervised, diagram-first GenAI practice and tablet interface for providing structured visual explanations in CS1 and early CS2 support settings. Large early computing courses routinely face a support bottleneck during labs and office hours because many student questions are best answered with a diagram rather than additional text, yet most AI tutoring tools remain text-forward and unreliable at producing accurate, pedagogically useful visuals. BoilerSketch addresses this gap through a dual-pane interaction model that combines chat with a pen-enabled whiteboard for student sketches and a prompting strategy that constrains the model to generate structured, renderable Mermaid diagrams rather than free-form images. To preserve academic integrity, the system is intentionally scoped to conceptual explanation: it forbids executable code and code-level debugging and uses a human-in-the-loop workflow in which teaching assistants remain accountable supervisors who can monitor sessions and intervene when responses require correction, deeper probing, or escalation to live help. We report a 45-minute expert evaluation with 21 instructional staff from a large programming course who used BoilerSketch on representative questions and completed a post-use survey. Two-thirds rated the system at least moderately helpful for conceptual understanding and at least moderately useful for typical support tasks. Staff saw the strongest value in routine diagram-based explanations and noted limits in diagram depth and applicability to more advanced topics. We conclude with practical guidance for adopting supervised, diagram-first GenAI support in early computing courses, emphasizing scope-and-escalation rules, prompt-as-policy guardrails, and reliable structured diagram rendering.

Figures

Figures reproduced from arXiv: 2608.00844 by Andres Bejarano, Anvit Sinha, Ethan Dickey, Vivan Tiwari.

Figure 1
Figure 1. Figure 1: Representative BoilerSketch interactions. The top panel shows a text [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

  1. [1]

    HelpMe: Student help seeking using office hours and email,

    K. Wang and R. Lawrence, “HelpMe: Student help seeking using office hours and email,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 1388–1394. [Online]. Available: https://doi.org/10.1145/3626252.3630867

    work page arXiv 2024

  2. [2]

    Challenges faced by teaching assistants in computer science education across Europe,

    E. Riese, M. Lor ˚as, M. Ukrop, and T. Effenberger, “Challenges faced by teaching assistants in computer science education across Europe,” inProceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V . 1, ser. ITiCSE ’21. New York, NY , USA: Association for Computing Machinery, 2021, p. 547–553. [Online]. Available...

    work page arXiv 2021

  3. [3]

    Teaching assistants’ experiences of tutoring and assessing in computer science education,

    E. Riese and V . Kann, “Teaching assistants’ experiences of tutoring and assessing in computer science education,” in2020 IEEE Frontiers in Education Conference (FIE), 2020, pp. 1–9

    work page 2020

  4. [5]

    Notional machines and introductory programming education,

    J. Sorva, “Notional machines and introductory programming education,” ACM Trans. Comput. Educ., vol. 13, no. 2, Jul. 2013. [Online]. Available: https://doi.org/10.1145/2483710.2483713

    work page arXiv 2013

  5. [6]

    Exploring the role of visualization and engagement in computer science education,

    T. L. Naps, G. R ¨oßling, V . Almstrum, W. Dann, R. Fleischer, C. Hundhausen, A. Korhonen, L. Malmi, M. McNally, S. Rodger, and J. A. Vel ´azquez-Iturbide, “Exploring the role of visualization and engagement in computer science education,” inWorking Group Reports from ITiCSE on Innovation and Technology in Computer Science Education, ser. ITiCSE-WGR ’02. ...

    work page arXiv 2002

  6. [7]

    A review of generic program visualization systems for introductory programming education,

    J. Sorva, V . Karavirta, and L. Malmi, “A review of generic program visualization systems for introductory programming education,”ACM Trans. Comput. Educ., vol. 13, no. 4, Nov. 2013. [Online]. Available: https://doi.org/10.1145/2490822

    doi:10.1145/2490822 2013

  7. [9]

    Automatic generation of programming exercises and code explanations using large language models,

    S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, “Automatic generation of programming exercises and code explanations using large language models,” inProceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1, ser. ICER ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 27–43. [Online]. Available: ht...

    work page arXiv 2022

  8. [11]

    Experiences from using code explanations generated by large language models in a web software development e-book,

    S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen, “Experiences from using code explanations generated by large language models in a web software development e-book,” inProceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2023. New York, NY , USA: Association for Computing...

    work page arXiv 2023

  9. [13]

    BoilerTAI: A platform for enhancing instruction using generative AI in educational forums,

    A. Sinha, S. Goyal, Z. Sy, R. Kuperus, E. Dickey, and A. Bejarano, “BoilerTAI: A platform for enhancing instruction using generative AI in educational forums,” in2024 IEEE Frontiers in Education Conference (FIE). Los Alamitos, CA, USA: IEEE Computer Society, 2024, pp. 1–8. [Online]. Available: https://doi.ieeecomputersociety.org/ 10.1109/FIE61694.2024.10893137

    work page arXiv 2024

  10. [14]

    Prompt problems: A new programming exercise for the generative AI era,

    P. Denny, J. Leinonen, J. Prather, A. Luxton-Reilly, T. Amarouche, B. A. Becker, and B. N. Reeves, “Prompt problems: A new programming exercise for the generative AI era,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 296–302. [Onli...

    work page arXiv 2024

  11. [15]

    ’Like a Nesting Doll’: Analyzing recursion analogies generated by CS students using large language models,

    S. Bernstein, P. Denny, J. Leinonen, L. Kan, A. Hellas, M. Littlefield, S. Sarsa, and S. Macneil, “’Like a Nesting Doll’: Analyzing recursion analogies generated by CS students using large language models,” in Proceedings of the 2024 on Innovation and Technology in Computer Science Education V . 1, ser. ITiCSE 2024. New York, NY , USA: Association for Com...

    work page arXiv 2024

  12. [17]

    Debugging with an AI tutor: Investigating novice help-seeking behaviors and perceived learning,

    S. Yang, H. Zhao, Y . Xu, K. Brennan, and B. Schneider, “Debugging with an AI tutor: Investigating novice help-seeking behaviors and perceived learning,” inProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1, ser. ICER ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 84–94. [Online]. Availa...

    work page arXiv 2024

  13. [18]

    Bridging novice programmers and LLMs with interactivity,

    T. Y . Yeh, K. Tran, G. Gao, T. Yu, W. O. Fong, and T.-Y . Chen, “Bridging novice programmers and LLMs with interactivity,” inProceedings of the 56th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSETS 2025. New York, NY , USA: Association for Computing Machinery, 2025, p. 1295–1301. [Online]. Available: https://doi.org/10.1145/3641...

    work page arXiv 2025

  14. [19]

    Lau and P

    S. Lau and P. Guo, “From ’ban it till we understand it’ to ’resistance is futile’: How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot,” inProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1, ser. ICER ’23. New York...

    work page arXiv 2023

  15. [20]

    Computing education in the era of generative ai,

    P. Denny, J. Prather, B. A. Becker, J. Finnie-Ansley, A. Hellas, J. Leinonen, A. Luxton-Reilly, B. N. Reeves, E. A. Santos, and S. Sarsa, “Computing education in the era of generative ai,”Commun. ACM, vol. 67, no. 2, p. 56–67, Jan. 2024. [Online]. Available: https://doi.org/10.1145/3624720

    doi:10.1145/3624720 2024

  16. [21]

    Bull and A

    C. Bull and A. Kharrufa, “Generative artificial intelligence assistants in software development education: A vision for integrating generative ar- tificial intelligence into educational practice, not instinctively defending against it,”IEEE Software, vol. 41, no. 2, pp. 52–59, 2024

    work page 2024

  17. [22]

    Feasibility study of augmenting teaching assistants with AI for CS1 programming feedback,

    U. Z. Ahmed, S. Sahai, B. Leong, and A. Karkare, “Feasibility study of augmenting teaching assistants with AI for CS1 programming feedback,” inProceedings of the 56th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSETS 2025. New York, NY , USA: Association for Computing Machinery, 2025, p. 11–17. [Online]. Available: https://doi.org...

    work page arXiv 2025

  18. [23]

    AI-Lab: A framework for intro- ducing generative artificial intelligence tools in computer programming courses,

    E. Dickey, A. Bejarano, and C. Garg, “AI-Lab: A framework for intro- ducing generative artificial intelligence tools in computer programming courses,”SN Computer Science, vol. 5, no. 6, p. 720, 2024

    2024

  19. [24]

    Training teaching assistants by offering an introductory course,

    E. Riese and V . Kann, “Training teaching assistants by offering an introductory course,” inProceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1, ser. SIGCSE

    work page

  20. [25]

    Mermaid: Generate diagrams from markdown-like text,

    K. Sveidqvist and Contributors to Mermaid, “Mermaid: Generate diagrams from markdown-like text,” https://github.com/mermaid-js/ mermaid, Dec. 2014, computer software, MIT License

    work page 2014

  21. [2022]

    New York, NY , USA: Association for Computing Machinery, 2022, p. 745–751. [Online]. Available: https://doi.org/10.1145/3478431. 3499270

    work page doi:10.1145/3478431 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.