Pith. sign in

REVIEW 4 major objections 5 minor 71 references

SimStep: Chain-of-Abstractions for Incremental Specification and Debugging of AI-Generated Interactive Simulations

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that decomposing AI code generation into four graph checkpoints lets non-programmers author and debug interactive simulations without writing code.

desk verdict A well-documented system paper whose core 'targeted revision' mechanism is never tested for edit propagation; the qualitative evidence is real, but the control claim outruns the data. read the letter →

arxiv 2507.09664 v1 pith:XNGHFLRJ submitted 2025-07-13 cs.HC cs.AI

Zoe Kaputa , Anika Rajaram , Vryan Almanon Feliciano , Zhuoyue Lyu , Maneesh Agrawala , Hari Subramonyam This is my paper
classification cs.HCcs.AI
keywords Chain-of-Abstractionsprogramming-by-promptingend-userprogramminginteractivesimulationauthoringLLMdebuggingeducationaltechnologydistributedcognitiongraphabstractions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Programming-by-prompting lets non-programmers describe what they want but strips away traceability and testability. SimStep's Chain-of-Abstractions (CoA) framework inserts four graph checkpoints between a natural-language prompt and the final simulation code. Teachers can inspect, edit, and validate each graph, and an inverse correction process surfaces hidden model assumptions for targeted fixes. The paper argues this recovers control for educators and generalizes to other end-user programming domains.

What carries the argument

The chain-of-abstractions is a sequence of four node-link graphs (concepts, scenario, learning goals, and UI interaction) that externalize the user's intent at task boundaries. Each graph acts as a checkpoint for the operations Inspect, Refine, Validate, and Direct, and the underspecification resolution engine provides inverse correction by turning final code back into editable abstraction widgets.

What would settle it

Have a group of teachers use SimStep with several simulation specifications, plant known factual errors into the Scenario and Learning Goal graphs, and check whether users detect and fix those errors at the checkpoints; if detection is at chance level, the claim that checkpoints make assumptions visible collapses.

Watch

Extended reading notes

Core claim

The central claim is that the Chain-of-Abstractions framework turns prompt-to-code synthesis into a human-guided sequence of task-aligned representations. SimStep instantiates this with a Concept Graph, Scenario Graph, Learning Goal Graph, and UI Interaction Graph, each a node-link diagram the user can inspect and refine before code is generated. A complementary inverse process realizes code assumptions back into editable abstractions, so debugging happens at the level of intent rather than syntax. The paper presents educator evaluations and fidelity ratings as evidence that this restores authoring control and interpretability.

Load-bearing premise

The framework's usefulness rests on the four graph abstractions being accurate enough that editing them reveals errors rather than masking them; the paper's own fidelity ratings put the middle graphs at 6.65 and 7.08 out of 10.

Editorial extensions

If this is right

  • Teachers can author and refine interactive simulations by editing graph nodes and links instead of HTML or JavaScript.
  • Model assumptions that would otherwise surface only as bugs are exposed at abstraction checkpoints, allowing targeted correction before code is generated.
  • Automated test-case generation and headless-browser verification can catch JavaScript, logic, and UI errors before a teacher sees them.
  • The CoA structure applies beyond education, for example to data scientists moving from analysis goals to visual outputs or game designers from core mechanics to interaction feedback.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the fidelity of the middle graphs (Scenario and Learning Goal) is the critical bottleneck; improving those two abstractions may matter more than any other system change.
  • A controlled comparison against direct prompting, measuring objective simulation correctness rather than subjective usability, would isolate whether the checkpoints themselves add value.
  • The framework's generalizability depends on finding task-aligned abstraction chains in each new domain, and the paper offers no recipe for discovering those chains automatically.
  • A testable prediction that goes beyond the reported studies is that teachers given a graph with a planted factual error will detect and correct it at the checkpoint more often than they would by re-prompting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Chain-of-Abstractions (CoA), a framework for programming-by-prompting in which a natural-language prompt is decomposed into a sequence of editable graph abstractions (Concept Graph, Scenario Graph, Learning Goal Graph, User Interaction Graph) before code generation, plus an inverse correction process that surfaces model assumptions. The authors instantiate the framework in SimStep, an authoring environment for educators creating interactive simulations, and describe the system architecture, prompting details, and automated testing pipeline. They report a usability study with N=11 educators using PSSUQ, NASA-TLX, and Cognitive Dimensions of Notations, and a technical fidelity evaluation over 13 curated specifications. The central claim is that CoA gives non-programmers greater authoring control and interpretability than direct programming-by-prompting.

Significance. The paper has clear strengths: a fully implemented system, a detailed prompt-engineering appendix, an automated headless-browser testing workflow, and a user study using standard instruments. The design idea of task-aligned graph checkpoints is well motivated by distributed cognition and is a plausible direction for end-user programming. However, the evidence presented does not yet establish the headline claims of 'greater control' or of effective checkpoint-based debugging. The single-arm usability study and single-rater fidelity evaluation are suggestive but not conclusive, and the core edit-propagation mechanism is untested. If the authors add a propagation study and a baseline comparison, or substantially soften the claims, the contribution would be significant for AI-assisted content creation and end-user programming research. As written, the gap between the claims and the evidence is the main barrier.

major comments (4)
  1. [Section 6, Table 2] The fidelity evaluation uses a single learning-design student rating 13 specifications on one 1-10 question ('How closely does this abstraction adhere to the previous abstraction and user inputs?'). The Scenario Graph and Learning Goal Graph, which are the central checkpoints of the CoA pipeline, receive mean scores of 6.65 and 7.08 with standard deviations above 2. A single subjective rater, combined with a question that conflates adherence to the previous abstraction with fidelity to user intent, is too weak to support the claim that the middle abstractions reliably encode teacher intent. The scorer's own note about potential topic bias reinforces this. The paper should report multiple raters with inter-rater reliability and should separately assess agreement with the intended learning content, not only consistency with the previous abstraction.
  2. [Section 3.2 and Section 7.3] The core mechanism of 'targeted revision' is that user edits to an early abstraction change later abstractions and final code. Section 4.1 describes each abstraction as generated from the previous one via LLM prompting, and Section 3.2 describes direct manipulation widgets, but no experiment, trace, or case study demonstrates that an edit to a Concept Graph, Scenario Graph, or Learning Goal Graph propagates to downstream abstractions and the final simulation. Section 7.3 explicitly concedes that 'current LLMs ... often fail to maintain consistency across stages.' The anchor example in Section 3.1 (editing the Concept Graph to correct the buoyancy equation) is presented as a scenario, not as evidence. Without a propagation test, the central claim of authoring control through CoA checkpoints is unverified; the inverse correction path in Section 4.2 is similarly without empirical support.
  3. [Section 5 and Abstract] The abstract claims that CoA 'enables greater authoring control and interpretability,' but the user study is single-arm with N=11 and no comparison condition. PSSUQ 4.66, TLX 2.64, and CDN 4.61 are descriptive scores from one system; they cannot substantiate 'greater' control relative to direct prompt-to-code generation or to prior systems. The paper should either add a comparative condition (e.g., a within-subjects direct-prompting baseline) or revise the abstract and conclusions to claim perceived usability and interpretability without the comparative 'greater.'
  4. [Section 2.2] The formal monotonicity chain Ω(P) ⊃ Ω(B1) ⊃ Ω(B1') ⊃ ... ⊃ Ω(C) and the corresponding inequalities for U are asserted without proof and are questionable as stated. The sets Ω(X) are not constructively defined for heterogeneous representations such as a natural-language prompt, graph abstractions, and code, and user refinement may add nodes or links, which can expand rather than shrink the set of compatible implementations. The formalism should be presented as a design goal or empirically tested, not as a mathematical consequence of the framework.
minor comments (5)
  1. [Throughout] There are several typos and grammatical slips: 'promting' in Section 4.1.2, 'undespecification' in the opening of Section 4, 'cognatively' in Section 4.4, and 'the the notations' in Section 5.2.3. A copyedit pass is needed.
  2. [Section 5.2.2] The NASA-TLX score is reported as unweighted and the physical demand dimension was not assessed. This modification of the standard instrument should be stated explicitly in the method, with a discussion of how the resulting score compares to published TLX norms.
  3. [Table 2 and Section 4.1.4] The naming is inconsistent: Section 4.1.4 defines the 'User Interaction Graph,' while Table 2 and several later passages call it the 'UI Graph.' Please use one name consistently.
  4. [Figure 11] The comparison of CoA-generated simulations to direct prompt-to-code outputs is presented without criteria for selecting the examples or a systematic evaluation. If this figure is retained, the caption should describe how representative examples were chosen and what the comparison is intended to show.
  5. [Section 4.3.3] The statement that the LLM 'does a very reliable job' of fixing logical errors is not supported by reported data. Either add pass-rate metrics for the automated repair pipeline or soften this claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CoA pipeline is a system design with independent user and technical evaluations; no fitted parameter or self-citation chain drives the central claim.

full rationale

SimStep is a system-design paper, not a fitted model or a first-principles derivation. The claimed chain from prompt P through Concept, Scenario, Learning Goal, and UI Interaction Graphs to code C is implemented by explicit LLM prompts (Appendix A) and evaluated with independent instruments: a user study with N=11 educators, PSSUQ, NASA-TLX, Cognitive Dimensions of Notations, and a separate 13-specification fidelity rating. No parameter is fit to an outcome and then renamed as a prediction; the forward transformations are generated by stated prompts, and the inverse correction process is implemented with widgets and LLM calls. Self-citations such as Spellburst [4] and Subramonyam et al. [52] appear only in related-work and motivation contexts; none provides a uniqueness theorem, forbids alternative representations, or carries the load of the central claim. Section 7.3 explicitly concedes that 'current LLMs are not inherently abstraction-aware and often fail to maintain consistency across stages,' which weighs against any suggestion that the framework's success is definitionally guaranteed. The closest concern is Section 6's fidelity metric, where the scorer is asked 'How closely does this abstraction adhere to the previous abstraction and user inputs?'; because each abstraction is generated from the previous one, this measures internal chain consistency rather than ground-truth intent, and the single-rater note in Table 2 acknowledges possible bias. That is a construct-validity limitation, not circularity by construction, and it does not constitute a circular step in the paper's derivation chain. The claim of 'greater authoring control' is also not supported by a baseline comparison, but that is an evaluation-design weakness rather than a circularity issue. Overall, the central contribution is a concrete system plus empirical evaluation, and no exhibitable reduction of a result to its own input is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 5 invented entities

No numeric parameters were fitted to data; the design choices are structural rather than numerical. The central claim rests on domain assumptions about teachers' mental workflows and about LLM reliability, plus a formal monotonicity claim that is asserted. The abstractions themselves are the contribution, but they are invented entities with no falsifiable handle outside the paper's own evaluation.

assumptions (4)
  • domain assumption Distributed cognition provides an appropriate theoretical basis for treating graph abstractions as cognitive checkpoints.
    Section 2 adopts Hutchins' framework to justify the CoA pipeline; if this framing is not apt for AI authoring, the claimed cognitive benefits lack theoretical grounding.
  • domain assumption The four abstractions (Concept, Scenario, Learning Goal, UI Interaction Graph) match teachers' natural task decomposition.
    Section 3 asserts teachers approach simulation design in four stages; this is not derived from data and is partially challenged by fidelity scores in Section 6.
  • domain assumption Each forward transformation reduces ambiguity and the set of compatible implementations (the Omega chain).
    Section 2.2 asserts Omega(P) superset Omega(B1) ... without proof; this monotonicity is definitional only if transformations are constrained to be refinements, which is not guaranteed for LLM outputs.
  • domain assumption LLMs can generate and propagate edits across abstractions and code with sufficient reliability.
    Section 4 relies on Claude for all transformations; Section 7.3 concedes current LLMs are not abstraction-aware and fail to maintain consistency across stages.
invented entities (5)
  • Concept Graph
    purpose: First checkpoint that externalizes domain concepts and relationships from the teacher's learning content.
    Introduced in Section 3.1 and Section 4.1.1; no external falsifiable prediction beyond the system's own evaluation.
  • Scenario Graph
    purpose: Instantiates the concept graph with a concrete scenario chosen by the teacher.
    Introduced in Section 3.1 and Section 4.1.2; fidelity scored 6.65/10 by a single rater.
  • Learning Goal Graph
    purpose: Prunes the scenario graph to nodes and links relevant to the selected learning objective.
    Introduced in Section 3.1 and Section 4.1.3; fidelity scored 7.08/10 by a single rater.
  • User Interaction Graph
    purpose: Represents simulation visuals, UI controls, and their relationships before code generation.
    Introduced in Section 3.1 and Section 4.1.4; it is the last forward abstraction before code.
  • Code Assumptions Abstraction
    purpose: Lists assumptions the generated code makes about each node so users can correct hidden choices.
    Introduced in Section 4.2 as part of the inverse correction set B; no external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SimStep: Chain-of-Abstractions for Incremental Specification and Debugging of AI-Generated Interactive Simulations." pith.science (2026). https://pith.science/paper/XNGHFLRJ

@misc{pith2026250709664,
  author       = {Pith},
  title        = {Pith review of: SimStep: Chain-of-Abstractions for Incremental Specification and Debugging of AI-Generated Interactive Simulations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNGHFLRJ}},
  note         = {Machine review of arXiv:2507.09664}
}
read the original abstract

Programming-by-prompting with generative AI offers a new paradigm for end-user programming, shifting the focus from syntactic fluency to semantic intent. This shift holds particular promise for non-programmers such as educators, who can describe instructional goals in natural language to generate interactive learning content. Yet in bypassing direct code authoring, many of programming's core affordances - such as traceability, stepwise refinement, and behavioral testing - are lost. We propose the Chain-of-Abstractions (CoA) framework as a way to recover these affordances while preserving the expressive flexibility of natural language. CoA decomposes the synthesis process into a sequence of cognitively meaningful, task-aligned representations that function as checkpoints for specification, inspection, and refinement. We instantiate this approach in SimStep, an authoring environment for teachers that scaffolds simulation creation through four intermediate abstractions: Concept Graph, Scenario Graph, Learning Goal Graph, and UI Interaction Graph. To address ambiguities and misalignments, SimStep includes an inverse correction process that surfaces in-filled model assumptions and enables targeted revision without requiring users to manipulate code. Evaluations with educators show that CoA enables greater authoring control and interpretability in programming-by-prompting workflows.

Figures

Figures reproduced from arXiv: 2507.09664 by the authors.

Figure 1
Figure 1. A teacher can use SimStep to generate interactive simulations that are accurate, integrate students’ context, and reflect [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The main screens of the SimStep application include a Learning Content Page (a and b), a Scenario Page (c and d), a [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The debugging process can be split into two main approaches: the manual approach and the automatic approach, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The system architecture of SimStep, consisting of a chain-of-abstractions pipeline for content generation and an [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The JSON LLM response representing the "Edit Assumptions" abstraction, along with its visual display in the chat. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Automated Testing and Error Resolution Workflow for Simulation Verification [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Teacher Participant Responses to Post-Study System [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: A selection of simulations and intermediate representations generated by teachers during our expert evaluation study. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Teacher Participant Responses to NASA’s Task Load [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Teacher Participant Responses to a Questionnaire [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: A selection of simulations generated through the CoA approach in comparison to simulations generated using a [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 50 canonical work pages

  1. [1]

    Ernest Adams. 2014. Fundamentals of game design . Pearson Education

    work page 2014

  2. [2]

    Garima Agrawal, Yuli Deng, Jongchan Park, Huan Liu, and Ying-Chih Chen. 2022. Building knowledge graphs from unstructured texts: Applications and impact analyses in cybersecurity education. Information 13, 11 (2022), 526

    work page 2022

  3. [3]

    Abdulaziz Alaboudi and Thomas D Latoza. 2023. Hypothesizer: A Hypothesis- Based Debugger to Find and Test Debugging Hypotheses. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1–14

    work page 2023

  4. [4]

    Tyler Angert, Miroslav Suzara, Jenny Han, Christopher Pondoc, and Hariharan Subramonyam. 2023. Spellburst: A node-based interface for exploratory creative coding with natural language prompts. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1–22

    2023

  5. [5]

    Anthropic. 2023. Claude: Language Model by Anthropic. https://www.anthropic. com/

    work page 2023

  6. [6]

    Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 2019. 3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019), 5663–5672. https://api.semanticscholar.org/ CorpusID:203837042

    work page 2019

  7. [7]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  8. [8]

    Michael P Barnett and WM Ruhsam. 1968. A natural language programming system for text processing. IEEE transactions on engineering writing and speech 11, 2 (1968), 45–52

    work page 1968

Show all 71 references
  1. [9]

    Michael Bostock and Jeffrey Heer. 2009. Protovis: A graphical toolkit for visu- alization. IEEE transactions on visualization and computer graphics 15, 6 (2009), 1121–1128. Chain-of-Abstractions for Authoring AI Simulations Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

    2009

  2. [10]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134 (2024)

    2024 arXiv

  3. [11]

    Nick C Bradley, Thomas Fritz, and Reid Holmes. 2018. Context-aware conversa- tional developer assistants. In Proceedings of the 40th International Conference on Software Engineering. 993–1003

    2018

  4. [12]

    Corey Brady, Brian Broll, Gordon Stein, Devin Jean, Shuchi Grover, Veronica Cateté, Tiffany Barnes, and Ákos Lédeczi. 2022. Block-based abstractions and expansive services to make advanced computing concepts accessible to novices. Journal of Computer Languages 73 (2022), 101156

    2022

  5. [13]

    Salman Cheema and Joseph LaViola. 2012. PhysicsBook: a sketch-based interface for animating physics diagrams. In Proceedings of the 2012 ACM international conference on Intelligent User Interfaces . 51–60

    2012

  6. [14]

    Penghe Chen, Yu Lu, Vincent W Zheng, Xiyang Chen, and Boda Yang. 2018. Knowedu: A system to construct knowledge graph for education. Ieee Access 6 (2018), 31553–31563

    2018

  7. [15]

    Bhavya Chopra, Yasharth Bajpai, Param Biyani, Gustavo Soares, Arjun Radhakr- ishna, Chris Parnin, and Sumit Gulwani. 2024. Exploring Interaction Patterns for Debugging: Enhancing Conversational Capabilities of AI-assistants. arXiv preprint arXiv:2402.06229 (2024)

    2024 arXiv

  8. [16]

    Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Educa- tion V. 1(Toronto ON, Canada)(SIGCSE 2023). ...

    2023

  9. [17]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 (2023)

    2023 arXiv

  10. [18]

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al. 2024. What’s Wrong with Your Code Generated by Large Language Models? An Extensive Study. arXiv preprint arXiv:2407.06153 (2024)

    2024

  11. [19]

    David Durman. 2024. joint.js. https://www.jointjs.com/

    2024

  12. [20]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1469–1481

    2023

  13. [21]

    OpenJS Foundation. 2024. Node.js. https://nodejs.org/en/

    2024

  14. [22]

    Silin Gao, Jane Dwivedi-Yu, Ping Yu, Xiaoqing Ellen Tan, Ramakanth Pasunuru, Olga Golovneva, Koustuv Sinha, Asli Celikyilmaz, Antoine Bosselut, and Tianlu Wang. 2024. Efficient tool use with chain-of-abstraction reasoning.arXiv preprint arXiv:2401.17464 (2024)

    2024 arXiv

  15. [23]

    Google. 2024. Firebase. https://firebase.google.com/

    2024

  16. [24]

    Thomas RG Green. 1989. Cognitive dimensions of notations.People and computers V (1989), 443–460

    1989

  17. [25]

    SG Hart. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload/Elsevier (1988)

    1988

  18. [26]

    Jaylin Herskovitz, Andi Xu, Rahaf Alharbi, and Anhong Guo. 2024. ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Program- ming. ArXiv abs/2408.10499 (2024). https://api.semanticscholar.org/CorpusID: 271909496

    2024 arXiv

  19. [27]

    TJ Holowaychuk. 2024. Express.js. https://expressjs.com/

    2024

  20. [28]

    Zichao Hu, Francesca Lucchetti, Claire Schlesinger, Yash Saxena, Anders Free- man, Sadanand Modak, Arjun Guha, and Joydeep Biswas. 2024. Deploying and evaluating llms to program service mobile robots. IEEE Robotics and Automation Letters 9, 3 (2024), 2853–2860

    2024

  21. [29]

    Edwin Hutchins. 1995. Cognition in the Wild . MIT press

    1995

  22. [30]

    Molina, Kristen Olson, Claire Kayacik, Aaron Dons- bach, Carrie J

    Ellen Jiang, Edwin Toh, A. Molina, Kristen Olson, Claire Kayacik, Aaron Dons- bach, Carrie J. Cai, and Michael Terry. 2022. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. Proceed- ings of the 2022 CHI Conference on Human ...

    2022

  23. [31]

    Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the syntax and strategies of natural language programming with generative language models. In Proceedings of the 2022 CHI Conference on H...

    2022

  24. [32]

    Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice

  25. [33]

    Ko, Robin Abraham, Laura Beckwith, Alan F

    Amy J. Ko, Robin Abraham, Laura Beckwith, Alan F. Blackwell, Margaret M. Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad A. Myers, M. Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The state of the art in end-user software enginee...

    2011

  26. [34]

    Amy J Ko and Brad A Myers. 2004. Designing the whyline: a debugging inter- face for asking questions about program behavior. In Proceedings of the SIGCHI conference on Human factors in computing systems . 151–158

    2004

  27. [35]

    Benjamin Lee, Arvind Satyanarayan, Maxime Cordeil, Arnaud Prouzeau, Bern- hard Jenny, and Tim Dwyer. 2023. Deimos: A grammar of dynamic embodied immersive visualisation morphs and transitions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–18

    2023

  28. [36]

    Teemu Lehtinen, Charles Koutcheme, and Arto Hellas. 2024. Let’s Ask AI About Their Programs: Exploring ChatGPT’s Answers To Program Comprehension Questions. In Proceedings of the 46th International Conference on Software Engi- neering: Software Engineering Education and Traini...

    2024

  29. [37]

    James R Lewis. 1992. Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ. In Proceedings of the human factors society annual meeting, Vol. 36. Sage Publications Sage CA: Los Angeles, CA, 1259–1260

    1992

  30. [38]

    What It Wants Me To Say

    Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin G. Zorn, J. Williams, Neil Toronto, and Andrew D. Gordon. 2023. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code- Generating Large Language Models. Proceedings of the 202...

    2023

  31. [39]

    Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Mar- ianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In The 13th International Joint Conference on Natural Language Pro- cessing and the 3rd Conference of the Asia-Pac...

    2023

  32. [40]

    John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn East- mond. 2010. The scratch programming language and environment. ACM Trans- actions on Computing Education (TOCE) 10, 4 (2010), 1–15

    2010

  33. [41]

    Meta. 2024. React. https://react.dev/

    2024

  34. [42]

    MUI. 2024. Material UI. https://mui.com/material-ui/

    2024

  35. [43]

    Brad A Myers, Amy J Ko, and Margaret M Burnett. 2006. Invited research overview: end-user programming. In CHI’06 extended abstracts on Human factors in computing systems. 75–80

    2006

  36. [44]

    Shep- herd

    Nico Ritschel, Felipe Fronchetti, Reid Holmes, Ronald Garcia, and David C. Shep- herd. 2022. Can guided decomposition help end-users write larger block-based programs? a mobile robot experiment. Proceedings of the ACM on Program- ming Languages 6 (2022), 233 – 258. https://api...

    2022

  37. [45]

    Karl Toby Rosenberg, Rubaiat Habib Kazi, Li-Yi Wei, Haijun Xia, and Ken Perlin

  38. [46]

    Steve Ruiz. 2021. tldraw: A tiny little drawing app. https://tldraw.com/

    2021

  39. [47]

    Nazmus Saquib, Rubaiat Habib Kazi, Li-yi Wei, Gloria Mark, and Deb Roy. 2021. Constructing embodied algebra by sketching. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–16

    2021

  40. [48]

    Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srini- vasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? arXiv preprint arXiv:2208.06213 (2022)

    2022 arXiv

  41. [49]

    Ahmed Seffah, Jan Gulliksen, and Michel C Desmarais. 2005. Human-centered software engineering-integrating usability in the software development lifecycle . Vol. 8. Springer Science & Business Media

    2005

  42. [50]

    David Canfield Smith, Allen Cypher, and Jim Spohrer. 1994. KidSim: Programming agents without a programming language. Commun. ACM 37, 7 (1994), 54–67

    1994

  43. [51]

    Da Song, Zijie Zhou, Zhijie Wang, Yuheng Huang, Shengmai Chen, Bonan Kou, Lei Ma, and Tianyi Zhang. 2023. An empirical study of code generation errors made by large language models. In 7th Annual Symposium on Machine Program- ming

    2023

  44. [52]

    Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–19

    2024

  45. [53]

    Hariharan Subramonyam, Colleen Seifert, Priti Shah, and Eytan Adar. 2020. Texsketch: Active diagramming through pen-and-ink annotations. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1–13

    2020

  46. [54]

    Sangho Suh, Jian Zhao, and Edith Law. 2022. Codetoon: Story ideation, auto comic generation, and structure mapping for code-driven storytelling. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology . 1–16

    2022

  47. [55]

    Ryo Suzuki, Rubaiat Habib Kazi, Li-Yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. Realitysketch: Embedding responsive graphics and visualizations in AR through dynamic sketching. InProceedings of the 33rd Annual ACM Symposium on User Interface Software and Te...

    2020

  48. [56]

    Knut Sveidqvist and Mermaid.js contributors. 2014. Mermaid: Generation of diagrams and flowcharts from text in a similar manner as markdown. https: //mermaid.js.org/

    2014

  49. [57]

    Chee Wei Tan, Shangxin Guo, Man Fai Wong, and Ching Nam Hang. 2023. Copilot for Xcode: exploring AI-assisted programming by prompting cloud-based large language models. arXiv preprint arXiv:2307.14349 (2023)

    2023 arXiv

  50. [58]

    Mei Tan and Hari Subramonyam. 2024. More than model documentation: uncov- ering teachers’ bespoke information needs for informed classroom integration of Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Kaputa et al. ChatGPT. In Proceedings of the CHI Conference on Huma...

    2024

  51. [59]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, et al. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621 (2024)

    2024 arXiv

  52. [60]

    Ektor Vrettakis, Christos Lougiakis, Akrivi Katifori, Vassilis Kourtis, Stamatis Christoforidis, Manos Karvounis, and Yannis Ioanidis. 2020. The story maker-an authoring tool for multimedia-rich interactive narratives. In Interactive Story- telling: 13th International Conferen...

    2020

  53. [61]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  54. [62]

    Sahraoui, and Bang Liu

    Martin Weyssow, Houari A. Sahraoui, and Bang Liu. 2022. Better Modeling the Programming World with Code Concept Graphs-augmented Multi-modal Learning. 2022 IEEE/ACM 44th International Conference on Software Engi- neering: New Ideas and Emerging Results (ICSE-NIER) (2022), 21–2...

    2022

  55. [63]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023)

    2023 arXiv

  56. [64]

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt

  57. [65]

    Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2015. Voyager: Exploratory analysis via faceted browsing of visualization recommendations.IEEE transactions on visualization and computer graphics 22, 1 (2015), 649–658

    2015

  58. [66]

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817 (2024)

    2024 arXiv

  59. [67]

    In Generative ai for effective software development

    Chatgpt prompt patterns for improving code quality, refactoring, require- ments elicitation, and software design. In Generative ai for effective software development. Springer, 71–108

  60. [68]

    p r e c o n n e c t

    Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, and Yuke Zhu. 2020. Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs. 2021 IEEE International Conference on Robotics and Automation (ICRA) (2020), 6541–6548. https://api.semanticscholar...

    2020

  61. [70]

    Xin Yin, Chao Ni, Tien N Nguyen, Shaohua Wang, and Xiaohu Yang. 2024. Rec- tifier: Code translation with corrector via llms. arXiv preprint arXiv:2407.07472 (2024)

    2024 arXiv

  62. [2014]

    In Proceedings of the 27th annual ACM symposium on User interface software and technology

    Kitty: sketching dynamic and interactive illustrations. In Proceedings of the 27th annual ACM symposium on User interface software and technology . 395–405

  63. [2024]

    arXiv preprint arXiv:2401.05631 (2024)

    DrawTalking: Building Interactive Worlds by Sketching and Speaking. arXiv preprint arXiv:2401.05631 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.