Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions

T0 review · 3 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompt2Constraints reduces student prompts to propositional logic and uses the reduction to spot when learners get stuck.

desk verdict Useful prompt-analysis method, but the paper's central claim about struggling students is contradicted by its own non-significant test. read the letter →

arxiv 2504.18691 v1 pith:FTDJ235Q submitted 2025-04-25 cs.HC cs.AIcs.SE

Ali Alfageeh , Sadegh AlMahdi Kazemi Zarkouei , Daye Nam , Daniel Prol , Matin Amoozadeh , Souti Chattopadhyay , James Prather , Paul Denny
show 4 more authors
Juho Leinonen Michael Hilton Sruti Srinivasa Ragavan Mohammad Amin Alipour
This is my paper · ORCID
classification cs.HCcs.AIcs.SE
keywords generativeartificialintelligencelargelanguagemodelspromptengineeringanalysiscomputingeducationpropositionallogicstudent-LLMinteractionconstraintextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether natural-language prompts that students write to an LLM can be converted into propositional-logic constraints without losing the semantic content that matters, and whether that conversion reveals who is struggling. It introduces Prompt2Constraints, which uses few-shot GPT-4 to translate each prompt into a conjunction of constraints such as "the response must be a Python function" or "the function must count occurrences of zero"; a manual audit of 234 sampled prompts found 225 translations correct (96.2%). Applied to 1,872 prompts from 203 students solving three introductory Python tasks, the representation lets the paper sort every prompt-to-prompt change into adding, modifying, rewording, or resubmitting constraints, and it reports that students who add constraints converge sooner while unsuccessful sessions show more substantial mid-session shifts in strategy. The payoff the paper argues for is scalable detection of struggling students in natural-language programming tasks, plus specific moments where an automated hint could intervene.

What carries the argument

The central object is the constraint set: a propositional-logic conjunction $\varphi(P) = C_1 \land C_2 \land \dots$ extracted from a prompt $P$, where each $C_i$ asserts a required property of the requested solution such as language, function name, parameters, behavior, or output format. Prompt2Constraints uses two hand-labeled consecutive prompts as few-shot examples to teach GPT-4 the extraction grammar; the LLM then outputs constraints, per-constraint descriptions, and a "semantic refinement" versus "core continuation" explanation for each new prompt. Because the formulas are in conjunctive normal form, set-like operations on the $C_i$ give a precise, machine-comparable account of how a student's intent changes from one turn to the next, and a symmetric difference between a student's constraint set and a solution prompt's constraint set yields a numeric "distance to solution."

What would settle it

Take a fresh random sample of 234 prompts, have two independent experts annotate the "true" constraint sets, and run Prompt2Constraints on the same prompts repeatedly with the same model and with a second model; if inter-expert agreement or test-retest agreement falls far below 96.2%, the accuracy claim and every evolution classification built on it do not hold.

Watch

Extended reading notes

Core claim

Prompt2Constraints treats each student prompt as a conjunction of atomic requirements $C_1 \land C_2 \land \dots$ and uses few-shot GPT-4 to derive those propositions directly from the prompt text. In a manual audit of 234 randomly sampled prompts, 225 translations matched an expert's reading (96.2%); the remaining 9 under-derived constraints, merging two distinct requirements into one proposition. On top of that representation, the paper classifies every transition between consecutive prompts as adding a constraint, modifying a constraint, rewording, or resubmission, and it reports that students who add constraints tend to finish sooner, whereas modifying, rewording, and resubmission correlate with longer sessions. Comparing successful and unsuccessful sessions, the paper claims the unsuccessful ones involve more substantive shifts in constraint sets, evidence of students abandoning one strategy and trying another, though the Mann-Whitney comparison of change sizes did not reach significance (p = 0.416). The paper's practical claim is that sudden large constraint changes mark moments when a hint could redirect a student who is close to a correct solution.

Load-bearing premise

Everything the paper concludes about student behavior follows from the assumption that GPT-4's few-shot extraction translates prompts into constraints faithfully and consistently; the paper validates that assumption once, by hand, on 234 prompts from a single dataset.

Editorial extensions

If this is right

  • Prompt corpora of thousands of interactions can be analyzed without per-prompt manual labeling: two few-shot examples sufficed to process 1,872 prompts.
  • Educators can locate intervention points automatically: a sudden, atypical jump in the constraint set between consecutive prompts identifies a moment for a hint.
  • The representation distinguishes semantic changes from pure wording changes: 35 of 41 minimized successful prompts kept the same constraint set, showing most students shortened prompts by rewording rather than by rethinking requirements.
  • Because adding constraints correlates negatively with session length, encouraging students to explicitly add missing requirements may be a more effective refinement strategy than repeatedly resubmitting or rephrasing.
  • Constraint distance to a known solution gives a quantitative progress measure usable by a hint generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not pursue: if constraint extraction is stable across models and over time, the constraint set becomes a model-independent semantic fingerprint of a prompt, letting researchers compare prompt quality across tasks and cohorts without LLM-version confounds.
  • A testable pedagogical experiment: assign one group of students an explicit "add one missing requirement at a time" rule and another group free prompting; the paper's negative correlation between adding constraints and session length predicts the constraint-adding group reaches a working prompt sooner.
  • The paper's binary success/failure contrast may mask ordering effects; an inference from its spider-graph examples is that failure often follows a specific trajectory of getting close, then abruptly discarding the close approach, so a process model of constraint trajectories could predict failure earlier than the aggregate comparison does.
  • The 9 under-extraction cases suggest a boundary: constraints that bundle two requirements are sometimes merged, so downstream analyses should be robust to constraint granularity, for example by measuring change at the level of semantic roles rather than individual propositions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Prompt2Constraints, a GPT-4 few-shot pipeline that converts student prompts into logical constraints (conjunctions of propositions such as 'a Python function is written' or 'the function counts the integer 0'), and applies it to 1,872 prompts from 203 CS1 students solving three programming tasks (counter, initials, repeat) from the Promptly dataset. The paper reports a manual accuracy check of 234 sampled prompts (225 correct, 96.2%), analyzes how constraints change across consecutive prompts (adding, modifying, rewording, resubmission), correlates those change categories with session length, and compares successful versus unsuccessful sessions on the size of constraint changes. The authors claim that successful and unsuccessful attempts use similar numbers of constraints overall, that failing students modify prompts more significantly, and that the method can detect struggling students and suggest intervention points.

Significance. If the accuracy and stability of Prompt2Constraints were established, the logic-based representation would be a useful scalable complement to qualitative prompt analysis: it makes semantic changes (additions, removals, rewordings) explicit and enables distance-to-solution measures. The framework is described in enough detail to be replicated, and the authors are transparent in Section 8 about dependence on a proprietary, evolving LLM. However, the paper's headline behavioral claim is directly contradicted by its own reported statistics, and the validation basis for all downstream analyses is thin. The contribution is interesting but requires substantial revision before the reported conclusions can be accepted.

major comments (3)
  1. [§6.3, Fig. 7, Abstract, §10] The central claim that failing students modify their prompts more significantly is contradicted by the paper's own results. Figure 7 reports an average constraint-change size of 0.83 for successful sequences and 0.71 for unsuccessful sequences, and the Mann-Whitney U-test gives p = 0.416. The Abstract states that 'when students fail, they often modify their prompts more significantly,' and the Conclusion states that 'struggling students tend to modify their prompts more, suggesting shifts in the problem-solving strategies.' Both the direction (unsuccessful < successful) and the significance level are inconsistent with this claim. Because RQ3 is the evidence for the paper's stated ability to 'detect struggling students,' the claim should be removed or explicitly reframed as a non-significant exploratory observation with the correct direction and effect size; no amount of translation accuracy in Section 6.1 can repair this statistical mismatch.
  2. [§5.1/§6.1, §8] The validation of Prompt2Constraints is too thin to support the 'robust and accurate' conclusion that underpins every downstream analysis. Section 6.1 reports one manual inspection of 234 sampled prompts with 225 correct, but it does not report the annotation protocol, the number of annotators, inter-rater reliability, or an error taxonomy beyond the nine 'fewer constraints' cases. Because GPT-4 is stochastic and proprietary (as Section 8 acknowledges), a single extraction run at one point in time does not establish stability. I ask for at least: (a) repeated extraction on a subsample to quantify run-to-run agreement; (b) agreement statistics between human annotators; and (c) a sensitivity analysis of the RQ2/RQ3 results with respect to the choice of the two few-shot examples. Without these, the constraint-level change categories cannot be treated as reliable measurements.
  3. [§6.2.1, Table 5] The sentence 'adding constraints helps students converge on a solution faster' is a causal interpretation of a correlation computed within sessions. Table 5 reports r = -0.338 between the percentage of adding-constraint changes and sequence length, but the direction of causality could be reversed (shorter sessions may consist mostly of early adding steps) or the association could be confounded by task difficulty. This wording should be softened to a descriptive association, and the four correlations in Table 5 should be reported with a multiple-comparison caveat.
minor comments (7)
  1. [§4.1 and §5.2] The logical form is described as a 'propositional logic formula' of the form P1 → (C1∧C2∧C3), yet Section 5.2 says all expressions are in conjunctive normal form; this is inconsistent and should be clarified.
  2. [Fig. 7] Figure 7 does not label its axes or define what 'size of changes between constraints' means; the caption's 'green line/red line/black line' also omits units. Please define the distance metric used.
  3. [Fig. 3] Figure 3's y-axis is not labeled; the text suggests average constraints per user, but it is unclear whether this is per task or across tasks.
  4. [§5.2] In the bullet defining 'Modifying Constraints,' 'previous prompt' should read 'previous constraints'; as written, the category overlaps with 'Adding Constraints' and needs a distinguishing criterion (e.g., removal of at least one existing constraint).
  5. [§6.3] Typo: 'in in the successful and unsuccessful prompt sequences' should read 'in the successful and unsuccessful prompt sequences.'
  6. [References] Reference [19] displays a corrupted author name ('P?durean'); this is likely an encoding issue and should be fixed.
  7. [§6.2.3, Table 6] The sentence 'Out of 41 pairs' should clarify how these pairs were identified and why the count is 41; the preceding paragraph does not define the sample selection procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the constraint-extraction pipeline is a measurement layer, and the RQ3 discrepancy is a reporting inconsistency, not a self-referential derivation.

full rationale

The paper's derivation chain is: prompt text → GPT-4 few-shot constraint extraction → constraint-change statistics → behavioral interpretation. The extraction is a representation/measurement step, not a parameter fitted to the outcome it later explains. Success/failure labels come from the external Promptly dataset's code-correctness outcomes, independent of the extracted constraints. The manual accuracy check (225/234 in §6.1) validates the measurement rather than reusing the conclusion. The Section 8 limitation about reliance on GPT-4 is a validity threat, not circularity. The only notable discrepancy—the abstract/conclusion claim that struggling students modify prompts more while §6.3 reports p = 0.416 and larger average changes for successful sequences (0.83 vs. 0.71)—is an internal statistical-reporting inconsistency and a correctness concern, not a circular reduction of outputs to inputs. Self-citations to the Promptly dataset and prior student-LLM studies supply data and context, but no load-bearing argument reduces to an unverified self-citation. Therefore, no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The method introduces a small number of analytical constructs (constraint categories, distance measure) but no physical or mathematical entities. The key unvalidated pillars are the faithfulness of GPT-4 extraction and the sufficiency of propositional constraints for representing prompt semantics.

free parameters (2)
  • Few-shot example selection = Two consecutive prompts from one user interaction
    The paper states 'only two consecutive prompts from one user interaction sufficed to generate reliable constraints for the entire dataset' (Section 4.1). The choice of which two prompts and which user is not shown to be representative or systematically varied.
  • Manual inspection sample size = 234 prompts
    Chosen to obtain a 95% confidence interval with 6% margin of error (Section 5.1). The 6% margin is a design choice that sets the tolerance for accuracy claims.
assumptions (3)
  • domain assumption GPT-4 few-shot constraint extraction is semantically faithful and stable across prompt wordings.
    The entire method depends on this. The paper provides a single manual inspection (96.2% correct on 234 prompts) and notes the risk in Section 8 that LLM behavior can change over time.
  • domain assumption A prompt's meaning can be adequately captured by a conjunction of propositional constraints.
    The method assumes that propositional logic constraints are sufficient to represent the intent of prompts, while the paper itself notes in Section 8 that 'nuanced or context-specific information' may not be captured.
  • domain assumption The Promptly dataset is representative of CS1 student prompting behavior.
    The findings are based on 1872 prompts from 203 students on three tasks. The paper acknowledges in Section 8 that generalization to other tasks and populations is a threat.
invented entities (1)
  • Constraint change categories
    purpose: Classify prompt evolution into Adding, Modifying, Rewording, and Resubmission to analyze student behavior.
    These are analytical categories introduced by the authors and derived from the constraint representations. They are not independently validated against, for example, expert judgments of student strategies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions." pith.science (2026). https://pith.science/paper/FTDJ235Q

@misc{pith2026250418691,
  author       = {Pith},
  title        = {Pith review of: From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTDJ235Q}},
  note         = {Machine review of arXiv:2504.18691}
}
read the original abstract

Background and Context. The increasing integration of large language models (LLMs) in computing education presents an emerging challenge in understanding how students use LLMs and craft prompts to solve computational tasks. Prior research has used both qualitative and quantitative methods to analyze prompting behavior, but these approaches lack scalability or fail to effectively capture the semantic evolution of prompts. Objective. In this paper, we investigate whether students prompts can be systematically analyzed using propositional logic constraints. We examine whether this approach can identify patterns in prompt evolution, detect struggling students, and provide insights into effective and ineffective strategies. Method. We introduce Prompt2Constraints, a novel method that translates students prompts into logical constraints. The constraints are able to represent the intent of the prompts in succinct and quantifiable ways. We used this approach to analyze a dataset of 1,872 prompts from 203 students solving introductory programming tasks. Findings. We find that while successful and unsuccessful attempts tend to use a similar number of constraints overall, when students fail, they often modify their prompts more significantly, shifting problem-solving strategies midway. We also identify points where specific interventions could be most helpful to students for refining their prompts. Implications. This work offers a new and scalable way to detect students who struggle in solving natural language programming tasks. This work could be extended to investigate more complex tasks and integrated into programming tools to provide real-time support.

Figures

Figures reproduced from arXiv: 2504.18691 by the authors.

Figure 1
Figure 1. Workflow in Prompt2Constraints Prompt 1 (P1) Write me a Python function that counts the number of ’0’s in the list. Formalization of P1: 𝐶1: A Python function is written. 𝐶2: The function counts the number of ’0’ (as a string) in the list. 𝐶3: The input to the function is a valid list. We can formalize P1 as: 𝑃1 → (𝐶1 ∧𝐶2 ∧𝐶3 ) Prompt 2 (P2) Write me a Python function that counts the number of 0 in the list. Formali… view at source ↗
Figure 2
Figure 2. Few-shot learning examples used in this paper for analysis of the [5]’s dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Distribution of Constraints per User 5 EVALUATION METHODOLOGY In this section, we describe the evaluation methodology. To answer the research questions, we evaluate a realization of Prompt2Constraints as described in Section 4.2 as a case study. 5.1 RQ1: Measure the accuracy of Prompt2Constraints To answer RQ1 and measure the accuracy of Prompt2Constraints, we first randomly sample a representative sample of prompts… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Changes in the constraints for each prompting task for students who finished all three tasks [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Changes in constraints potentially be used to find where an intervention to help students can be helpful. We note that we will use two examples to motivate the potential solution. In future work, a thorough evaluation of the generalizability and usefulness of the propo…
Figure 6
Figure 6. Figure 6: The average number of words in each subsequent submission compared to the average number of constraints and the number [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Size of changes in consecutive prompts A Case Study. Here we demonstrate the application of 1 to failed prompting. We use the failed prompts in Figure 5b along with the last prompt in Figure 5a which is a successful prompt [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Using constraints to guide failing prompting [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Srinivasa Ragavan, S., and Alipour, M. A. Trust in generative ai among students: An exploratory study. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (2024), pp. 67–73

    2024

  2. [2]

    Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J., Hilton, M., Srinivasa Ragavan, S., and Alipour, M. A. Student-ai interaction: A case study of CS1 students. In Proceedings of the 24th Koli Calling International Conference on Computing Education Research (New York, NY, USA, 2024), Koli Calling ’24, Association for Computing Machinery. Manuscrip...

    work page 2024

  3. [3]

    M., Nguyen, S., Zi, Y., Guha, A., Feldman, M

    Babe, H. M., Nguyen, S., Zi, Y., Guha, A., Feldman, M. Q., and Anderson, C. J. Studenteval: a benchmark of student-written prompts for large language models of code. arXiv preprint arXiv:2306.04556 (2023)

    arXiv 2023

  4. [4]

    Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language

    Denny, P., Kumar, V., and Giacaman, N. Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (2023), pp. 1136–1142

    work page 2023

  5. [5]

    A., and Reeves, B

    Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., and Reeves, B. N. Prompt problems: A new programming exercise for the generative ai era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (2024), pp. 296–302

    work page 2024

  6. [6]

    A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B

    Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., and Sarsa, S. Computing education in the era of generative ai. Communications of the ACM 67 , 2 (2024), 56–67

    work page 2024

  7. [7]

    H., Fowler, M., Prather, J., Becker, B

    Denny, P., Smith IV, D. H., Fowler, M., Prather, J., Becker, B. A., and Leinonen, J. Explaining code with a purpose: An integrated approach for developing code comprehension and prompting skills. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 . 2024, pp. 283–289

    work page 2024

  8. [8]

    Promoting early engagement with programming assignments using scheduled automated feedback

    Denny, P., Whalley, J., and Leinonen, J. Promoting early engagement with programming assignments using scheduled automated feedback. In Proceedings of the 23rd Australasian Computing Education Conference (2021), pp. 88–95

    work page 2021

Show all 23 references
  1. [9]

    Towards detecting prompt knowledge gaps for improved llm-guided issue resolution

    Ehsani, R., Pathak, S., and Chatterjee, P. Towards detecting prompt knowledge gaps for improved llm-guided issue resolution. arXiv preprint arXiv:2501.11709 (2025)

    2025 arXiv

  2. [10]

    Understanding the human-llm dynamic: A literature survey of llm use in programming tasks

    Etsenake, D., and Nagappan, M. Understanding the human-llm dynamic: A literature survey of llm use in programming tasks. arXiv preprint arXiv:2410.01026 (2024)

    2024 arXiv

  3. [11]

    A., Qin, H., Macedo, M., Tian, Y., Ding, S

    Hao, H., Hasan, K. A., Qin, H., Macedo, M., Tian, Y., Ding, S. H., and Hassan, A. E. An empirical study on developers’ shared conversations with chatgpt in github pull requests and issues. Empirical Software Engineering 29 , 6 (2024), 150

    2024

  4. [12]

    Exploring zero-shot and few-shot learning capabilities in llms for complex query handling, 2022

    Kalluri, K. Exploring zero-shot and few-shot learning capabilities in llms for complex query handling, 2022

    2022

  5. [13]

    H., Park, S., Lee, J., Jeong, K., and Kang, P

    Kim, J., Lee, S., Han, S. H., Park, S., Lee, J., Jeong, K., and Kang, P. Which is better? exploring prompting strategy for llm-based metrics. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems (2023), pp. 164–183

    2023

  6. [14]

    Organizational research: Determining appropriate sample size in survey research appropriate sample size in survey research

    Kotrlik, J., and Higgins, C. Organizational research: Determining appropriate sample size in survey research appropriate sample size in survey research. Information technology, learning, and performance journal 19 , 1 (2001), 43

    2001

  7. [15]

    C., and V anLehn, K

    Lane, H. C., and V anLehn, K. Coached program planning: Dialogue-based support for novice program design. In Proceedings of the 34th SIGCSE technical symposium on Computer science education (2003), pp. 148–152

    2003

  8. [16]

    ban it till we understand it

    Lau, S., and Guo, P. From" ban it till we understand it" to" resistance is futile": How university programming instructors plan to adapt as more students use ai code generation and explanation tools such as chatgpt and github copilot. In Proceedings of the 2023 ACM Conference ...

    2023

  9. [17]

    Comparing code explanations created by students and large language models

    Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., and Hellas, A. Comparing code explanations created by students and large language models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (2...

    2023

  10. [18]

    Using an llm to help with code understanding

    Nam, D., Macvean, A., Hellendoorn, V., V asilescu, B., and Myers, B. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (2024), pp. 1–13

    2024

  11. [19]

    Bugspotter: Automated generation of code debugging exercises

    P?durean, V.-A., Denny, P., and Singla, A. Bugspotter: Automated generation of code debugging exercises. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 (New York, NY, USA, 2025), SIGCSETS 2025, Association for Computing Machinery, p. 896–902

    2025

  12. [20]

    it’s weird that it knows what i want

    Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., and Santos, E. A. “it’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers. ACM transactions on computer-human in...

    2023

  13. [21]

    S., and Alipour, M

    Ragavan, S. S., and Alipour, M. A. Revisiting human information foraging: Adaptations for llm-based chatbots. arXiv preprint arXiv:2406.04452 (2024)

    2024 arXiv

  14. [22]

    H., Denny, P., and Fowler, M

    Smith IV, D. H., Denny, P., and Fowler, M. Prompting for comprehension: Exploring the intersection of explain in plain english questions and prompt writing. In Proceedings of the Eleventh ACM Conference on Learning@ Scale (2024), pp. 39–50

    2024

  15. [23]

    Y., Hartmann, B., and Y ang, Q

    Zamfirescu-Pereira, J., Wong, R. Y., Hartmann, B., and Y ang, Q. Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2023), pp. 1–21. Manuscript submitted to ACM

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.