Pith. sign in

REVIEW 4 major objections 7 minor 65 references

Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Equal access to an in-notebook LLM assistant does not erase the expertise gap: in a graduate data-science class, technical skill was the only significant predictor of grades, and logs show experts prompt strategically while novices lean…

desk verdict Valuable dataset and annotation method, but the abstract overstates a fragile regression; deserves peer review with robustness checks. read the letter →

arxiv 2509.21890 v2 pith:C3BTA4XE submitted 2025-09-26 cs.HC

Qianou Ma , Kenneth Koedinger , Tongshuang Wu This is my paper · ORCID
classification cs.HC
keywords largelanguagemodelsprogrammaticdataanalysisAIliteracyexpertisegaphuman-AIinteractioncomputationalnotebooksbehaviorallogscienceeducation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that giving students equal access to an LLM assistant inside a computational notebook does not level the playing field: self-reported technical expertise still significantly predicts homework grades, while AI familiarity and communication skill do not. Using logs from 36 graduate students completing Python data-analysis assignments in Google Colab with the built-in Gemini assistant, the authors show that experts use the LLM more strategically — clearer prompts, more context, more proactive exploration — while novices rely on it reactively to debug or explain errors. The authors introduce a behavior-grounded codebook that segments thousands of logged events into episodes and steps, annotated by an LLM, to identify where and why effective AI use breaks down. They argue that lightweight demonstrations improve surface skills like prompt clarity, but deeper evaluation, metacognitive, and dispositional skills require scaffolded instruction. If the claim is right, AI-literacy training must go beyond prompt tips and teach when to delegate, how to verify, and how to persist.

What carries the argument

The paper's central analytical device is a behavior-grounded codebook for AI use, built from thematic analysis of logs and screen recordings, that segments raw log events into intent-driven episodes and four steps modeled on Norman's Stages of Action: forming intent, expressing input, understanding output, and assessing output. Each step is annotated for challenges, step success or failure, which AI-use behaviors appeared (improve, code, explain, evaluate) and their quality (clear instruction, provided context, decomposed task), plus missed opportunities where AI could have helped. The annotation is scaled up by an LLM that applied the codebook automatically, because manual annotation of a single session took over two hours. This machinery is what lets the authors move from outcomes (grades) to process (how students actually interact with the assistant), and it is what generates the claims about which skills improve after demonstration and which remain bottlenecks.

What would settle it

A concrete test: have two independent human coders, blind to student expertise, hand-annotate all 1,483 steps of the homework logs using the paper's codebook, then re-examine whether experts still show higher prompt-quality ratios, lower challenge-driven AI use, and more missed evaluation opportunities. If those expert-novice differences shrink or disappear at human-level annotation accuracy rather than the reported 75%, the behavioral account is an artifact of the automated annotator rather than a genuine expertise effect.

Watch

Extended reading notes

Core claim

The central claim is that in realistic, LLM-supported programmatic data analysis, technical expertise remains a significant predictor of success even when every student has identical access to an AI assistant. In a linear mixed-effects model with participant and assignment random intercepts, self-reported technical expertise significantly predicted normalized homework grades ($\beta = 6.09$, $p = .041$), while LLM expertise ($p = .266$) and communication skills ($p = .381$) did not. Log analysis shows that experts and novices use AI at similar overall frequencies, but experts write clearer prompts, provide more context, decompose tasks, and ask for explanations mainly when they hit genuine obstacles; novices turn to AI reactively and struggle at the stages of forming intent, expressing input, interpreting output, and assessing results. The paper further claims that a 90-minute live-coding demonstration plus extended time-on-task improved prompt-quality behaviors by roughly 30% in the appropriate-use ratio, yet left evaluation behaviors (critiquing outputs, reading long explanations) below half of their opportunities, indicating those skills require guided practice and feedback rather than demonstration alone.

Load-bearing premise

The behavioral findings rest on the assumption that the automated annotation of student logs, which matched a single human coder's review of 10% of steps in about 75% of cases (71% for AI-use behaviors), correctly identifies what students actually did with the assistant.

Editorial extensions

If this is right

  • Instructors should treat equal tool access as necessary but not sufficient: in this classroom, giving every student the same notebook-embedded assistant left the performance gap intact, so effective AI-literacy training must target the behaviors that actually separate users.
  • The episode-step codebook gives teachers and tool designers a diagnostic vocabulary: a failure can be located in forming intent, expressing input, understanding output, or assessing results, and each failure type points to a different AI-use skill to teach.
  • Lightweight instruction has a measurable but bounded payoff: after one live-coding demonstration, prompt-clarity and context-providing behaviors increased in the logs, but evaluation behaviors stayed below half of their available opportunities.
  • AI-use competence is not a single skill: the paper's four knowledge dimensions imply that curricula should combine conceptual facts about what a given assistant can see and do, procedural practice in phrasing requests, metacognitive training in when to delegate, and support for persisting through errors.
  • Assistants embedded in notebooks should make their data access legible: recurring hallucination loops stem from students assuming the model can see the dataframe, so surfacing exactly what context the model has should reduce an entire failure class.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: randomly assign a 'verify your output' checklist scaffold to half of the novices and measure whether the evaluation-skill bottleneck closes faster than it did from demonstration alone.
  • The paper's own observation that the course tasks were not 'LLM-hard' suggests a boundary condition: on harder, open-ended tasks where current models struggle, prior technical expertise may matter even more, so the size of the expertise gap is probably not a constant.
  • A methodological corollary: with human annotation of a single student log taking over two hours, the LLM-annotation route is what makes log-scale behavioral studies feasible, but its reported 75% accuracy means future replications should report uncertainty bounds on behavior frequencies.
  • A reporting caveat the paper itself gives: the appropriate-use ratios for rare behaviors (improve-prompt, decompose-task, edit-partial-code) rest on one or two log instances per student, so those post-demo gains should be read as suggestive rather than measured.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. Ma et al. present a mixed-methods classroom study of 36 graduate students performing Python-based data analysis in Google Colab with the embedded Gemini assistant. They ask whether three self-reported expertise dimensions—technical, LLM, and communication—predict homework grades (RQ1); whether expertise shapes AI-use behaviors in interaction logs (RQ2); and which AI-use skills improve after lightweight demonstration and extended time (RQ3). The quantitative centerpiece is a linear mixed-effects model (Table 2) in which technical expertise significantly predicts normalized homework grades (beta=6.09, SE=2.86, p=.041) while LLM and communication expertise do not. Using a codebook of AI-use behaviors and an LLM-based annotation pipeline that segments 7,315 log events into episodes and steps, the authors report that experts write clearer prompts, use AI more strategically, and succeed more often in AI-assisted coding steps, while novices use AI reactively when challenged. A missed-opportunity analysis and surveys ground a four-dimensional taxonomy of AI-use competencies (conceptual, procedural, metacognitive, dispositional), and pre/post-demonstration comparisons suggest prompt-quality behaviors improve while evaluative behaviors lag. The paper also reports a counterbalanced HW0 comparison in which the expertise gap is significant without AI (p=.027) but not with AI (p>.05) under time pressure.

Significance. The study's raw material is a genuine asset: grades with substantial inter-rater reliability (kappa=0.71, §3.2), interaction logs, surveys, think-aloud recordings, a counterbalanced no-AI baseline in HW0, an open-source analysis pipeline, and complete annotation prompts and survey instruments in the appendices. If the headline result survives robustness checks, the paper would provide valuable naturalistic evidence on AI-assisted data science learning and a reusable behavior-grounded codebook for studying AI-use skills. The four-dimensional AI-literacy taxonomy and the 'less is more' scaffolding recommendation for evaluative feedback are actionable. However, the current evidence does not yet match the force of the claims: the RQ1 result is a single marginal p-value from a small, collinear, self-report-based model with post hoc composites, and the only between-condition comparison in the paper (HW0) points to a boundary condition—the gap disappeared under time pressure with AI—that is absent from the abstract. The RQ2/RQ3 findings depend on LLM annotation of moderate accuracy (70.9% on the key label) validated on 10% of steps by one author. The contribution is real but provisional.

major comments (4)
  1. [§5, Table 2; §4.1; §8.2] The headline RQ1 result in Table 2 is not yet robust enough to carry the paper's central claim. The technical-expertise effect is marginal (beta=6.09, SE=2.86, p=.041) in a model with 36 participants; the predictors are self-reported composites constructed post hoc after inspecting item correlations (§4.1), and their constituent dimensions are themselves correlated (rho=.52 and .72, §4.1). Section 8.2 concedes that self-perceived expertise is subject to known miscalibration (Kruger and Dunning, ref. [24]). No multicollinearity diagnostics, leave-one-out analysis, or alternative operationalizations of the predictors are reported, and the 'not communication skills' part of the claim is weakened by the restricted variance of communication scores (median 4/5, acknowledged in §5). I request VIF or inter-composite correlations and a leave-one-participant-out check on the technical coefficient.
  2. [§5 vs. §8.2 (HW0 baseline)] As specified, the Table 2 model cannot distinguish 'LLM access fails to close the gap' from 'stable individual differences predict grades with or without LLMs,' because it contains no no-AI baseline. The authors do possess such a baseline: HW0 counterbalanced tasks with and without AI, and §8.2 reports that the expertise gap was significant without AI (Mann–Whitney p=.027) but not with AI (p>.05) under time pressure. This is a direct boundary condition on the abstract's unqualified claim that technical experience 'remains a significant predictor of success,' and it should appear in the RQ1 analysis or the conclusion. I recommend a condition-by-expertise interaction analysis on HW0, and/or including HW0 no-AI task grades as a covariate in the Table 2 model.
  3. [§6.1.2; Figures 4–9] The RQ2 and RQ3 behavioral findings (Figures 4–9, the expert–novice comparisons, and the missed-opportunity ratios) rest on an LLM-based annotation whose validation is thin. Section 6.1.2 reports overall accuracy of 75%, and specifically 70.9% for the AI-use behavior labels, checked by a single author on 10% of steps; no per-code accuracy, confusion matrix, human inter-rater reliability, or label base rates are reported. For rare codes such as ai_breakdown_intent (0.65%) and decompose_task_in_prompt (0.31%), 70.9% overall accuracy could correspond to poor per-class utility, and because both the assistant under study and the annotator are LLMs, shared failure modes are a credible risk. I ask for per-code precision and recall, oversampling of rare codes in the validation sample, and a second human annotator on a shared subset with a reported kappa.
  4. [§6.2, §7.1 (Figures 6–8)] Several load-bearing comparisons in RQ2/RQ3 are quantitative claims presented without inferential statistics: expert vs. novice success on AI-assisted coding steps (90% vs. 79%, Figure 6A), the challenge-driven vs. proactive AI-use contrast (Figure 7), and the pre/post-demonstration changes in appropriate-AI-use ratio (Figure 8). The data are clustered (steps within episodes within students), so nominal percentages overstate precision. Section 7.1's caveat that the Figure 8 changes were 'not fully mastered or statistically compared' is to the authors' credit, but the surrounding narrative in §7 and the abstract nevertheless treats the skill-acquisition findings as results. I recommend mixed-effects logistic regressions on step success and on the used/missed labels, or a consistently exploratory framing in the abstract and conclusion.
minor comments (7)
  1. [§4.2, §6.1.1, abstract] Section 4.2 reports 16,315 collected events, Section 6.1.1 analyzes 7,315 events after exclusions, and the abstract says 'over 7,000 raw log events'; the relationship between these numbers should be stated at first mention.
  2. [§6.1.1] Section 6.1.1 excludes logs shorter than the median as 'disengaged'; this rule conflates disengagement with fast, efficient task completion and should be justified or tested as a sensitivity analysis.
  3. [Table 3] In Table 3, the decompose_task_in_prompt row mixes a positive definition with a negative example ('[copied output] without intent would be too vague'); clarify whether the example column intentionally contains counterexamples.
  4. [Figures 6–8] Figures 6–8 present expert–novice and pre–post differences as point estimates without error bars or confidence intervals; adding them would materially aid interpretation.
  5. [Table 2; §4.1] In Table 2, the LLM predictor row is formatted as '−3.052.69'; it should read '−3.05 (2.69)', and Section 4.1 contains a typo ('tookit' for 'toolkit').
  6. [§4.1] The expert/novice median split (§4.1) of skewed self-report scores discards information; reporting the RQ2 comparisons against the continuous technical score would strengthen the behavioral claims.
  7. [§6.1.1] Section 6.1.1 excludes HW3–4 logs for comparability with HW0, but HW0 used a different dataset and 15-minute time limits; one sentence clarifying what is held constant across the compared assignments would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical regressions and log-based behavioral measurements, not derivations that reduce to their own inputs.

full rationale

The paper's RQ1 result is a standard linear mixed-effects regression in which self-reported expertise composites collected at the start of the course are used to predict homework grades from HW1-HW4 (Section 5, Table 2). The outcome (grades) is external to the predictors, and the technical-expertise composite was formed by grouping survey items before the regression, so the significant coefficient (beta = 6.09, p = .041) is a statistical fit to data rather than a tautology. RQ2 and RQ3 rest on an independently developed codebook (Section 6.1.2), built through thematic analysis of logs and recordings and scaled with LLM annotation; while the annotation accuracy of 70.9% for AI-use behaviors is a reliability limitation, the claimed expert-novice behavioral differences are comparisons between groups defined by pre-survey expertise and therefore do not reduce to the annotation schema by construction. The paper's self-citations (e.g., [30] and [31]) are used as related-work support for the value of requirement-driven prompting and debugging instruction, not as load-bearing premises that forbid alternatives or supply an unverified uniqueness result. Potential weaknesses noted by the paper itself - small sample (N=36), correlated expertise dimensions, the HW0 result that the expert-novice gap disappeared under time pressure with LLMs, and the absence of a no-AI baseline in HW1-4 - are threats to validity or generalizability, not evidence that any claim is equivalent to its inputs by definition. No circular step could be quoted and reduced to an equation or fitted parameter renamed as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on the validity of self-report measures, the quality of LLM annotation, the applicability of Norman's stages, and the filtering criteria. There are no free parameters or invented entities. These axioms are domain assumptions explicitly or implicitly invoked by the study design.

assumptions (4)
  • domain assumption Self-reported expertise scores accurately reflect students' true expertise.
    The regression in Section 5 uses self-report survey items as predictors. The paper itself cites known miscalibration issues (Kruger and Dunning, 1999), so this assumption is load-bearing for RQ1.
  • domain assumption LLM annotation with 75% accuracy, validated on 10% of steps by one author, is sufficient for behavioral conclusions.
    The behavioral analysis in Sections 6 and 7 relies entirely on LLM-generated annotations. The paper reports accuracy of 75% overall and 70.9% for AI-use behaviors, which is modest.
  • domain assumption Norman's stages of action apply to AI-assisted computational workflows.
    The episode-step segmentation in Section 6.1.2 is inspired by Norman's stages (intent, input, understand, assess). If this model does not fit the log data, the behavioral patterns could be artifacts of the segmentation.
  • domain assumption Students excluded from log analysis (external tool use, low log length, HW3-4) would not have changed the conclusions.
    Section 6.1.1 filters from 36 to 28 students and drops HW3-4. The exclusions could bias the behavioral findings toward compliant, more engaged students.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science." pith.science (2026). https://pith.science/paper/C3BTA4XE

@misc{pith2026250921890,
  author       = {Pith},
  title        = {Pith review of: Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3BTA4XE}},
  note         = {Machine review of arXiv:2509.21890}
}
read the original abstract

LLMs promise to democratize technical work in complex domains like programmatic data analysis, but not everyone benefits equally. We study how students with varied experiences use LLMs to complete Python-based data analysis in computational notebooks in a graduate course. Drawing on homework logs, recordings, and surveys from 36 students, we ask: Which experience matters most, and how does it shape AI use? Our mixed-methods analysis shows that technical experience -- not AI familiarity or communication skills -- remains a significant predictor of success. Students also vary widely in how they leverage LLMs, struggling at stages of forming intent, expressing inputs, interpreting outputs, and assessing results. We identify success and failure behaviors, such as providing context or decomposing prompts, that distinguish effective use. These findings inform AI literacy interventions, highlighting that lightweight demonstrations improve surface fluency but are insufficient; deeper training and scaffolds are needed to cultivate resilient AI use skills.

Figures

Figures reproduced from arXiv: 2509.21890 by the authors.

Figure 1
Figure 1. Google Colab notebook environment with embedded Gemini assistant. Students could interact with Gemini in two [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of self-rated technical expertise, LLM expertise, and communication expertise scores. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. An example log event format. browser localStorage, which students exported as JSON files upon submission (or auto-save when the storage is full). In total, we collected 16,315 logged events (see example in Sec￾tion 4.2) and 44 hours of screen recordings across five assignments. Logs captured a rich sequence of AI interactions, code edits, and executions, supporting both quantitative distributional analyses and quali… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (A) Average frequency of AI use codes per student (refer to Table [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 8
Figure 8. Figure 8: Appropriate AI use ratio pre- to post-instructions. Ratio = [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: The distribution of the last successful step in an [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 29 canonical work pages

  1. [24]

    Justin Kruger and David Dunning. 1999. Unskilled and unaware of it: How diffi- culties in recognizing one’s own incompetence lead to inflated self-assessments. J. Pers. Soc. Psychol.77, 6 (1999), 1121–1134. https://doi.org/10.1037/0022- 3514.77.6.1121

    doi:10.1037/0022- 1999

  2. [1]

    Ravinithesh Annapureddy, Alessandro Fornaroli, and Daniel Gatica-Perez. 2025. Generative AI literacy: Twelve defining competencies.Digital Government: Re- search and Practice6, 1 (2025), 1–21

    work page 2025

  3. [2]

    Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models Usinglme4.J. Stat. Softw.67, 1 (Oct. 2015), 1–48. https://doi.org/10.18637/jss.v067.i01

    doi:10.18637/jss.v067.i01 2015

  4. [3]

    Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. 2025. Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv [cs.AI](July 2025). arXiv:2507.09089 [cs.AI] http://arxiv.org/abs/2507.09089

    arXiv 2025

  5. [4]

    Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology3, 2 (2006), 77–101

    2006

  6. [5]

    John Chen, Xi Lu, Yuzhou Du, Michael Rejtig, Ruth Bagley, Mike Horn, and Uri Wilensky. 2024. Learning agent-based modeling with LLM companions: Experiences of novices and experts using ChatGPT & NetLogo chat. InProceedings of the CHI Conference on Human Factors in Computing Systems, Vol. 13. ACM, New York, NY, USA, 1–18. https://doi.org/10.1145/3613904.3642377

    arXiv 2024

  7. [6]

    Jiaqi Chen, Yanzhe Zhang, Yutong Zhang, Yijia Shao, and Diyi Yang. 2025. Generative Interfaces for Language Models.arXiv [cs.CL](Aug. 2025). arXiv:2508.19227 [cs.CL] http://arxiv.org/abs/2508.19227

    arXiv 2025

  8. [7]

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. 2025. Need Help? Designing Proactive AI Assistants for Pro- gramming. InProceedings of the 2025 CHI Conference on Human Factors in Com- puting Systems. 1–18

    2025

Show all 65 references
  1. [8]

    Ruijia Cheng, Titus Barik, Alan Leung, Fred Hohman, and Jeffrey Nichols. 2024. BISCUIT: Scaffolding LLM-generated code with ephemeral UIs in computational notebooks.arXiv [cs.HC](April 2024). arXiv:2404.07387 [cs.HC] http://arxiv. org/abs/2404.07387

    2024 arXiv

  2. [9]

    Michelene T H Chi and Ruth Wylie. 2014. The ICAP Framework: Linking Cogni- tive Engagement to Active Learning Outcomes.Educ. Psychol.49, 4 (Oct. 2014), 219–243. https://doi.org/10.1080/00461520.2014.965823

    2014

  3. [10]

    Mark Claypool, David Brown, Phong Le, and Makoto Waseda. 2001. Inferring user interest.IEEE Internet Computing5, 6 (2001), 32–39

    2001

  4. [11]

    Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. 2019. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom.Proc. Natl. Acad. Sci. U. S. A.116, 39 (Sept. 2019), 19251–19257. https://do...

    2019 doi

  5. [12]

    Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development.International journal of qualitative methods5, 1 (2006), 80–92

    2006

  6. [13]

    Google. 2025. Google Colab. https://colab.research.google.com/. Accessed: 2025-9-8

    2025

  7. [14]

    Yue Guo, Joseph Chee Chang, Maria Antoniak, Erin Bransom, Trevor Cohen, Lucy Lu Wang, and Tal August. 2024. Personalized jargon identification for enhanced interdisciplinary communication. InProceedings of the conference. Asso- ciation for Computational Linguistics. North Amer...

    2024

  8. [15]

    Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. InAdvances in Psychol- ogy, Peter A Hancock and Najmedin Meshkati (Eds.). Advances in psychology, Teaching Effective AI Use in Computational Da...

    1988 doi

  9. [16]

    Yoshinori Hijikata. 2004. Implicit user profiling for on demand relevance feedback. InProceedings of the 9th international conference on Intelligent user interfaces (IUI ’04). Association for Computing Machinery, New York, NY, USA, 198–205. https://doi.org/10.1145/964442.964480

    2004

  10. [17]

    White, Georg Buscher, and Kuansan Wang

    Jeff Huang, Ryen W. White, Georg Buscher, and Kuansan Wang. 2012. Improving searcher models using mouse cursor activity. InThe 35th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR ’12, Portland, OR, USA, August 12-16, 2012, Willia...

    2012

  11. [18]

    Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara Jane Ericson, David Weintrop, and Tovi Grossman. 2023. How novices use LLM-based code generators to solve CS1 coding tasks in a self-paced learning environment. InProceedings of the 23rd Koli calling international confer...

    2023

  12. [19]

    Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Zachary Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving steering and verification in AI-assisted data analysis with interactive task decomposition. In Proceedings of the 37th Annual ACM Symposium...

    2024

  13. [20]

    Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs.arXiv [cs.HC](Jan. 2024). https://doi...

    2024 arXiv

  14. [21]

    Tae Soo Kim. 2025. Interacting with AI by manipulating intents. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–5. https://doi.org/10.1145/3706599.3707611

    2025

  15. [22]

    Michal Kosinski. 2024. Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences121, 45 (2024), e2405460121

    2024

  16. [23]

    David R Krathwohl. 2002. A revision of bloom’s taxonomy: An overview.Theory Pract.41, 4 (Nov. 2002), 212–218. https://doi.org/10.1207/s15430421tip4104_2

    2002 doi

  17. [25]

    Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human- ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1–19

    2022

  18. [26]

    Ivan Letteri and Pierpaolo Vittorini. 2025. Enhancing student feedback in data science education: Harnessing the power of AI-generated approaches.Int. J. Artif. Intell. Educ.(June 2025), 1–24. https://doi.org/10.1007/s40593-025-00492-8

    2025 doi

  19. [27]

    Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM international conference on software engineering. 1–13

    2024

  20. [28]

    Nelson Lojo, Rafael González, Rohan Philip, José Antonio Parejo, Amador Durán Toro, Armando Fox, and Pablo Fernández. 2025. Using large language mod- els to develop requirements Elicitation skills.arXiv [cs.SE](March 2025). arXiv:2503.07800 [cs.SE] http://arxiv.org/abs/2503.07800

    2025 arXiv

  21. [29]

    I’m not sure, but

    Christopher Lum, Guoxuan Xu, and Sam Lau. 2025. “I’m not sure, but. . . ”: Expert Practices that Enable Effective Code Comprehension in Data Science, Vol. 1. Association for Computing Machinery. https://doi.org/10.1145/3641554.3701933

    2025

  22. [30]

    Qianou Ma, Weirui Peng, Chenyang Yang, Hua Shen, Ken Koedinger, and Tong- shuang Wu. 2025. What should we engineer in prompts? Training humans in requirement-driven LLM use.ACM Trans. Comput. Hum. Interact.32, 4 (Aug. 2025), 1–27. https://doi.org/10.1145/3731756

    2025 doi

  23. [31]

    Qianou Ma, Hua Shen, Kenneth Koedinger, and Sherry Tongshuang Wu. 2024. How to teach programming in the AI era? Using LLMs as a teachable agent for debugging. InProceedings of International Conference on Artificial Intelligence in Education. Springer Nature Switzerland, Cham, ...

    2024

  24. [32]

    Mary L McHugh. 2012. Interrater reliability: the kappa statistic.Biochem. Med. (Zagreb)22, 3 (2012), 276–282. https://doi.org/10.11613/bm.2012.031

    2012 doi

  25. [33]

    2010.Mann-Whitney U Test

    Patrick E McKnight and Julius Najab. 2010.Mann-Whitney U Test. John Wiley & Sons, Inc., Hoboken, NJ, USA. https://doi.org/10.1002/9780470479216.corpsy0524

    2010 doi

  26. [34]

    Andrew M Mcnutt, Chenglong Wang, Robert A Deline, and Steven M Drucker

  27. [35]

    Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman. 2024. How beginning programmers and code llms (mis) read each other. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–26

    2024

  28. [36]

    Sydney Nguyen, Hannah Mclean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman. 2024. How Beginning Programmers and Code LLMs (Mis)read Each Other. InProceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24, Article 651). Associa...

    2024

  29. [37]

    Shramay Palta, Nirupama Chandrasekaran, Rachel Rudinger, and Scott Counts

  30. [38]

    Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, and Satish Chandra. 2025. How much does AI impact development speed? An enterprise-based randomized controlled trial. In2025 IEEE/ACM 47th International Conference ...

    2025

  31. [39]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The impact of ai on developer productivity: Evidence from github copilot.arXiv preprint arXiv:2302.06590(2023)

    2023 arXiv

  32. [40]

    Madison Pickering, Helena Williams, Alison Gan, Weijia He, Hyojae Park, Francisco Piedrahita Velez, Michael L Littman, and Blase Ur. 2025. How hu- mans communicate programming tasks in natural language and implications for end-user programming with LLMs. InProceedings of the 2...

    2025

  33. [41]

    Brett Puppart and Jaan Aru. 2025. Short-term AI literacy intervention does not reduce over-reliance on incorrect ChatGPT recommendations.arXiv preprint arXiv:2503.10556(2025)

    2025 arXiv

  34. [42]

    Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. 2018. Machine theory of mind. InInternational conference on machine learning. PMLR, 4218–4227

    2018

  35. [43]

    Fangchen Song, Ashish Agarwal, and Wen Wen. 2024. The impact of generative AI on collaborative open-source software development: Evidence from GitHub Copilot.arXiv preprint arXiv:2410.02091(2024)

    2024 arXiv

  36. [44]

    James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. 2024. Testing theory of mind in large language models and humans. Nature Human Behaviour(2024), 1–11

    2024

  37. [45]

    Anselm Strauss and Juliet Corbin. 1998. Basics of qualitative research techniques. (1998)

    1998

  38. [46]

    Edward Tenner. 2015. The design of everyday things by Donald Norman.Tech- nology and Culture56, 3 (2015), 785–787

    2015

  39. [47]

    Jiessie Tie, Bingsheng Yao, Tianshi Li, Syed Ishtiaque Ahmed, Dakuo Wang, and Shurui Zhou. 2024. LLMs are imperfect, then what? An empirical study on LLM failures in software engineering.arXiv [cs.SE](Nov. 2024). arXiv:2411.09916 [cs.SE] http://arxiv.org/abs/2411.09916

    2024 arXiv

  40. [48]

    Endel Tulving and Zena Pearlstone. 1966. Availability versus accessibility of information in memory for words.J. Verbal Learning Verbal Behav.5, 4 (Aug. 1966), 381–391. https://doi.org/10.1016/s0022-5371(66)80048-8

    1966 doi

  41. [49]

    Manuel Valle Torre, Thom van der Velden, Marcus Specht, and Catharine Oertel

  42. [50]

    Vijay Viswanathan, Kiril Gashteovski, Kiril Gashteovski, Carolin Lawrence, Tong- shuang Wu, and Graham Neubig. 2024. Large language models enable few-shot clustering.Transactions of the Association for Computational Linguistics12 (2024), 321–333

    2024

  43. [51]

    April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. 2022. Documentation matters: Human-centered AI system to assist data science code documentation in compu- tational notebooks.ACM Trans. Comput. Hum. Int...

    2022 doi

  44. [52]

    In AIED 2025

    JELAI: Integrating AI and learning analytics in jupyter notebooks. In AIED 2025. Lecture Notes in Computer Science (Lecture notes in computer science). Springer Nature Switzerland, Cham, 68–75. https://doi.org/10.1007/978-3-031- 98465-5_9

    2025 doi

  45. [53]

    J D Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang

  46. [54]

    Jinjin Zhao, Avidgor Gal, and Sanjay Krishnan. 2024. Data makes better data scientists.arXiv [cs.HC](May 2024). arXiv:2405.17690 [cs.HC] http://arxiv.org/ abs/2405.17690

    2024 arXiv

  47. [55]

    2005.Understanding by Design

    Grant P Wiggins and Jay McTighe. 2005.Understanding by Design. ASCD, USA. https://play.google.com/store/books/details?id=N2EfKlyUN4QC

    2005

  48. [56]

    What were the challenges for you during the tasks when you didnotuse GenAI?

    Yangtian Zi, Luisa Li, Arjun Guha, Carolyn Anderson, and Molly Q Feldman. 2025. “I Would Have Written My Code Differently’: Beginners Struggle to Understand LLM-Generated Code. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1...

    2025

  49. [57]

    InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23, Article 437)

    Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23, Article 437). Association for Computing Machinery, New York, NY, USA, 1–21. https://doi.org/10.1145/354454...

    2023

  50. [59]

    Yong Zheng. 2023. ChatGPT for teaching and learning: An experience from data science education. InThe 24th Annual Conference on Information Technology Education. ACM, New York, NY, USA, 66–72. https://doi.org/10.1145/3585059. 3611431

    2023 doi

  51. [61]

    •Ensure all log events are covered

    Segment logs into episodes: • Start a new episode when you observe a clear change in intent (see decision boundary above),ORif there is a time gap ≥ 15 minutes between consecutive events (usedatetime_timestamp). •Ensure all log events are covered. Episodes should be consecutiv...

  52. [62]

    episode_id

    Identify the episode’s overall intent: •Infer from code, prompts, and outputs (e.g., remove outliers in a column, create a bar chart, refine a visual by sorting by count). •Write 1–2 sentences describing what the user was trying to do and what happened, inferred from the log e...

    2025

  53. [63]

    For each step of the episode, trace key events and summarize into Actions & Returns with evidence: • Action: initiate an attempt (e.g., edit code, enter prompt, send Gemini prompt, UI button like execute after editing code, press_explain_error, etc.) •Return: immediate observa...

  54. [64]

    •If AI was used well, tag behaviors (e.g.,ai_explain_concepts,clear_instruction_in_prompt) and briefly explain how it helped

    For each step, annotate if a challenge was encountered, and whether AI was used strategically to overcome it: •Use the challenge/success behavior examples to markencountered_challenge. •If AI was used well, tag behaviors (e.g.,ai_explain_concepts,clear_instruction_in_prompt) a...

  55. [65]

    step_id": 1,

    Decide if the step was successful, and explain: 2025, Preprint, USA Qianou Ma, Kenneth Koedinger, and Tongshuang Wu •step_successcan be true/false (partial progress or step not reached counts as false). • Provide 2–3 sentences citing how the outcome is inferred from logs; incl...

    2025

  56. [2023]

    InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23, Article 434)

    On the Design of AI-powered Code Assistants for Notebooks. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23, Article 434). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3544548.3580940

    2023

  57. [2025]

    Speaking the right language: The impact of expertise alignment in user-ai interactions.arXiv preprint arXiv:2502.18685(2025)

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.