Pith. sign in

REVIEW 2 major objections 2 minor 7 references

Constructing Epistemic AI Literacy: Detecting Epistemic Aims and Processes in Student-AI Co-Programming

T0 review · 2 major / 2 minor · reviewed 2026-07-02 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most student-GenAI programming interactions rely on non-mastery aims and less reliable strategies like outsourcing.

desk verdict The paper defines observable dimensions for epistemic engagement in student-GenAI programming but the headline percentages rest on unverified coding from text data. read the letter →

arxiv 2607.00211 v1 pith:WTFMTU4Q submitted 2026-06-30 cs.AI cs.HC

Mengqian Wu This is my paper
classification cs.AIcs.HC
keywords epistemicAIliteracygenerativestudent-AIinteractionaimsAIRframeworkprogrammingeducationdialogueanalysisco-programming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Epistemic AI Literacy as a process-oriented view of how students build and evaluate knowledge while using generative AI for programming. It applies the AIR framework to label dialogue turns for aims such as mastery orientation and for processes such as outsourcing, verification seeking, and epistemic justification. Analysis of a large set of student-AI exchanges shows that nearly 79 percent of interactions use non-mastery aims paired with less reliable moves, while only about 11 percent reach high engagement through mastery aims combined with justification. A reader would care because the pattern suggests current everyday use of AI tools may not automatically produce thoughtful problem-solving habits in learners.

What carries the argument

Epistemic AI Literacy (EAIL), which detects mastery-oriented aims and reliable epistemic processes such as epistemic justification within human-AI dialogue data by applying the AIR framework.

What would settle it

A fresh round of independent annotation on the same dialogue dataset that produces markedly different shares of mastery-oriented aims or high-epistemic-engagement turns.

Watch

Extended reading notes

Core claim

Drawing on the AIR framework, the study identifies observable dimensions of epistemic aims (mastery-oriented aims) and epistemic processes (outsourcing, explanation seeking, verification seeking, prompt monitoring, and epistemic justification) in GenAI-supported co-programming dialogues. The results show that 78.8 percent of student-GenAI interactions rely on non-mastery-oriented aims and less reliable epistemic strategies, whereas only 11.1 percent of interactions display high epistemic engagement in which mastery-oriented aims are coupled with epistemic justification in a more reliable epistemic process.

Load-bearing premise

The observable dimensions extracted from text dialogues accurately capture the underlying epistemic aims and processes defined by the AIR framework.

Editorial extensions

If this is right

  • Most student-GenAI co-programming interactions operate with non-mastery aims and less reliable strategies such as outsourcing and verification seeking.
  • A small share of interactions reach high epistemic engagement when mastery aims pair with epistemic justification.
  • Observable dimensions of aims and processes can be identified and scaled across large dialogue datasets from human-AI co-programming.
  • Epistemic AI literacy can be treated as an emergent property of dynamic interactions rather than a static skill set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Educational designs that prompt students to state justifications may raise the share of high-engagement turns.
  • GenAI interfaces for classrooms could add cues that favor explanation seeking over direct outsourcing.
  • Differences in EAIL levels across interactions may correspond to different long-term gains in programming competence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces the Epistemic AI Literacy (EAIL) framework by adapting the AIR (aims, ideals, reliable processes) model to student-GenAI co-programming interactions. It identifies observable dimensions (mastery-oriented aims; outsourcing, explanation seeking, verification seeking, prompt monitoring, epistemic justification) in a large dialogue dataset and reports that 78.8% of interactions exhibit low EAIL (non-mastery aims plus less reliable strategies) while only 11.1% exhibit high epistemic engagement (mastery aims coupled with advanced strategies such as epistemic justification).

Significance. If the annotation procedure is shown to be valid and reliable, the work supplies a process-oriented lens on AI literacy that could guide both assessment and instructional design in programming education. The use of an established external framework (AIR) on interaction data is a strength, but the absence of dataset size, collection protocol, coding details, and reliability metrics prevents evaluation of whether the headline percentages are reproducible or generalizable.

major comments (2)
  1. [Abstract / Results] Abstract and results section: the claims of 78.8% low EAIL and 11.1% high engagement are presented without any information on total interactions analyzed, sampling method, annotation protocol, inter-rater reliability, or statistical tests. These omissions are load-bearing because the percentages constitute the central empirical claim.
  2. [Methods] Methods / coding procedure: the mapping from surface dialogue features (e.g., "outsourcing", "epistemic justification") to AIR constructs is asserted but not validated against text-only data; no evidence is supplied that the annotation recovers intended epistemic aims and processes without substantial context loss or misclassification.
minor comments (2)
  1. [Abstract] The term "large dialogue dataset" is used without a citation or size; a reference to the source corpus and its scale would improve reproducibility.
  2. [Framework section] Notation for the five process dimensions is introduced without a table or explicit operational definitions; a summary table would aid clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback emphasizing the need for methodological transparency. We agree that the submitted manuscript lacks essential details on the dataset and annotation procedure, which are required to substantiate the central empirical claims, and we will revise the paper to address these points.

read point-by-point responses

  1. Referee: [Abstract / Results] Abstract and results section: the claims of 78.8% low EAIL and 11.1% high engagement are presented without any information on total interactions analyzed, sampling method, annotation protocol, inter-rater reliability, or statistical tests. These omissions are load-bearing because the percentages constitute the central empirical claim.

    Authors: We agree that these omissions are significant. The current manuscript does not report the total number of interactions analyzed, sampling method, annotation protocol, inter-rater reliability, or statistical tests. In the revised version, we will add this information to the Methods section (including dataset size, collection protocol, coding details, and reliability metrics such as inter-rater agreement) and update the abstract and results sections to reference these details, enabling evaluation of the reported percentages. revision: yes

  2. Referee: [Methods] Methods / coding procedure: the mapping from surface dialogue features (e.g., "outsourcing", "epistemic justification") to AIR constructs is asserted but not validated against text-only data; no evidence is supplied that the annotation recovers intended epistemic aims and processes without substantial context loss or misclassification.

    Authors: The manuscript asserts the mapping of dialogue features to AIR constructs without providing validation evidence or reliability metrics for text-only annotation. We will revise the Methods section to include a detailed description of the coding scheme development, explicit mappings with examples, and inter-rater reliability statistics. A comprehensive external validation study to rule out context loss may exceed the scope of the current work, but we will add available evidence from the annotation process and acknowledge limitations. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; analysis applies external AIR framework to independent dialogue data

full rationale

The paper's central results (78.8% low EAIL, 11.1% high engagement) are produced by applying the pre-existing AIR framework to classify observable dimensions in an external human-AI dialogue dataset. No equations, fitted parameters, or self-citations are used to derive the percentages; the mapping from surface features to epistemic constructs is presented as an operationalization step rather than a definitional equivalence. The derivation chain remains self-contained against external benchmarks and does not reduce any claim to its own inputs by construction.

Assumptions & free parameters 1 free parameters · 1 assumptions · 1 invented entities

The central claim rests on the applicability of the AIR framework to GenAI co-programming and the accuracy of the coding scheme for detecting the listed dimensions in dialogue data.

free parameters (1)
  • threshold for high epistemic engagement
    The 11.1% figure depends on how coupling of mastery-oriented aims with advanced strategies is operationalized, which may involve unstated rules or cutoffs.
assumptions (1)
  • domain assumption The AIR framework provides a valid model for epistemic thinking in educational contexts involving AI.
    The study draws on it to examine aims and processes in GenAI-supported co-programming activities.
invented entities (1)
  • Epistemic AI Literacy (EAIL)
    purpose: Reframing AI literacy as a process-oriented epistemic phenomenon that emerges through dynamic human-AI interactions.
    New conceptual framework introduced by the study to organize the analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constructing Epistemic AI Literacy: Detecting Epistemic Aims and Processes in Student-AI Co-Programming." pith.science (2026). https://pith.science/paper/WTFMTU4Q

@misc{pith2026260700211,
  author       = {Pith},
  title        = {Pith review of: Constructing Epistemic AI Literacy: Detecting Epistemic Aims and Processes in Student-AI Co-Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WTFMTU4Q}},
  note         = {Machine review of arXiv:2607.00211}
}
read the original abstract

Epistemic thinking plays a central role in students' learning processes when applying generative artificial intelligence (GenAI), particularly in programming contexts where learners must construct queries, evaluate and validate AI-generated outputs, and regulate problem-solving strategies. This study introduces the conceptual framework of Epistemic AI Literacy (EAIL), reframing AI literacy as a process-oriented epistemic phenomenon that emerges through dynamic human-AI interactions across different domains. Drawing on the AIR (epistemic aims, ideals and reliable epistemic processes) framework, this study examines how epistemic aims and epistemic processes are enacted in GenAI-supported co-programming activities and explores scalable approaches for operationalizing these constructs in interaction data. Using a large dialogue dataset of human-AI co-programming, this study identifies observable dimensions of epistemic aims (i.e., mastery-oriented aims) and epistemic processes (i.e., outsourcing, explanation seeking, verification seeking, prompt monitoring, and epistemic justification). The results reveal a prevalent lack of EAIL, with 78.8% of student-GenAI interactions relying on non-mastery-oriented aims and less reliable epistemic strategies like outsourcing and verification-seeking. Conversely, only 11.1% of interactions showed high epistemic engagement, where mastery-oriented aims were coupled with advanced epistemic strategies like epistemic justification in a more reliable epistemic process.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

  1. [1]

    F. Miao, W. Holmes, International Forum on AI and the Futures of Education: Developing Compe- tencies for the AI Era, 7-8 December 2020: Synthesis Report, Technical Report, UNESCO, Paris,

    work page 2020

  2. [2]

    scale for the assessment of non-experts’ ai literacy

    URL: https://unesdoc.unesco.org/. I. H. Y. Yim, A critical review of teaching and learning artificial intelligence (ai) literacy: Develop- ing an intelligence-based ai literacy framework for primary school education, Computers and Education: Artificial Intelligence 7 (2024) 100319. D. Long, B. Magerko, What is ai literacy? competencies and design consider...

    work page 2024

  3. [3]

    M. Cosgun Ogeyik, Using bloom’s digital taxonomy as a framework to evaluate webcast learning experience in the context of covid-19 pandemic, Education and Information Technologies 27 (2022) 11219-11235. T. Lintner, A systematic review of ai literacy scales, npj Science of Learning 9 (2024)

    work page 2022

  4. [4]

    S. C. Kong, J. Zhu, Y. N. Yang, Developing and validating a scale of empowerment in using artificial intelligence for problem-solving for senior secondary and university students, Computers and Education: Artificial Intelligence 8 (2025) 100359. S. Barzilai, A. Zohar, Reconsidering personal epistemology as metacognition: A multifaceted approach to the ana...

    work page 2025

  5. [5]

    Richter, S

    T. Richter, S. Schmid, Epistemological beliefs and epistemic strategies in self-regulated learning, Metacognition and Learning 5 (2010) 47-65. K. R. Muis, G. M. Franco, Epistemic profiles and metacognition: Support for the consistency hypothesis, Metacognition and learning 5 (2010) 27-45. K. R. Muis, G. M. Franco, Epistemic beliefs: Setting the standards ...

    work page 2010

  6. [6]

    McNichols, F

    H. McNichols, F. Ikram, A. Lan, The studychat dataset: Student dialogues with chatgpt in an artificial intelligence course, arXiv preprint arXiv:2503.07928 (2025). D. S. McNamara, A. C. Graesser, P. M. McCarthy, Z. Cai, Automated evaluation of text and discourse with Coh-Metrix, Cambridge University Press,

    work page arXiv 2025

  7. [7]

    Y. R. Tausczik, J. W. Pennebaker, The psychological meaning of words: Liwc and computerized text analysis methods, Journal of language and social psychology 29 (2010) 24-54. C. Rosé, Y.-C. Wang, Y. Cui, J. Arguello, K. Stegmann, A. Weinberger, F. Fischer, Analyzing collaborative learning processes automatically: Exploiting the advances of computational li...

    work page arXiv 2010

Pith tools

Reviewed July 2, 2026 · model on record in the stance chip above.