Empowering Educators in the Age of AI: An Empirical Study on Creating custom GPTs in Qualitative Research Method education

Qian Huang; Thijs Willems

REVIEW 4 major objections 5 minor 8 references

Empowering Educators in the Age of AI: An Empirical Study on Creating custom GPTs in Qualitative Research Method education

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two instructors built four custom GPT tools for a qualitative research methods course and found that the tools deepened student reflexivity, improved interview technique, and supported structured analysis when the design was aligned with…

desk verdict A candid, useful case study of educator-built GPTs for qualitative methods teaching, but the causal claims outrun the evidence from 14 self-reports. read the letter →

arxiv 2507.21074 v1 pith:T2KKRCH3 submitted 2025-06-10 cs.HC cs.AI

Qian Huang , Thijs Willems This is my paper

classification cs.HCcs.AI

keywords customGPTgenerativeAIineducationTPACKframeworkqualitativeresearchmethodseducator-leddesignactionstudentreflexivityliteracy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Two instructors in a Master's-level qualitative research methods course built four custom GPT tools—for research question formulation, interview practice, fieldnote analysis, and design thinking—and studied what happened when students used them. Based on thematic analysis of student reflections, chat logs, and final assignments, the paper claims the tools deepened reflexivity, sharpened interview technique, and supported structured analytic thinking, while also producing cognitive overload, reduced immersion in data, and formulaic responses. The central argument is that AI becomes pedagogically meaningful in qualitative education when educators, not just students, design it: aligning technological choices with content knowledge and pedagogical intent through the TPACK framework. The paper concludes that custom GPTs can serve as cognitive partners in iterative research practice, provided human facilitation remains in place.

What carries the argument

The carrying mechanism is the pairing of four instructor-built GPTs — QualiQuest Buddy (research question refinement with epistemological and positionality prompts), Research Interview Simulator (adaptive personas with ethical dilemmas and probing feedback), Observation Station (fieldnote analysis separating observation, interpretation, and reflection), and DT X Urban Studies (applying the Design Thinking Double Diamond framework) — with the TPACK framework as the design lens and an action-research plan-act-reflect cycle as the iterative method. Thematic analysis (Braun & Clarke) of weekly reflections, anonymized chat logs, and final assignments supplies the evidence for claimed learning gains.

What would settle it

A controlled comparison in which one section uses the four custom GPTs and a matched section does equivalent non-AI exercises (peer role-play of interviews, instructor feedback on fieldnotes), with final assignments and reflections coded by independent raters blind to condition, would settle whether the tools themselves cause the gains. Simpler: two independent coders re-analyze the existing reflection logs; if the reported themes do not replicate with acceptable inter-rater agreement, the findings are likely expectation artifacts.

Watch

Extended reading notes

Core claim

The authors claim that educator-designed custom GPTs embedded in active learning improve qualitative research education in measurable ways. Students reported greater awareness of their positionality and biases, more responsive and less leading interview questions after simulated practice, and clearer frameworks for interpreting fieldnotes and urban challenges. The authors interpret these outcomes as evidence that AI can scaffold—not replace—the interpretive, reflective work central to qualitative inquiry, and that the TPACK alignment of content, pedagogy, and technology is what distinguishes meaningful AI integration from generic chatbot use. Their key insight for future practice is that educators should be empowered as AI designers through no-code customization, iterative feedback loops, and institutional support.

Load-bearing premise

The claim of learning gains rests on students' self-reported reflections and AI chat logs, interpreted by the two instructors who designed and taught with the tools, with no independent coding, inter-rater reliability check, or comparison group to rule out expectation and social-desirability effects.

Editorial extensions

If this is right

Custom GPTs offer a no-code route for instructors to build discipline-specific AI scaffolds in other methods courses, not just qualitative ones.
The reported gains suggest AI can support reflective, interpretive skills—reflexivity and positionality awareness—rather than only automating surface tasks.
The documented challenges imply AI feedback should be scaled and scaffolded, with human facilitation retained to prevent over-reliance and shallow data engagement.
The study supports institutional investment in faculty AI-design training and pedagogical AI labs, since educator design capacity becomes the bottleneck.
AI literacy in this model includes critiquing the epistemology of AI outputs, not just writing better prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would run the same four GPT designs in a second, unrelated qualitative methods course and have independent raters blind to condition code the reflections; if 'enhanced reflexivity' does not replicate, the finding is partly an artifact of instructor expectation.
The design pattern likely transfers to quantitative methods education—for example, a GPT that forces students to justify model choices—which would test whether the reflexivity gains generalize beyond qualitative content.
Because the tools were iteratively revised on student feedback, the active ingredient may be the feedback loop rather than any specific prompt; a comparison course using fixed, non-iterated GPTs could isolate that.
The 'reduced immersion' concern suggests a design variant where students must hand-code a passage before seeing AI suggestions, then compare reflection depth against the current tool version.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is a readable, honest case study, not a rigorous evaluation. It is new in a narrow but real sense: I have not seen a published worked example of instructors building custom GPTs specifically for qualitative research methods teaching. The four tools are described, and the design process is tied to TPACK concretely. Credit where due: the authors include student criticisms — cognitive overload, reduced immersion in data, formulaic responses — which is more candid than most AI-in-education papers. The no-code customization point, where faculty with pedagogical expertise can shape tools without programming, is worth making.

The soft spot is exactly where your reader put it: the causal verbs in the abstract and findings. "Enhanced reflexivity," "improved interview techniques," and "supported structured analytic thinking" all rest on 14 students' self-reports, analyzed by the same two instructors who designed and taught with the tools. There is no baseline, no comparison condition, no inter-rater reliability check, and no independent audit. So the evidence supports "students perceived benefits and also raised concerns" — not "the tools enhanced learning." The RQ3 conclusion that educator-led design is "critical" is best read as a design principle the authors brought in, not a result that falls out of the data. The paper would be stronger if reframed as an exploratory case study of perceived usefulness and design process.

I would also flag that the appendix offers QR codes to the GPTs but no prompts or configurations, so the custom design is not inspectable. There is no limitations section addressing the researcher-instructor double role or the single-course context, which matters in a qualitative study where the researchers are the intervention.

That said, the stress-test overstates the damage in one respect. The negative quotes are not garnish; some students explicitly said the AI "robbed" them of data immersion, so the picture is mixed rather than one-sidedly positive. And the paper sits in an interpretive tradition — if the abstract said "students reported enhanced reflexivity," the evidence would be adequate. The gap is between "reported" and "enhanced."

My read: this is a useful worked example for educators and for the AI-in-education community. It deserves a serious referee, with expected major revisions being a moderation of claims, a genuine limitations paragraph, and sharing of tool specifications. I would not cite it for the causal claim, but I would cite it as a case of educator-led customization.

Recommendation: send to peer review. It is not ready as is, but it is a legitimate empirical case with honest data.

Referee Report

4 major / 5 minor

Summary. This paper reports an action-research study in which two instructors designed four custom GPT tools (QualiQuest Buddy, Research Interview Simulator, Observation Station, DT X Urban Studies) for a Master's-level Qualitative Research Methods course in Urban Planning Policy. Using the TPACK framework, the instructors embedded the tools into active-learning activities and collected student reflections, GPT conversation logs, and final assignments. Thematic analysis led the authors to report that the tools enhanced reflexivity, improved interview techniques, and supported structured analytic thinking, while also surfacing concerns about cognitive overload, reduced data immersion, prompting dependence, access limits, and formulaic responses. The paper's three central insights are that AI can scaffold active learning with human facilitation, that custom GPTs can serve as cognitive partners, and that educator-led design is critical for meaningful AI integration. The contribution is positioned as addressing gaps in educator-driven AI design and AI use in qualitative research methods education.

Significance. If the central claims were adequately supported, the paper would offer a useful empirical illustration of how TPACK can guide educators in creating domain-specific GPT tools, a growing area of interest in AI-and-education research. The study is commendable for including critical student voices, for documenting concrete tool designs, and for attempting a multi-source data collection in a real classroom setting. The no-code customization pathway and the explicit link to disciplinary content are valuable practical contributions. However, as presented, the significance is limited by the gap between the strong causal language in the abstract and conclusion and the self-report, single-condition evidence base. The paper is better read as an exploratory design case offering hypotheses about educator-led AI design, not as a demonstration of learning gains or a proof that educator-led design is uniquely critical.

major comments (4)

[Abstract; Findings (RQ2)] The central claim that the custom GPTs 'enhanced student reflexivity, improved interview techniques, and supported structured analytic thinking' uses causal learning-outcome language. The evidence consists of students' self-reported reflections, interpreted by the instructor-researchers who designed and taught with the tools, with no comparison condition, no baseline measure, and no objective learning assessment. This supports only a claim of perceived usefulness in one context. Please either soften the abstract and findings to 'students reported enhanced reflexivity...' or add a design that can support causal inference.
[Data Analysis: Thematic Analysis Approach] The thematic analysis was conducted by the two instructor-researchers who designed the tools, taught the course, and elicited the reflections. The paper reports no inter-coder reliability check, no external audit, and no coding scheme beyond broad theme names. If the authors intend a reflexive thematic analysis in the Braun and Clarke (2021) tradition, that choice can make inter-coder reliability inappropriate, but then the paper should explicitly say so and explain how the dual instructor-researcher role was managed. As written, the coding process lacks transparency, and the themes may be influenced by the researchers' expectations, which is load-bearing for RQ2 and RQ3.
[Data Collection Methods; Findings] The methods section lists GPT conversation logs and final assignments as data sources for triangulation, but the findings section does not report any systematic analysis of these sources. No excerpts from chat logs are presented, and no evidence links final assignment quality to use of the tools. The claim that AI-supported learning 'translated into improved research skills' is therefore not supported by the data actually shown. Please either report on the log and assignment analyses or limit the claims to students' self-reported experiences.
[RQ3 — Insights for Future Educator-Led AI Design and Pedagogical Innovation] The third insight, that 'educator-led design is critical,' is not empirically testable with this study's design: all tools were educator-designed, and no comparison was made with generic, non-educator-designed, or student-designed AI tools. The conclusion is embedded in the study's premise and in the selection of the TPACK framework. This should be reframed as a design principle or a hypothesis for future research, not a finding emerging from the data.

minor comments (5)

[Introduction] There is a typo, 'Existant qualitative research pedagogy', in the second paragraph of the Literature Review; it should be 'Existing'.
[Methodology: Participants and Context] The text lists 'Instructor (n = 2)' but then says 'The course instructor, who also served as the primary researcher'; please clarify whether one or both instructors were also researchers and how roles were divided.
[Methodology: Data Collection Methods] In the sentence 'These GPTs were embedded into post-class activities, onering students opportunities', 'onering' appears to be a typo for 'offering'.
[References] The reference 'McNin, J., & Whitehead, J. (2011)' should be 'McNiff, J., & Whitehead, J.'; also, the Mishra and Koehler (2006) reference appears twice with identical details.
[Appendix] The appendix provides QR codes for the four GPTs, but these are not reproducible or fully evaluable by readers without OpenAI accounts. Consider including prompt templates, system instructions, or design logs as supplemental material to support replication.

Circularity Check

1 steps flagged · score 6.0 of 10

The RQ3 conclusion that educator-led design is critical restates the TPACK-based design premise that produced the intervention, making the central insight circular by construction.

self definitional [Findings, RQ3 (Insight 3: 'Empowering Educators to Lead AI Design through Pedagogical Customization'); cf. Theoretical Framework and Methodology/Participants]
"While much research emphasizes student-AI interaction, this study highlights the pivotal role of educators as AI designers. The instructors' deep disciplinary knowledge and pedagogical intent enabled them to create custom GPTs that aligned with specific learning outcomes... This approach ensured AI functioned not as an external plug-in, but as an extension of the instructors' pedagogical vision."

The RQ3 claim that educator-led design is pivotal/critical is the same TPACK premise used to construct the intervention: the Theoretical Framework defines educator-led design as aligning CK, PK, and TK, and RQ1 documents that the two instructors did exactly that for all four GPTs. All tools were educator-designed; no non-educator-designed or generic GPT arm was studied, and the instructor 'also served as the primary researcher.' There is no variation from which 'educator-led design' as a cause could be inferred. The insight is thus the design protocol restated as an empirical outcome, equivalent to its input by construction.

full rationale

This paper has no self-citation chains, imported uniqueness theorems, or fitted parameters, so the usual circularity suspects are absent. The circularity is in the central qualitative inference: the headline contribution—'educator-led design is critical to pedagogically meaningful AI integration'—is embedded in the single-arm action-research design where the instructors built every GPT using TPACK, taught with them, and then coded the student reflections themselves. RQ3's three insights (AI as scaffold, cognitive partner, educator-led design) are the design commitments that generated the tools, re-presented as findings; no comparison condition or independent audit lets the data adjudicate whether the observed benefits came from the educator-designed GPTs, the active-learning activities, or demand characteristics. The RQ2 student quotes and negative comments are genuine evidence of perceived usefulness, which prevents this from being a fully forced derivation; the paper also reports challenges, not only successes. Because the central causal claim reduces to the study's premise rather than being tested against an alternative, the circularity score is 6, not 0-2. This is a partial, design-level circularity rather than a formal equation-level one.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The paper introduces no numeric free parameters. The central inference rests on interpretive assumptions about TPACK, action research, self-reported learning, and analyst objectivity, plus the specific custom GPT artifacts.

assumptions (4)

domain assumption TPACK provides a valid lens for analyzing AI integration
The paper uses TPACK (Mishra & Koehler, 2006) as its theoretical framework to justify tool design and interpretation, but does not test or compare TPACK against alternative frameworks.
domain assumption Action research yields trustworthy practical knowledge
The study relies on the action-research tradition (Kemmis & McTaggart, 2000) to claim the findings are applicable to practice, despite the absence of external validation.
domain assumption Student self-reports reflect actual learning
The findings on reflexivity and interview improvement come from student reflections and chat logs; the paper assumes these are accurate rather than socially desirable or prompted.
domain assumption Thematic analysis by the instructor-researchers is objective
The coding was done by the two instructors who designed the tools, with no independent coders or audit trail described, so the assumption is that their interpretation is credible.

invented entities (1)

Four custom GPT tools (QualiQuest Buddy, Research Interview Simulator, Observation Station, DT X Urban Studies)
purpose: Scaffold qualitative research tasks: question formulation, interview practice, fieldnote analysis, and design thinking
The tools are the intervention. The paper gives only QR codes and broad behavioral descriptions; their exact prompts, system instructions, and outputs are not available for independent verification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Empowering Educators in the Age of AI: An Empirical Study on Creating custom GPTs in Qualitative Research Method education." pith.science (2026). https://pith.science/paper/T2KKRCH3

@misc{pith2026250721074,
  author       = {Pith},
  title        = {Pith review of: Empowering Educators in the Age of AI: An Empirical Study on Creating custom GPTs in Qualitative Research Method education},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2KKRCH3}},
  note         = {Machine review of arXiv:2507.21074}
}

read the original abstract

As generative AI (Gen-AI) tools become more prevalent in education, there is a growing need to understand how educators, not just students, can actively shape their design and use. This study investigates how two instructors integrated four custom GPT tools into a Masters-level Qualitative Research Methods course for Urban Planning Policy students. Addressing two key gaps: the dominant framing of students as passive AI users, and the limited use of AI in qualitative methods education. The study explores how Gen-AI can support disciplinary learning when aligned with pedagogical intent. Drawing on the Technological Pedagogical Content Knowledge (TPACK) framework and action research methodology, the instructors designed GPTs to scaffold tasks such as research question formulation, interview practice, fieldnote analysis, and design thinking. Thematic analysis of student reflections, AI chat logs, and final assignments revealed that the tools enhanced student reflexivity, improved interview techniques, and supported structured analytic thinking. However, students also expressed concerns about cognitive overload, reduced immersion in data, and the formulaic nature of AI responses. The study offers three key insights: AI can be a powerful scaffold for active learning when paired with human facilitation; custom GPTs can serve as cognitive partners in iterative research practice; and educator-led design is critical to pedagogically meaningful AI integration. This research contributes to emerging scholarship on AI in higher education by demonstrating how empowering educators to design custom tools can promote more reflective, responsible, and collaborative learning with AI.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 6 canonical work pages

[1]

Restricted Restricted Empowering Educators in the Age of AI: An Empirical Study on Creating custom GPTs in Qualitative Research Method education Qian Huang, Lee Kuan Yew Centre for Innovative Cities, Singapore University of Technology and Design, Singapore. Email: qian_huang@sutd.edu.sg Thijs Willems, Lee Kuan Yew Centre for Innovative Cities, Singapore U...

work page 2022
[2]

The use of AI to support research methods education has been far more common in quantitative disciplines, with relatively little work on its pedagogical role in qualitative research education. Situated within a Master’s-level course on Qualitative Research Methods for Urban Planning Policy students, this study investigates how two instructors designed, im...

work page 2006
[3]

developed AI prompting techniques and customization skills (TK). By applying TPACK, this study provides a framework for future AI-driven pedagogical innovations, demonstrating how AI can be contextualized within subject-specific education rather than being applied generically. Restricted Restricted Methodology Research Design: Action Research Approach Thi...

work page 2000
[4]

DT X Urban Studies GPT – Supports students in applying the Design Thinking Double Diamond Framework to urban challenges. These GPTs were embedded into post-class activities, onering students opportunities to engage interactively with qualitative research concepts and practice skills outside of formal lecture hours. Data Collection Methods Data were collec...

work page 2017
[7]

Empowering Educators to Lead AI Design through Pedagogical Customization While much research emphasizes student-AI interaction, this study highlights the pivotal role of educators as AI designers. The instructors’ deep disciplinary knowledge and pedagogical intent enabled them to create custom GPTs that aligned with specific learning outcomes (e.g., quest...

arXiv 2023
[109]

https://doi.org/10.3390/educsci13020109 Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x Roulston, K. (2010). Reflective interviewing: A guide to theory and practice. SAGE Publications. Saldaña, J...

arXiv 2006
[2021]

AI-enhanced reflexivity,

was employed, allowing for a systematic examination of student reflections, GPT interaction logs, and class discussions. This method was chosen for its flexibility in capturing patterns, meanings, and nuanced insights from qualitative data. The analysis followed a structured six-step process. First, the researchers engaged in familiarization with the data...

work page 2021
[2023]

In other words, AI should enhance, not replace, traditional learning methods, ensuring critical engagement rather than passive AI use

have shown, for instance, that effective AI integration requires teachers to balance technology with disciplinary expertise. In other words, AI should enhance, not replace, traditional learning methods, ensuring critical engagement rather than passive AI use. This would require educators to have AI literacy training to develop and customize tools rather t...

work page 2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Restricted Restricted Empowering Educators in the Age of AI: An Empirical Study on Creating custom GPTs in Qualitative Research Method education Qian Huang, Lee Kuan Yew Centre for Innovative Cities, Singapore University of Technology and Design, Singapore. Email: qian_huang@sutd.edu.sg Thijs Willems, Lee Kuan Yew Centre for Innovative Cities, Singapore U...

work page 2022

[2] [2]

The use of AI to support research methods education has been far more common in quantitative disciplines, with relatively little work on its pedagogical role in qualitative research education. Situated within a Master’s-level course on Qualitative Research Methods for Urban Planning Policy students, this study investigates how two instructors designed, im...

work page 2006

[3] [3]

developed AI prompting techniques and customization skills (TK). By applying TPACK, this study provides a framework for future AI-driven pedagogical innovations, demonstrating how AI can be contextualized within subject-specific education rather than being applied generically. Restricted Restricted Methodology Research Design: Action Research Approach Thi...

work page 2000

[4] [4]

DT X Urban Studies GPT – Supports students in applying the Design Thinking Double Diamond Framework to urban challenges. These GPTs were embedded into post-class activities, onering students opportunities to engage interactively with qualitative research concepts and practice skills outside of formal lecture hours. Data Collection Methods Data were collec...

work page 2017

[5] [7]

Empowering Educators to Lead AI Design through Pedagogical Customization While much research emphasizes student-AI interaction, this study highlights the pivotal role of educators as AI designers. The instructors’ deep disciplinary knowledge and pedagogical intent enabled them to create custom GPTs that aligned with specific learning outcomes (e.g., quest...

arXiv 2023

[6] [109]

https://doi.org/10.3390/educsci13020109 Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x Roulston, K. (2010). Reflective interviewing: A guide to theory and practice. SAGE Publications. Saldaña, J...

arXiv 2006

[7] [2021]

AI-enhanced reflexivity,

was employed, allowing for a systematic examination of student reflections, GPT interaction logs, and class discussions. This method was chosen for its flexibility in capturing patterns, meanings, and nuanced insights from qualitative data. The analysis followed a structured six-step process. First, the researchers engaged in familiarization with the data...

work page 2021

[8] [2023]

In other words, AI should enhance, not replace, traditional learning methods, ensuring critical engagement rather than passive AI use

have shown, for instance, that effective AI integration requires teachers to balance technology with disciplinary expertise. In other words, AI should enhance, not replace, traditional learning methods, ensuring critical engagement rather than passive AI use. This would require educators to have AI literacy training to develop and customize tools rather t...

work page 2023