AI Slop or AI-enhancement? Student perceptions of AI-generated media for an English for Academic Purposes course
Pith reviewed 2026-05-21 09:43 UTC · model grok-4.3
The pith
Teacher-prompted AI tools can produce videos and infographics that students in academic English courses rate as useful and some adopt for extra help.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Teacher-prompted retrieval-augmented generation with tools such as Google Notebook LM converts course materials and student work into diverse multimedia supplements. Students in the EAP course perceived these materials as useful and easy to use, favoring visual and multimodal formats for assessment-linked content. Video preference showed a positive correlation with academic performance, whereas higher cognitive load correlated negatively with grades. Some lower-performing students independently treated the materials as remedial scaffolds, and the approach enabled scalable individualized feedback that would be difficult to produce manually.
What carries the argument
Teacher-prompted retrieval-augmented generation (RAG) applied to course materials to produce videos, podcasts, infographics, and individualized feedback reports for EAP learners.
If this is right
- Students accept and prefer visual and multimodal AI content when it connects directly to assessments.
- Preference for video formats associates with higher course performance.
- Material complexity must stay within manageable cognitive load to avoid negative effects on grades.
- Lower-performing students can adopt AI materials independently as remedial resources.
- Scalable personalized feedback becomes feasible for groups where manual creation would be impractical.
Where Pith is reading between the lines
- The same teacher-directed RAG method could be tested in other language or subject courses if cognitive load and goal alignment are maintained.
- Departments might explore training instructors on prompt design to keep generated materials focused and low-load.
- Longer-term tracking would be needed to check whether reliance on AI supplements affects independent skill development over a full program.
- Institutions could compare costs and outcomes of RAG supplements against hiring additional teaching assistants for similar personalization.
Load-bearing premise
Self-reported survey answers, interviews, and grade correlations in one course accurately reflect genuine learning gains rather than response bias or other unmeasured factors.
What would settle it
A follow-up study that randomly assigns students to AI-supplemented or standard sections and measures language skill gains on an independent standardized test instead of self-reports and course grades would show no advantage or a disadvantage for the AI group.
read the original abstract
Artificial intelligence (AI) retrieval-augmented generation (RAG) tools now enable educators to transform course materials into diverse multimedia at scale. However, it remains unclear whether such AI-generated content functions as a pedagogical scaffold or AI slop: high volume, low quality material. This innovative practice paper reports on the development, implementation, and evaluation of teacher-prompted, AI-generated supplemental materials in an English for Academic Purposes (EAP) course at a Hong Kong Community College. Using primarily Google Notebook LM, the instructor generated videos, podcasts, infographics, and individualized feedback reports from course materials and student work for 106 English as a Foreign Language learners. An explanatory sequential mixed-methods design comprising a survey, semi-structured interviews, and correlation analysis with academic scores was employed to examine students' preferences, perceptions, and learning outcomes. Findings are framed through the Technology Acceptance Model and Cognitive Load Theory. Students rated the materials highly for perceived usefulness and ease of use, and preferred assessment-linked content presented in visual and multimodal formats, particularly videos and infographics. Video preference correlated positively with academic performance; however, higher cognitive load was negatively associated with course grades, indicating that material complexity must be carefully calibrated. Notably, some lower-performing students independently adopted the materials as remedial scaffolds. The practice demonstrates that RAG tools enable scalable personalized feedback that would be less feasible through traditional methods. When aligned with student goals and cognitive principles, teacher-prompted AI generation can meaningfully enhance the EAP learning ecosystem rather than producing AI slop.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes an innovative practice study in which an instructor used Google Notebook LM (a RAG tool) to generate supplemental videos, podcasts, infographics, and individualized feedback reports from existing EAP course materials and student work for 106 EFL learners at a Hong Kong community college. An explanatory sequential mixed-methods design (TAM-based survey, semi-structured interviews, and correlation of material preferences with final grades) is used to examine student perceptions, format preferences, and associations with academic performance. Key findings include high ratings for perceived usefulness and ease of use, preference for assessment-linked visual/multimodal content (especially videos and infographics), a positive correlation between video preference and course grades, a negative association between reported cognitive load and grades, and independent remedial use by some lower-performing students. The authors conclude that teacher-prompted AI generation, when aligned with student goals and Cognitive Load Theory principles, can enhance rather than degrade the EAP learning ecosystem.
Significance. If the empirical associations hold after addressing methodological limitations, the work offers a concrete, scalable example of RAG-enabled personalization in language education that would be labor-intensive to produce manually. The integration of TAM and Cognitive Load Theory to interpret preferences and load calibration provides a useful interpretive lens, and the observation that lower-performing students voluntarily adopted the materials as scaffolds is a noteworthy practical insight. The study contributes to the emerging literature on AI-generated educational media by focusing on teacher-controlled prompting rather than fully autonomous generation.
major comments (2)
- [Methods] Methods section (explanatory sequential design): the absence of a control group, pre-post language-skill measures, or any randomization means that the positive correlation between video preference and final grades cannot distinguish genuine pedagogical enhancement from selection effects, prior ability, or motivation confounds. This directly bears on the central claim that the materials 'meaningfully enhance' the learning ecosystem.
- [Results] Results (correlation analysis with academic scores): the reported positive association between video preference and grades and the negative association between cognitive load and grades are presented without statistical controls or effect-size details; given the single-course, non-randomized setting, these correlations remain vulnerable to unmeasured confounders and do not yet support causal inferences about enhancement over traditional methods.
minor comments (3)
- [Abstract] Abstract and Methods: the survey instrument is described only as 'TAM-based'; providing the exact items, response scale, and any reliability statistics (Cronbach's alpha) would allow readers to assess measurement quality.
- [Discussion] Discussion: the claim that 'some lower-performing students independently adopted the materials as remedial scaffolds' is intriguing but would benefit from a brief quote or theme excerpt from the interview data to illustrate how this adoption was identified.
- [Discussion] The manuscript would be strengthened by an explicit limitations paragraph that addresses single-institution scope, response bias in self-report data, and the lack of objective learning-outcome metrics beyond final grades.
Simulated Author's Rebuttal
We thank the referee for their constructive feedback, which highlights key limitations in interpreting our findings from this innovative practice study. We agree that the design precludes strong causal claims and will revise the manuscript to moderate language around enhancement while preserving the value of the observed associations and perceptions. We address each major comment below.
read point-by-point responses
-
Referee: [Methods] Methods section (explanatory sequential design): the absence of a control group, pre-post language-skill measures, or any randomization means that the positive correlation between video preference and final grades cannot distinguish genuine pedagogical enhancement from selection effects, prior ability, or motivation confounds. This directly bears on the central claim that the materials 'meaningfully enhance' the learning ecosystem.
Authors: We agree that the explanatory sequential mixed-methods design, conducted in a single course without randomization or a control group, does not support causal inferences regarding pedagogical enhancement. The positive correlation with grades is reported as an association that could reflect selection effects, motivation, or prior ability. In revision we will explicitly acknowledge this limitation in the methods and discussion sections and will revise the abstract and conclusion to replace phrasing such as 'meaningfully enhance' with more precise language focused on 'perceived usefulness,' 'observed associations,' and 'potential as scaffolds' within the constraints of a practice study. revision: yes
-
Referee: [Results] Results (correlation analysis with academic scores): the reported positive association between video preference and grades and the negative association between cognitive load and grades are presented without statistical controls or effect-size details; given the single-course, non-randomized setting, these correlations remain vulnerable to unmeasured confounders and do not yet support causal inferences about enhancement over traditional methods.
Authors: The referee is correct that the correlations lack statistical controls for confounders and do not include effect-size reporting. We will add Pearson correlation coefficients with 95% confidence intervals and effect-size interpretations in the revised results section. We will also expand the discussion to note the exploratory nature of these associations and the potential influence of unmeasured variables. However, because the study used existing course data from one cohort, we cannot introduce post-hoc controls for variables such as prior English proficiency without new data collection. revision: partial
Circularity Check
No circularity: empirical mixed-methods study based on primary data
full rationale
The paper reports an explanatory sequential mixed-methods design using student surveys (TAM), semi-structured interviews, and correlations between material preferences and academic scores from 106 learners in one EAP course. No equations, fitted parameters, predictions, or first-principles derivations appear; claims rest directly on collected responses and grade data rather than any self-referential reduction, self-citation load-bearing premise, or renaming of inputs as outputs. External frameworks (TAM, Cognitive Load Theory) are invoked for interpretation but do not create definitional loops within the reported analysis.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
(TAM: Perceive usefulness; ease of use) Can you explain how the combination of audio, text or visuals in your supplemental material aided your understanding?
-
[2]
(TAM and Cognitive Load: Coherence Principle) Can you provide an example of how the length of your supplemental material affected your ability to focus and learn?
-
[3]
(Cognitive load) Were there features in the supplemental material that helped you identify important information?
-
[4]
(Germane load) How can (the first author) improve your supplemental material to engage students in deeper thinking or reflection? Please explain
-
[5]
(Germane load) How can (the first author) improve your supplemental material to connect new information in (the EAP course) to students' existing knowledge? 23
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.