pith. sign in

arxiv: 2605.17055 · v1 · pith:F52D6PDVnew · submitted 2026-05-16 · 💻 cs.CY

Generative AI Feedback, English Writing and Teacher Rubrics: A Multiple-Case Study of CyberScholar

Raigul Zheldibayeva , Ana Karina de Oliveira Nascimento , Vania Castro , Bill Cope , Mary Kalantzis This is my paper

Pith reviewed 2026-05-20 15:22 UTC · model grok-4.3

classification 💻 cs.CY
keywords generative AIformative feedbackK-12 writingrubric-based assessmentstudent perceptionsteacher workloadmultiple-case studyrevision practices
0
0 comments X

The pith

CyberScholar delivers immediate generative AI feedback based on teacher rubrics that students use to revise and improve their writing.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This multiple-case study tests a generative AI tool called CyberScholar that folds teacher rubrics, materials, and examples into its responses through retrieval-augmented generation. Data from classroom observations, student surveys, focus groups, and teacher reports across five schools show students value the quick, criterion-specific comments and report gains in organization, elaboration, and style during revisions. Teachers say the tool cuts time spent on routine feedback and lets them focus on higher-order teaching moves. A sympathetic reader would care because the findings point to one practical way to give students more frequent writing support without removing teacher judgment from the process.

Core claim

The study found that students valued CyberScholar's immediate, rubric-based feedback and noticed improvements in their writing as they revised, using it to refine organization, elaboration, and style. The tool's interactive qualities fostered revision and reduced reliance on teacher feedback, while teachers reported time savings and support for more targeted instructional practices, though inconsistencies in automated ratings and occasional misalignment with expectations were also observed.

What carries the argument

CyberScholar, a generative AI system that uses retrieval-augmented generation to incorporate teacher-provided rubrics, materials, and exemplars for producing criterion-specific formative feedback and ratings.

If this is right

  • Students can complete more iterative revision cycles with less dependence on direct teacher input.
  • Teachers can shift attention from routine feedback to higher-order instructional practices.
  • Improvements appear in specific writing dimensions such as organization, elaboration, and style.
  • The same approach can be used across disciplines and in grades 7 through 11.
  • Human oversight is still required to catch and correct rating inconsistencies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Adding blinded pre-post writing assessments would test whether the reported improvements hold up beyond student perception.
  • The same rubric-grounded method could be tried for feedback on science lab reports or history essays.
  • Once rating calibration improves, the tool might support writing practice in larger classes or after-school settings.
  • Longer-term tracking could reveal whether frequent AI assistance changes how students develop independent revision habits.

Load-bearing premise

That student self-reports of writing improvement and teacher perceptions of time savings, together with classroom observations, accurately reflect real gains in skills and instructional changes without objective pre-post measures or controlled comparisons.

What would settle it

A follow-up experiment that collects student writing samples before and after use of CyberScholar, scores them blindly with the original rubrics, and compares the size of improvement against a control group that receives only traditional feedback.

Figures

Figures reproduced from arXiv: 2605.17055 by Ana Karina de Oliveira Nascimento, Bill Cope, Mary Kalantzis, Raigul Zheldibayeva, Vania Castro.

Figure 1
Figure 1. Figure 1: Work editor (Source: CyberScholar interface. Used with permission.) [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Workflow (Source: CyberScholar interface. Used with permission.) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
read the original abstract

This multiple-case study examined the potential of a Generative AI (GenAI) tool, CyberScholar, to support K-12 students' writing across disciplines. This tool integrates teacher-provided rubrics, materials, and exemplars through Retrieval-Augmented Generation (RAG), producing criterion-specific formative feedback and ratings. The study involved 143 students and five teachers in grades 7 through 11 across five U.S. middle and high schools. Data sources included classroom observations, student post-surveys (n = 79), student focus group interviews (n = 18), and teacher surveys (n = 5). Qualitative analysis followed two cycles of coding to identify patterns within and across cases. Findings indicate that students valued CyberScholar's immediate, rubric-based feedback and noticed improvements in their writing as they revised, using it to refine organization, elaboration, and style. They also highlighted the tool's interactive, iterative qualities, which fostered revision and reduced reliance on teacher feedback. However, participants noted inconsistencies in the automated rating system and occasional misalignment with assignment expectations. Teachers reported that CyberScholar saved time on feedback and supported more targeted, higher-order instructional practices. The study underscores the promise of rubric-grounded GenAI formative feedback for developing writing skills, while emphasizing the need for human oversight, calibration of automated ratings, and attention to contextual factors shaping adoption.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. This multiple-case study examines CyberScholar, a generative AI tool that employs Retrieval-Augmented Generation to deliver criterion-specific formative feedback aligned with teacher-provided rubrics, materials, and exemplars. Conducted across five U.S. middle and high schools with 143 students in grades 7-11 and five teachers, the study draws on classroom observations, student post-surveys (n=79), focus group interviews (n=18), and teacher surveys (n=5). Two cycles of qualitative coding identify patterns showing that students valued the tool's immediate feedback for refining organization, elaboration, and style through iterative revisions and reduced teacher dependence, while teachers reported time savings and opportunities for higher-order instruction. The paper also notes inconsistencies in automated ratings and occasional misalignment with expectations, concluding with recommendations for human oversight and calibration.

Significance. If the reported perceptions are borne out by additional evidence, the work would contribute timely insights into rubric-grounded GenAI applications in K-12 writing instruction. It documents practical benefits such as fostering student revision cycles and freeing teacher time for targeted feedback, alongside implementation challenges like rating reliability. These findings could inform the design of future educational AI systems and highlight contextual factors affecting adoption, adding to the literature on responsible integration of generative tools in classrooms.

major comments (3)
  1. [Abstract] Abstract and Findings: The central claim that students 'noticed improvements in their writing as they revised, using it to refine organization, elaboration, and style' rests solely on post-intervention self-reports from surveys and focus groups. No pre-post writing samples, blinded rubric scoring, or independent quality metrics are described to anchor these perceptions against actual skill gains.
  2. [Findings] Findings: Reports of reduced reliance on teacher feedback and iterative revision are interpreted as indicators of skill development, yet without objective pre-post measures or controls for teacher variability, these cannot reliably distinguish genuine writing improvement from placebo, social-desirability, or confirmation effects.
  3. [Methods] Methods: With data from only five teachers and 18 focus-group students, the cross-case patterns would benefit from explicit discussion of case selection criteria, potential response biases, and how the two cycles of qualitative coding ensured consistency across the small sample.
minor comments (2)
  1. [Abstract] The abstract flags 'inconsistencies in the automated rating system' but provides no details on their frequency, nature, or impact on student revisions; adding this would strengthen context.
  2. Consider including a table summarizing participant demographics, response rates, and data sources by case to improve clarity and reproducibility.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for their constructive and detailed review. We address each major comment point by point below, clarifying the scope of our qualitative multiple-case study and indicating where revisions will strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract and Findings: The central claim that students 'noticed improvements in their writing as they revised, using it to refine organization, elaboration, and style' rests solely on post-intervention self-reports from surveys and focus groups. No pre-post writing samples, blinded rubric scoring, or independent quality metrics are described to anchor these perceptions against actual skill gains.

    Authors: Our study is a multiple-case qualitative exploration of students' and teachers' experiences with rubric-aligned GenAI feedback, not a controlled evaluation of writing skill acquisition. The abstract and findings sections report participants' self-described perceptions of improvement during revision cycles, which is consistent with the data sources and design. We will revise the abstract to foreground that these are students' reported perceptions rather than measured outcomes, and we will add an explicit limitations subsection acknowledging the absence of pre-post assessments or objective metrics. revision: partial

  2. Referee: [Findings] Findings: Reports of reduced reliance on teacher feedback and iterative revision are interpreted as indicators of skill development, yet without objective pre-post measures or controls for teacher variability, these cannot reliably distinguish genuine writing improvement from placebo, social-desirability, or confirmation effects.

    Authors: The findings present students' accounts of iterative use and reduced teacher dependence as observed behaviors within the tool-supported revision process. We do not equate these reports with objective skill gains. To reduce any risk of overinterpretation, we will edit the findings and discussion sections to frame these strictly as self-reported engagement patterns and will add discussion of potential social-desirability and confirmation biases as study limitations. revision: yes

  3. Referee: [Methods] Methods: With data from only five teachers and 18 focus-group students, the cross-case patterns would benefit from explicit discussion of case selection criteria, potential response biases, and how the two cycles of qualitative coding ensured consistency across the small sample.

    Authors: We agree that additional methodological detail will improve transparency. In the revised manuscript we will expand the Methods section to describe the convenience-based selection of the five schools and teachers, note the voluntary nature of survey and focus-group participation and associated response biases, and elaborate on the two-cycle coding process, including how consistency was supported through team consensus meetings and analytic memoing. revision: yes

standing simulated objections not resolved
  • Providing pre-post writing samples, blinded rubric scoring, or other objective quality metrics, as these were outside the original qualitative case-study design and cannot be added without new data collection.

Circularity Check

0 steps flagged

No circularity in empirical qualitative study

full rationale

The paper is a multiple-case qualitative study relying on classroom observations, student post-surveys (n=79), focus groups (n=18), teacher surveys (n=5), and two cycles of coding to identify patterns. No mathematical derivations, equations, fitted parameters, predictions, or first-principles results are present. Claims about valued feedback and noticed improvements derive directly from coded empirical data sources without reduction to inputs by construction, self-definitional loops, or load-bearing self-citations. This is a standard interpretive research design whose findings are self-contained against the collected evidence.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical qualitative study with no mathematical modeling, free parameters, or new theoretical entities; relies on standard assumptions of qualitative research such as trustworthiness of self-report data.

pith-pipeline@v0.9.0 · 5795 in / 1085 out tokens · 42091 ms · 2026-05-20T15:22:05.640927+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages

  1. [1]

    Akgun, S., & Greenhow, C. (2022). Artificial intelligence in education: Addressing ethical challenges in K–12 settings. AI and Ethics, 2, 431–440. https://doi.org/10.1007/s43681-021- 00096-7 Brookhart, S. M. (2018). Appropriate criteria: Key to effective rubrics. Frontiers in Education, 3, Article

    work page doi:10.1007/s43681-021- 2022

  2. [2]

    https://doi.org/10.3389/feduc.2018.00022 Castro, V., Nascimento, A. K. de O., Zheldibayeva, R., Zapata, G. C., Searsmith, D., Cope, B., & Kalantzis, M. (2026). Implementing Rubric -Aligned Generative AI Feedback in K –12 Classrooms. Ubiquitous Learning: An International Journal. https://doi.org/10.18848/1835- 9795/cgp/a370 Cope, B., & Kalantzis, M. (2019)...

    work page doi:10.3389/feduc.2018.00022 2018

  3. [3]

    https://doi.org/10.1007/s44217-025-00919-3 Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (pp. 119–161). Macmillan. Fesler, L., Martinez Claeys, J. P ., Agnew, C., & Loeb, S. (2026). The evidence base on AI in K–12: A 2026 review. AI Hub for Education, SCALE Initiative, Stanford ...

    work page doi:10.1007/s44217-025-00919-3 1986