From Coders to Critics: Empowering Students through Peer Assessment in the Age of AI Copilots

Santiago Berrezueta-Guzman; Stefan Wagner; Stephan Krusche

REVIEW 4 major objections 5 minor 19 references

From Coders to Critics: Empowering Students through Peer Assessment in the Age of AI Copilots

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured, anonymized peer assessment with a detailed rubric can approximate instructor grading in an introductory programming course, with moderate correlation and positive student reception.

desk verdict Solid single-course evidence for moderate peer-instructor agreement, but the 'reliably' in the conclusion outruns the statistics. read the letter →

arxiv 2505.22093 v1 pith:ZBAWZTJW submitted 2025-05-28 cs.CY cs.AIcs.HC

Santiago Berrezueta-Guzman , Stephan Krusche , Stefan Wagner This is my paper

classification cs.CYcs.AIcs.HC

keywords peerassessmentprogrammingeducationrubric-basedevaluationAIcodingassistantsacademicintegritycodereviewstudentengagement2Dgameproject

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that structured, anonymized peer assessment can stand in for instructor grading in a large introductory programming course. In a first-semester class where the final project, a 2D game, carried 60 percent of the grade, 47 teams of three students graded two peer projects using a detailed rubric, and their scores were compared with instructor grades. The two review rounds showed moderate correlation (r=0.55 and r=0.50) and mean absolute errors of 9.18 and 10.68 points on a 100-point scale. Surveys added that all teams considered their own evaluations fair, 83 percent enjoyed the evaluator role, and most teams could recognize the relative quality of their own projects compared with the reviewed games. If this holds, peer review offers a scalable assessment path that keeps students engaged as critics while AI copilots make submission-level originality harder to verify.

What carries the argument

The load-bearing mechanism is the rubric: a point-based evaluation scheme that decomposes a game project into functional, design, and code-quality criteria totaling 100 points plus bonus. It standardizes what students look for, lets each team justify its scores, and provides the common scale on which peer and instructor grades are compared. Around it are an anonymizing assignment algorithm that ensures no team reviews itself or repeats a project, and a requirement that teams discuss each criterion collaboratively before submitting.

What would settle it

Have two independent instructor teams grade the same set of projects with the same rubric. If instructor-instructor agreement is no better than the peer-instructor agreement reported here (r approximately 0.50 to 0.55), then the claim that peer assessment is less accurate than expert grading loses its footing; if instructor-instructor agreement is much higher, the peer results should be rechecked against that consensus standard.

Watch

Extended reading notes

Core claim

The paper's central claim is that students in an introductory programming course can reliably evaluate their peers' final projects when given a detailed rubric and anonymity. On a 100-point rubric covering gameplay, main character mechanics, GUI, sound, graphics, code structure, and documentation, Peer Review 1 correlated with instructor grades at r=0.55 with mean absolute error (MAE) 9.18 and root mean square error (RMSE) 14.87; Peer Review 2 correlated at r=0.50 with MAE 10.68 and RMSE 16.36. The authors read these as moderate agreement with visible outliers, not exact equivalence, and note that all 47 teams believed their evaluations were fair while 83 percent enjoyed acting as evaluators. They conclude that rubric-based, anonymized peer assessment approximates instructor evaluation well enough to be pedagogically useful, and that it builds evaluative judgment and self-awareness, competencies they argue are increasingly important when AI assistants generate code.

Load-bearing premise

The paper's argument rests on treating instructor grades as the true quality of each project, yet it never measures how consistent instructors are with one another, so the reported correlations and errors could partly reflect noise in the reference standard rather than pure peer-grader accuracy.

Editorial extensions

If this is right

In courses where AI-assisted submissions undermine homework honesty, a rubric-based anonymous peer review can serve as a scalable complement to instructor grading rather than a last resort.
Students gain transferable critical skills: most teams accurately judged the relative quality of their own project, supporting the idea that assessment trains evaluative judgment.
Because the second peer review was less accurate than the first, future deployments should include reviewer training, calibration sessions, or score weighting to tighten agreement.
Peer assessment can double as an engagement tool: 83 percent of teams enjoyed the reviewer role, which suggests grading duties can be structured as learning experiences rather than chores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: because no inter-rater reliability among instructors is reported, the true quality signal is unknown; a fairer test would compare peer grades against a consensus score from multiple instructors.
Beyond the paper: a testable extension would run two or more calibration-trained review rounds and check whether the peer-instructor correlation rises and the MAE falls, which would separate reviewer skill from assignment randomness.
Beyond the paper: the 68 percent preference for taking the highest peer score suggests grade-aggregation policy is a fairness lever; a system that drops outlier reviews or weights by past reviewer accuracy may address that concern more fairly than averaging.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a solid, honest single-course case study, not a breakthrough. The genuinely useful bit is the concrete dataset: 47 teams, two anonymous rubric-based peer reviews each, compared to instructor grades, plus a perception survey. That is a clean design for a teaching report, and the abstract is appropriately careful in using 'moderate accuracy.'

What it does well: the rubric is detailed and reproduced in the paper, the assignment procedure is transparent, and the survey adds a human dimension that pure correlation numbers miss. The descriptive statistics (r = 0.55 and 0.50, MAE 9.18 and 10.68) are simple but match the research question. The authors do not oversell in the results section—they note variability and outliers. The related work is adequate, and the self-citations are for course context, not a red flag.

The soft spots land mostly in the conclusion. Section VII says students 'can reliably evaluate each other's work using detailed rubrics,' but r = 0.55 is moderate, and an MAE of 9–11 points on a 100-point scale is roughly a letter grade. That is 'reasonable agreement,' not 'reliably' in any psychometric sense. There are no confidence intervals or significance tests around the correlations, no inter-rater reliability statistic (ICC or weighted kappa), and the clustered structure (each team reviews twice; each project is reviewed twice) is ignored. Instructor grades are treated as ground truth, but no instructor-instructor reliability is reported; we cannot tell whether the peer-instructor correlation is an under- or over-estimate. Also, the 100% self-reported fairness almost certainly includes social desirability bias; it should be framed as perceived fairness.

None of this sinks the paper. The central claim—peer assessment can approximate instructor grading with moderate accuracy—holds up as a course-specific finding. The overclaim in the conclusion is fixable by softening the language or adding the missing statistics.

Who should read it: instructors considering peer assessment in programming courses, and computing-education researchers looking for a recent data point in the AI-copilot context. It deserves a serious referee, not a desk reject. With a revision that tightens the conclusion and adds the obvious robustness checks, it would be a reasonable contribution to a teaching-and-learning venue.

I'd engage with it. Ask the authors to either replace 'reliably' with a more measured claim or provide ICC, confidence intervals, and instructor-instructor reliability. That is a normal revision, not a fundamental flaw.

Referee Report

4 major / 5 minor

Summary. The paper reports an empirical study of a structured, anonymized, rubric-based peer assessment process in an introductory programming course. Student teams (N=47) evaluated two peer projects each using a 100-point rubric; peer scores were compared with instructor scores on the same rubric via Pearson correlation, MAE, and RMSE. A post-hoc survey captured student perceptions of fairness, strictness, grade-aggregation preferences, and engagement. The authors report moderate correlations (r=0.55 and 0.50) and MAE values around 9–11 points, and they interpret these as evidence that students can reliably evaluate peer work, while also highlighting perceived pedagogical benefits. The paper's central claim is that structured peer review can approximate instructor evaluation with moderate accuracy and foster evaluative thinking.

Significance. If the central claim were fully supported, the study would be a useful contribution to computing-education research on peer assessment, particularly in the context of AI-assisted coding. The strengths are the real-world, large-course setting, the use of an external instructor benchmark, the combination of quantitative and qualitative data, and the concrete rubric structure. The paper also makes a reasonable pedagogical argument that peer review can promote critical evaluation skills. However, the evidence as presented supports only 'moderate agreement with instructor grades in one course,' not the broader statement that students can 'reliably' evaluate each other's work. The missing reliability statistics, absent instructor-instructor benchmark, and lack of inferential confidence intervals are load-bearing gaps that must be addressed before the conclusions can be accepted.

major comments (4)

[Section VII and Section V.A, Table IV] The conclusion that 'students can reliably evaluate each other's work using detailed rubrics' overstates the evidence. Pearson correlations of r=0.55 and 0.50 are moderate correlations, not reliability coefficients; reliability is typically assessed with intraclass correlation, weighted kappa, or a pre-specified agreement threshold. With MAE of 9.18 and 10.68 on a 100-point scale, a substantial fraction of peer scores differ from instructor scores by more than 10 points. The paper should either temper the conclusion to 'moderate agreement with instructor grades' or add additional agreement metrics (e.g., proportion of scores within 5 or 10 points, Bland-Altman limits) that would support the stronger claim.
[Section IV.B] The analysis treats instructor grades as the ground truth for project quality, but no inter-rater reliability among instructors or calibration of instructor grading is reported. If instructor grading is itself noisy or biased, the reported correlation and error metrics measure agreement with a fallible reference, not accuracy. The manuscript should report inter-rater reliability for instructors (e.g., double-scoring a subset of projects) or explicitly discuss this limitation as a threat to the validity of the reliability claim.
[Section V.A] The statistical evidence lacks confidence intervals, significance tests, and a treatment of the data's clustered structure. The correlations, MAE, and RMSE are reported as point estimates without uncertainty, and the data have a hierarchical structure (each team performs two reviews; each project receives two reviews) that is not accounted for in any inferential model. Additionally, Figure 1 shows visible outliers, and Pearson correlation is sensitive to outliers and range restriction. The manuscript should provide confidence intervals for the reported metrics, or a robust/permutation-based analysis, and should justify the effective sample size for inference.
[Section V.B and Section VI, Finding 3] The claim that 'students took the responsibility seriously' is based on a 100% self-report of fair evaluations, which is likely influenced by social desirability and is not independently validated. The survey was administered after the peer-review task, and respondents may have rationalized their own grading behavior. The paper should frame these results as perceived fairness, not as evidence of actual fairness or responsibility, and should discuss this as a methodological limitation.

minor comments (5)

[Section V.A] There is a numerical inconsistency: the text reports an RMSE of 16.37 for Peer Review 2, while Table IV lists 16.36. Please correct this mismatch.
[Abstract and Section VII] The abstract appropriately says 'moderate accuracy,' but the conclusions use 'reliably.' These phrasings should be aligned; the stronger claim is not supported by the reported metrics.
[Section IV.A] The term 'provocative questions' is informal; consider describing the purpose more neutrally, e.g., 'comparative reflection questions.'
[Section III, Tables II and III] The rubric includes bonus points that can raise the maximum above 100; the paper should clarify whether instructor and peer scores were both converted to a common scale (e.g., percentage of total possible points) before computing MAE/RMSE, since this affects the interpretation of the error magnitudes.
[Section V.B] The survey response rate and administration details are not fully specified (e.g., whether all 47 teams responded to every question, or whether some items were omitted). Please provide the number of respondents per item.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the peer-assessment study compares student scores against an external instructor benchmark; nothing in the reported statistics is fitted to or constructed from the outcome it claims to measure.

full rationale

The paper's central claim is that rubric-based, anonymized peer review approximates instructor evaluation. This is an empirical agreement analysis, not a derivation. Peer-assigned scores and instructor-assigned scores are independent measurements in the sense that one group's grades are not computed from the other's; the instructor grades are described as a separate evaluation ('Each project was also evaluated by the course instructors using the same rubric', Section IV.B). The reported Pearson r, MAE, and RMSE are descriptive statistics over these two sets of scores. No parameter is fitted to a subset of the data and then 'predicted' on a closely related quantity, and no equality is imposed by construction. The shared rubric is a standardization choice that makes the comparison meaningful; it does not force agreement, as evidenced by the observed MAE of 9-11 and RMSE of 15-16. The paper does cite prior work by its own authors (e.g., [4], [17], [18], [19]), but these citations provide course context, an AI tutor system, and teamwork background; none of them is the load-bearing justification for the empirical finding. There is no imported uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result. The reviewer-flagged concern that instructor grades are treated as ground truth without inter-rater reliability is a measurement-validity threat, not a circularity: it questions what the comparison means, but the comparison itself is not equivalent to its inputs. The conclusion is appropriately hedged as 'moderate accuracy' and 'can approximate' in the abstract and Section VII, so no structural circularity is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central analysis depends on three untested assumptions: the validity of instructor grades as ground truth, the truthfulness of self-reports, and the consistency of rubric use. These are standard assumptions in educational research but are not independently verified in this study.

free parameters (1)

Rubric category weights
The point values for rubric categories (e.g., Game World 25, Main Character 15) were chosen by the instructors and set the scale on which peer-instructor agreement is measured. They are hand-selected design choices, not fitted to data.

assumptions (3)

domain assumption Instructor grades are a valid and reliable ground truth for project quality.
Section IV.B states each project was evaluated by course instructors using the same rubric; the entire accuracy analysis (correlation, MAE, RMSE) treats these grades as the reference standard. No inter-rater reliability among instructors is reported.
domain assumption Self-reported survey responses accurately reflect students' actual attitudes and behavior.
Section IV.C and V.B rely on Likert and open-ended self-reports for fairness, strictness, and engagement. Social desirability bias and the absence of behavioral validation threaten this assumption.
domain assumption The rubric is a valid measure of project quality and is used consistently across teams.
The rubric (Tables II and III) is assumed to capture the relevant dimensions of quality. The paper does not test its construct validity or inter-rater reliability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Coders to Critics: Empowering Students through Peer Assessment in the Age of AI Copilots." pith.science (2026). https://pith.science/paper/ZBAWZTJW

@misc{pith2026250522093,
  author       = {Pith},
  title        = {Pith review of: From Coders to Critics: Empowering Students through Peer Assessment in the Age of AI Copilots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBAWZTJW}},
  note         = {Machine review of arXiv:2505.22093}
}

read the original abstract

The rapid adoption of AI powered coding assistants like ChatGPT and other coding copilots is transforming programming education, raising questions about assessment practices, academic integrity, and skill development. As educators seek alternatives to traditional grading methods susceptible to AI enabled plagiarism, structured peer assessment could be a promising strategy. This paper presents an empirical study of a rubric based, anonymized peer review process implemented in a large introductory programming course. Students evaluated each other's final projects (2D game), and their assessments were compared to instructor grades using correlation, mean absolute error, and root mean square error (RMSE). Additionally, reflective surveys from 47 teams captured student perceptions of fairness, grading behavior, and preferences regarding grade aggregation. Results show that peer review can approximate instructor evaluation with moderate accuracy and foster student engagement, evaluative thinking, and interest in providing good feedback to their peers. We discuss these findings for designing scalable, trustworthy peer assessment systems to face the age of AI assisted coding.

Figures

Figures reproduced from arXiv: 2505.22093 by the authors.

**Figure 2.** Distribution of Peer Review 1, Peer Review 2, and Instructor Grades [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

[1]

A systematic literature review of the opportunities and advantages for aigc (openai chatgpt, copilot, codex) in programming course,

C. I. Chang, W. C. Choi, and I. C. Choi, “A systematic literature review of the opportunities and advantages for aigc (openai chatgpt, copilot, codex) in programming course,” in Proceedings of the 2024 7th International Conference on Big Data and Education , pp. 29–35, 2024

work page 2024
[2]

From” ban it till we understand it

S. Lau and P. Guo, “From” ban it till we understand it” to” resistance is futile”: How university programming instructors plan to adapt as more students use ai code generation and explanation tools such as chatgpt and github copilot,” in Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1 , pp. 106–121, 2023

work page 2023
[3]

Using github copilot to solve simple programming problems,

M. Wermelinger, “Using github copilot to solve simple programming problems,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, pp. 172–178, 2023

work page 2023
[4]

Code collaborate: Dissecting team dynamics in first-semester programming students,

S. Berrezueta-Guzman, P. Bassner, S. Wagner, and S. Krusche, “Code collaborate: Dissecting team dynamics in first-semester programming students,” in 2024 21st International Conference on Information Tech- nology Based Higher Education and Training (ITHET) , pp. 1–10, IEEE, 2024

work page 2024
[5]

Ai generated code plagiarism detection in computer science courses: A literature mapping,

A. Simmons, M. Holanda, C. Chamon, and D. Da Silva, “Ai generated code plagiarism detection in computer science courses: A literature mapping,” in 2024 IEEE Frontiers in Education Conference (FIE) , pp. 1–7, IEEE, 2024

work page 2024
[6]

Rethinking plagiarism in the era of generative ai,

J. Hutson, “Rethinking plagiarism in the era of generative ai,” Journal of Intelligent Communication , vol. 3, no. 2, pp. 20–31, 2024

work page 2024
[7]

Use of chatgpt in academia: Academic integrity hangs in the balance,

S. A. Bin-Nashwan, M. Sadallah, and M. Bouteraa, “Use of chatgpt in academia: Academic integrity hangs in the balance,” Technology in Society, vol. 75, p. 102370, 2023

work page 2023
[8]

Ai chatbots in programming education: guiding success or encouraging plagiarism,

G. Akc ¸apınar and E. Sidan, “Ai chatbots in programming education: guiding success or encouraging plagiarism,” Discover Artificial Intelli- gence, vol. 4, no. 1, p. 87, 2024

work page 2024

Show all 19 references

[9]

Chatting and cheating: Ensuring academic integrity in the era of chatgpt,

D. R. Cotton, P. A. Cotton, and J. R. Shipway, “Chatting and cheating: Ensuring academic integrity in the era of chatgpt,” Innovations in education and teaching international, vol. 61, no. 2, pp. 228–239, 2024

2024
[10]

Addressing bias to improve reliability in peer review of programming coursework,

S. Bradley, “Addressing bias to improve reliability in peer review of programming coursework,” in Proceedings of the 19th Koli Calling International Conference on Computing Education Research , pp. 1–10, 2019

2019
[11]

Does peer code review change my mind on my sub- mission?,

S. Strickroth, “Does peer code review change my mind on my sub- mission?,” in Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V . 1, pp. 498–504, 2023

2023
[12]

Using peer code review to improve computational thinking in a blended learning environment: A randomized control trial,

X. Lin, Y . Ma, W. Ma, Y . Liu, and W. Tang, “Using peer code review to improve computational thinking in a blended learning environment: A randomized control trial,” Computer Applications in Engineering Education, vol. 29, no. 6, pp. 1825–1835, 2021

2021
[13]

Student perspectives of peer assessment in programming courses,

A. Alkhalifa and M. Devlin, “Student perspectives of peer assessment in programming courses,” in Proceedings of the 2021 Conference on United Kingdom & Ireland Computing Education Research , pp. 1–7, 2021

2021
[14]

Using peer code review to support pedagogy in an introductory computer programming course,

T. Brown, M. R. Narasareddygari, M. Singh, and G. Walia, “Using peer code review to support pedagogy in an introductory computer programming course,” in 2019 IEEE Frontiers in Education Conference (FIE), pp. 1–7, IEEE, 2019

2019
[15]

Gam- ification improves the quality of student peer code review,

T. D. Indriasari, P. Denny, D. Lottridge, and A. Luxton-Reilly, “Gam- ification improves the quality of student peer code review,” Computer Science Education, vol. 33, no. 3, pp. 458–482, 2023

2023
[16]

A review of peer code review in higher education,

T. D. Indriasari, A. Luxton-Reilly, and P. Denny, “A review of peer code review in higher education,”ACM Transactions on Computing Education (TOCE), vol. 20, no. 3, pp. 1–25, 2020

2020
[17]

Iris: An ai-driven virtual tutor for computer science education,

P. Bassner, E. Frankford, and S. Krusche, “Iris: An ai-driven virtual tutor for computer science education,” in Proceedings of the Conference on Innovation and Technology in Computer Science Education , ITiCSE 2024, p. 394–400, ACM, 2024

2024
[18]

Introduction to programming using interactive learning,

S. Krusche and J. Berrezueta-Guzman, “Introduction to programming using interactive learning,” in 2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T) , pp. 178– 182, IEEE, 2023

2023
[19]

Assessing teamwork dynamics in software development projects,

S. Berrezueta-Guzman, I. Parmacli, M. K. Habib, S. Krusche, and S. Wagner, “Assessing teamwork dynamics in software development projects,” arXiv preprint arXiv:2501.11965 , 2025

2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

A systematic literature review of the opportunities and advantages for aigc (openai chatgpt, copilot, codex) in programming course,

C. I. Chang, W. C. Choi, and I. C. Choi, “A systematic literature review of the opportunities and advantages for aigc (openai chatgpt, copilot, codex) in programming course,” in Proceedings of the 2024 7th International Conference on Big Data and Education , pp. 29–35, 2024

work page 2024

[2] [2]

From” ban it till we understand it

S. Lau and P. Guo, “From” ban it till we understand it” to” resistance is futile”: How university programming instructors plan to adapt as more students use ai code generation and explanation tools such as chatgpt and github copilot,” in Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1 , pp. 106–121, 2023

work page 2023

[3] [3]

Using github copilot to solve simple programming problems,

M. Wermelinger, “Using github copilot to solve simple programming problems,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, pp. 172–178, 2023

work page 2023

[4] [4]

Code collaborate: Dissecting team dynamics in first-semester programming students,

S. Berrezueta-Guzman, P. Bassner, S. Wagner, and S. Krusche, “Code collaborate: Dissecting team dynamics in first-semester programming students,” in 2024 21st International Conference on Information Tech- nology Based Higher Education and Training (ITHET) , pp. 1–10, IEEE, 2024

work page 2024

[5] [5]

Ai generated code plagiarism detection in computer science courses: A literature mapping,

A. Simmons, M. Holanda, C. Chamon, and D. Da Silva, “Ai generated code plagiarism detection in computer science courses: A literature mapping,” in 2024 IEEE Frontiers in Education Conference (FIE) , pp. 1–7, IEEE, 2024

work page 2024

[6] [6]

Rethinking plagiarism in the era of generative ai,

J. Hutson, “Rethinking plagiarism in the era of generative ai,” Journal of Intelligent Communication , vol. 3, no. 2, pp. 20–31, 2024

work page 2024

[7] [7]

Use of chatgpt in academia: Academic integrity hangs in the balance,

S. A. Bin-Nashwan, M. Sadallah, and M. Bouteraa, “Use of chatgpt in academia: Academic integrity hangs in the balance,” Technology in Society, vol. 75, p. 102370, 2023

work page 2023

[8] [8]

Ai chatbots in programming education: guiding success or encouraging plagiarism,

G. Akc ¸apınar and E. Sidan, “Ai chatbots in programming education: guiding success or encouraging plagiarism,” Discover Artificial Intelli- gence, vol. 4, no. 1, p. 87, 2024

work page 2024