Pith. sign in

REVIEW 4 major objections 4 minor 23 references

Auto-grader Feedback Utilization and Its Impacts: An Observational Study Across Five Community Colleges

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Students who check auto-grader feedback after a non-perfect submission are more likely to improve on their next attempt, the paper reports.

desk verdict A genuinely useful observational study with a new per-submission feedback log, but the causal framing outruns the evidence and the key Fisher test ignores student-level clustering. read the letter →

arxiv 2507.14235 v1 pith:LRAYKEND submitted 2025-07-17 cs.CY

Adam Zhang , Heather Burte , Jaromir Savelka , Christopher Bogart , Majd Sakr This is my paper · ORCID
classification cs.CY
keywords auto-graderfeedbackprogrammingeducationcommunitycollegelearningoutcomesobservationalstudysubmissionbehaviorPythoncourseutilization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether auto-grader feedback in an introductory programming course actually helps students improve. Using logged navigation and submission data from 199 community-college students across five colleges, it compares submission pairs where a student checked the feedback page with pairs where they did not. The authors find that checking feedback between consecutive submissions is associated with a higher probability of a higher score next time (38.46% vs 33.77%, p = 0.0063). They also find that students who check feedback more often tend to receive higher final project scores. The authors interpret this as evidence that auto-grader feedback has a positive effect, while cautioning that the data cannot prove students actually read the feedback or that the relationship is causal.

What carries the argument

The key mechanism is the platform's logging of student navigation to submission-specific feedback webpages. Because feedback is available nowhere else, a logged visit to the feedback page is treated as the student having checked the feedback. The analysis then focuses on consecutive submissions by the same student to the same task where the first submission scored below maximum and the student later tried again; comparing the next-score outcome between pairs with and without a feedback check isolates the association between feedback use and improvement.

What would settle it

Re-analyze the same logged data while statistically controlling for the number of submissions and estimated project hours; if the 4.69-percentage-point improvement gap shrinks to zero once engagement is held constant, the claim that feedback itself causes the improvement would be falsified. A randomized experiment that hides feedback from a random subset of submissions would settle it directly.

Watch

Extended reading notes

Core claim

The central claim is that checking auto-grader feedback after a submission that did not earn full marks increases the likelihood that the student's next submission to the same task will score higher. Across all such non-maximal, non-terminal submission pairs, the probability of improvement is 38.46% when feedback was checked versus 33.77% when it was not, a difference the authors report as significant by Fisher's exact test (p = 0.0063). The paper also reports a moderate positive correlation (r = 0.27, p < 0.001) between how often students check feedback and their final project score. The authors treat these results as evidence that auto-grader feedback supports learning outcomes, while noting that the effect could be confounded by time spent or overall engagement.

Load-bearing premise

The comparison assumes that students who check feedback and students who do not are otherwise similar in effort, ability, and persistence; the paper's own correlation table shows that time spent on a project correlates with scores (r = 0.34) more strongly than feedback checks do (r = 0.27), so the apparent benefit could be driven by engagement rather than the feedback itself.

Editorial extensions

If this is right

  • If instructors encourage students to check auto-grader feedback after every non-perfect submission, average resubmission scores should improve.
  • Feedback-checking frequency could serve as an early-warning indicator for students at risk of low project scores.
  • Course designers should keep feedback immediately accessible and submission-specific, since the measured benefit depends on students visiting those pages.
  • The observed improvement gap of 4.69 percentage points gives a concrete effect size for decisions about investing in auto-grader feedback.
  • Because the platform logs each check, feedback utilization can be measured at scale in future studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The association may partly reflect that checking feedback is a proxy for time on task or persistence; the paper's own correlation matrix leaves this open.
  • Editorial inference: A randomized A/B test in which feedback pages are withheld for a randomly chosen subset of submissions would directly test whether the improvement gap is causal.
  • Editorial inference: The 'checked' definition captures a page visit, not comprehension; logging dwell time or scroll depth on the feedback page could sharpen the measure.
  • Editorial inference: Because the course required successful compilation before submission, these results may not generalize to auto-graders that accept and comment on code that does not compile.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript analyzes logged interactions from 199 students in an introductory Python course across five U.S. community colleges, using the Sail() platform. It addresses two research questions: whether students consistently check auto-grader feedback, and whether feedback checking is associated with better scores. RQ1 reports a near one-to-one relationship between submissions and feedback checks, with 28% of feedback pages never visited. RQ2 reports positive Pearson correlations between feedback checks and project scores, and a Fisher's Exact Test on pooled non-maximal, non-terminal submission pairs showing that a checked submission is followed by a higher score 38.46% of the time versus 33.77% when not checked (p = 0.0063), interpreted as evidence of positive impact. The authors acknowledge that they cannot know whether feedback was read and that time-on-task may explain the correlation, but the central claim is framed causally in the abstract, introduction, discussion, and conclusions.

Significance. If the association survives proper hierarchical analysis, the study would be a useful observational contribution to the auto-grader feedback literature, because it logs actual feedback-page navigation rather than relying on self-report or treatment-group assignment. The multi-college community college sample and the comparison with Gabbay and Cohen (2022) add practical relevance. However, the central inferential step treats repeated submissions by the same student as independent, and the manuscript's own correlation matrix provides direct evidence that engagement or time-on-task is a plausible alternative explanation. The strength of the paper is its rich behavioral dataset; the weakness is that the analysis does not yet match the causal language used to present it.

major comments (4)
  1. [Section 5.2, Fisher's Exact Test] The pooled per-submission analysis treats every non-maximal, non-terminal submission pair as an independent observation, but the data are hierarchical: students contribute an average of 66 submissions (σ = 72), so a small number of high-activity students can dominate the 38.46% versus 33.77% difference. Unmeasured student-level traits such as ability, persistence, and effort plausibly influence both feedback checking and score improvement, making the reported p = 0.0063 insufficient to support the stated causal claim. Please re-analyze with a mixed-effects model that includes a student random intercept and fixed effects for project/task, or use a student-level permutation test that respects the nesting.
  2. [Section 5.2, Figure 7] The manuscript's own correlation matrix shows that nFeedbackChecks correlates 0.961 with nSubmissions and that ProjectHours correlates with ProjectScore (r = 0.344) more strongly than nFeedbackChecks does (r = 0.273). The submission-pair analysis adjusts for neither time-on-task nor overall submission activity, so the observed association may reflect general engagement rather than feedback content. The text acknowledges this possibility for the correlation analysis but does not apply the same caution to the Fisher test; please add appropriate controls or explicitly restrict the claim to association.
  3. [Section 5.2, definition of consecutive submissions] Two submissions are considered consecutive even if the student attempted another task or another project between them. Because the outcome is defined as a higher score on the later submission, this mixes comparisons across tasks with different point scales and difficulty levels. Please report an analysis restricted to consecutive submissions to the same task, or justify why cross-task score comparisons are valid, and include a confidence interval or effect-size estimate for the 4.69 percentage-point difference.
  4. [Abstract, Sections 6 and 7] The abstract and conclusions use causal language such as 'impact', 'positive impact', and 'evidence on auto-grader feedback's effectiveness', while the design is observational and the authors explicitly acknowledge that reading behavior and effort are unmeasured. Please revise these passages to state the results in associational terms, reserving causal language for designs that can support it.
minor comments (4)
  1. [Section 8, Future Work] The final sentence contains the typo 'well-prepare'; it should read 'well-prepared'.
  2. [Section 6, Discussion] The phrase 'we could not determine if past a certain threshold students who check feedback more often per submission end up spending less time' is unclear; please specify what threshold is meant.
  3. [Figure 7] The correlation matrix would be easier to interpret if scatter plots with confidence bands were included in the supplementary material, especially for the nSubmissions-nFeedbackChecks pair.
  4. [Section 4, Dataset] Please clarify whether the 66-submission average includes students who attempted only one project, since the earlier Figure 6 shows substantial attrition in later projects; a per-project breakdown would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the feedback-impact result is an observed association computed from independently logged navigation and score data, with no fitted parameter or self-citation chain bearing the load.

full rationale

The paper's central claims—the correlation between nFeedbackChecks and ProjectScore (r = 0.273, p < 0.001) and the conditional-probability contrast P(Higher|Check)=38.46% vs. P(Higher|NotCheck)=33.77% with p=0.0063 on a Fisher's Exact Test (Section 5.2)—are computed directly from two operationally distinct logged variables: navigation to the submission-specific feedback webpage (Section 3.2) and the score of the subsequent submission to the same task. Neither quantity is defined in terms of the other, no parameter is fitted to data and then renamed as a prediction, and the 4.69 percentage-point difference is an observed sample statistic, not the output of a model whose inputs include the outcome. The self-citations (Bogart et al. 2024; Savelka et al. 2023; An et al. 2021; Nguyen et al. 2024) supply course context, platform details, and prior working-habit analyses, but they are not invoked to establish the feedback effect; the one external benchmark, Gabbay and Cohen (2022), is used after the fact as corroboration ('our result resembles that of a previous study'), so it is not an input that forces the result. The paper even flags the most plausible non-feedback explanation itself, noting that ProjectHours correlates with ProjectScore (r = 0.344) more strongly than nFeedbackChecks does (r = 0.273) and that 'the impact may instead come from spending more time on the project' (Section 5.2). The near-tautological correlation between nSubmissions and nFeedbackChecks (r = 0.961) is acknowledged as a behavioral regularity, not presented as causal evidence. The residual concerns—student-level clustering of submission-pairs and unmeasured effort confounding the Fisher test—are threats to statistical validity and causal interpretation, not instances of the result being equivalent to its inputs by construction. No circular step can be exhibited by quotation, so the honest finding is no circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the analysis reports correlations and a Fisher exact test. The main assumptions are measurement ones: page navigation means feedback consumption, distinct clock hours approximate time on task, and the non-maximal, non-terminal comparison is unconfounded. No invented entities are introduced.

assumptions (3)
  • domain assumption Navigating to the feedback webpage indicates feedback checking.
    Section 4 states that a student has checked the feedback if navigation to the feedback webpage is logged. The paper acknowledges this equates page view with consumption and that it does not know if the student read the feedback.
  • domain assumption Estimating time spent via distinct clock hours of navigation or submissions approximates total time on project.
    Section 4 defines ProjectHours as distinct clock hours in which students navigate project web pages or make submissions. This proxy may miss offline coding time and is a key confound in the correlation analysis.
  • domain assumption Non-maximal, non-terminal submissions are comparable except for feedback checking, with no unmeasured confounding.
    Section 5.2 computes conditional probabilities without adjusting for student-level traits such as prior ability, motivation, or time on task. The validity of the causal interpretation depends on this exchangeability assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Auto-grader Feedback Utilization and Its Impacts: An Observational Study Across Five Community Colleges." pith.science (2026). https://pith.science/paper/LRAYKEND

@misc{pith2026250714235,
  author       = {Pith},
  title        = {Pith review of: Auto-grader Feedback Utilization and Its Impacts: An Observational Study Across Five Community Colleges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LRAYKEND}},
  note         = {Machine review of arXiv:2507.14235}
}
read the original abstract

Automated grading systems, or auto-graders, have become ubiquitous in programming education, and the way they generate feedback has become increasingly automated as well. However, there is insufficient evidence regarding auto-grader feedback's effectiveness in improving student learning outcomes, in a way that differentiates students who utilized the feedback and students who did not. In this study, we fill this critical gap. Specifically, we analyze students' interactions with auto-graders in an introductory Python programming course, offered at five community colleges in the United States. Our results show that students checking the feedback more frequently tend to get higher scores from their programming assignments overall. Our results also show that a submission that follows a student checking the feedback tends to receive a higher score than a submission that follows a student ignoring the feedback. Our results provide evidence on auto-grader feedback's effectiveness, encourage their increased utilization, and call for future work to continue their evaluation in this age of automation

Figures

Figures reproduced from arXiv: 2507.14235 by the authors.

Figure 1
Figure 1. Example of Handout Code 3.2 The Auto-grader Feedback Auto-grader feedback is generated within seconds for each submission. The feedback is hosted on a dedi￾cated webpage accessible via a hyperlink on the sub￾missions table for the task on the course website. The feedback is not provided anywhere else. The feed￾back is available only to the student who made the submission. The feedback is exclusively textual; an exam… view at source ↗
Figure 2
Figure 2. American States Represented by Participants; [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. NFeedbackChecks v. NSubmissions; Grouped by Project and by College However, not all feedback is checked. Recall that feedback is generated by the auto-grader for each sub￾mission, on a webpage uniquely identified by the sub￾mission, and accessible only to the student who made the submission. 28% of the feedback pages were never checked. 56% were checked once. 16% were checked more than once. The number of feedback c… view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Example of Auto-grader Feedback for A Student’s Submission [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 5
Figure 5. Figure 5: Number of Submissions and Feedback Checks [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Correlation Matrix with our earlier observation that students check feedback almost as often as they submit regardless of project and college. nSubmissions and ProjectHours also have a strong and positive correlation (cor = 0.474, p < 0.001). We hypothesize that this i…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    An, M., Zhang, H., Savelka, J., Zhu, S., Bogart, C., and Sakr, M. (2021). Are working habits different between well-performing and at-risk students in online project-based courses? In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 , pages 324--330

    work page 2021

  3. [3]

    and Caspersen, M

    Bennedsen, J. and Caspersen, M. E. (2007). Failure rates in introductory programming. SIGCSE Bull. , 39(2):32–36

    work page 2007

  4. [4]

    Bogart, C., An, M., Keylor, E., Singh, P., Savelka, J., and Sakr, M. (2024). What factors influence persistence in project-based programming courses at community colleges? In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 , pages 116--122

    work page 2024

  5. [5]

    and Cohen, A

    Gabbay, H. and Cohen, A. (2022). Exploring the connections between the use of an automated feedback system and learning behavior in a mooc for programming. In Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption: 17th European Conference on Technology Enhanced Learning, EC-TEL 2022, Toulouse, France, September 12–16, 2022, Pro...

    work page 2022

  6. [6]

    C., Zhang, H., Sakr, M., An, H., and Dashti, C

    Goldstein, S. C., Zhang, H., Sakr, M., An, H., and Dashti, C. (2019). Understanding how work habits influence student performance. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education , pages 154--160

    work page 2019

  7. [7]

    Keuning, H., Jeuring, J., and Heeren, B. (2018). A systematic literature review of automated feedback generation for programming exercises. ACM Trans. Comput. Educ. , 19(1)

    work page 2018

  8. [8]

    Kokotsaki, D., Menzies, V., and Wiggins, A. (2016). Project-based learning: A review of the literature. Improving Schools , 19

    work page 2016

Show all 23 references
  1. [9]

    Kumar, A. N. (2005). Generation of problems, answers, grade, and feedback---case study of a fully automated tutor. J. Educ. Resour. Comput. , 5(3):3–es

    2005

  2. [10]

    M., Al Hoque, I., Lee, N

    Kurniawan, O., Poskitt, C. M., Al Hoque, I., Lee, N. T. S., Jégourel, C., and Sockalingam, N. (2023). How helpful do novice programmers find the feedback of an automated repair tool? In 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TA...

    2023

  3. [11]

    Messer, M., Brown, N. C. C., K\" o lling, M., and Shi, M. (2024). Automated grading and feedback tools for programming education: A systematic review. ACM Trans. Comput. Educ. , 24(1)

    2024

  4. [12]

    Mitra, J. (2023). Studying the impact of auto-graders giving immediate feedback in programming assignments. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 , SIGCSE 2023, page 388–394, New York, NY, USA. Association for Computing Machinery

    2023

  5. [13]

    Narciss, S. (2008). Feedback strategies for interactive learning tasks. In Spector, J., Merrill, M., van Merrienboer, J., and Driscoll, M., editors, Handbook of Research on Educational Communications and Technology , chapter 11, pages 125--144. Lawrence Erlbaum Associates, Mah...

    2008

  6. [14]

    Undergraduate degree earners: Academic year 2022-23

    National Student Clearninghouse (2024). Undergraduate degree earners: Academic year 2022-23. Technical report, National Student Clearinghouse Research Center

    2024

  7. [15]

    A., Bogart, C., S avelka, J., Zhang, A., and Sakr, M

    Nguyen, H. A., Bogart, C., S avelka, J., Zhang, A., and Sakr, M. (2024). Examining the trade-offs between simplified and realistic coding environments in an introductory python programming class. In European Conference on Technology Enhanced Learning , pages 315--329. Springer

    2024

  8. [16]

    Pettit, R., Homer, J., Holcomb, K., Simone, N., and Mengel, S. (2015). Are automated assessment tools helpful in programming courses? ASEE Annual Conference and Exposition, Conference Proceedings , 122

    2015

  9. [17]

    and Prather, J

    Pettit, R. and Prather, J. (2017). Automated assessment tools: too many cooks, not enough collaboration. J. Comput. Sci. Coll. , 32(4):113–121

    2017

  10. [18]

    A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al

    Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023). The robots are here: Navigating the generative ai revolution in computing education. In Proceedings of the 2023 Working Group Repo...

    2023

  11. [19]

    G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al

    Prather, J., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2024). Beyond the hype: A comprehensive review of current trends in generative ai research, teaching practices, and tools. arXiv preprint arXiv:...

    2024 arXiv

  12. [20]

    Savelka, J., Agarwal, A., An, M., Bogart, C., and Sakr, M. (2023). Thrilled by your progress! large language models (gpt-4) no longer struggle to pass assessments in higher education programming courses. In Proceedings of the 2023 ACM Conference on International Computing Educ...

    2023

  13. [21]

    Savelka, J., Kultur, C., Agarwal, A., Bogart, C., Burte, H., Zhang, A., and Sakr, M. (2025). Ai technicians: Developing rapid occupational training methods for a competitive ai workforce. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1

    2025

  14. [22]

    Sim, T. Y. and Lau, S. L. (2018). Online tools to support novice programming: A systematic review. In 2018 IEEE Conference on e-Learning, e-Management and e-Services (IC3e) , pages 91--96

    2018

  15. [23]

    Wang, T., Su, X., Ma, P., Wang, Y., and Wang, K. (2011). Ability-training-oriented automated assessment in introductory programming course. Comput. Educ. , 56(1):220–226

    2011

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.