{"id":"9554666d-138a-4fbb-a294-2ede827b6675","arxiv_id":"2507.14235","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Students who checked auto-grader feedback more often earned higher assignment and resubmission scores, but the association may be driven by time spent and overall engagement.","lead":"A study of 199 community college students found that those who opened auto-grader feedback pages more often tended to earn higher scores on Python programming assignments. As automated and AI-generated feedback spreads through coding courses, this work is an early attempt to measure whether students actually benefit from that feedback.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Fisher test's reported p=0.0063 may not be robust to student-level confounding and non-independence of submission-pairs; the central claim needs a student-level control.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: unmeasured student-level effort/persistence confounding the feedback-score association. My stress-test focuses the critique on the exact statistical procedure used for the headline claim (the Fisher's Exact Test on pooled submission-pairs), which is where the independence assumption is most clearly violated. This is not an external-consensus disagreement; it is a methodological correctness risk internal to the analysis. The paper's own correlation matrix provides concrete evidence of the confound. The recommended verdict remains CONDITIONAL: the association is likely real in the descriptive sense, but the causal 'impact' framing and the headline p-value require a student-level control. A logistic mixed-effects model with student and task random intercepts is a single, feasible check that would resolve the concern. I agree with the reader's assessment and propose the same verdict adjustment, with the concrete test as the specification for what would change the verdict.","tokens_in":7919,"tokens_out":1519,"duration_ms":16235,"concrete_test":"Re-analyze the consecutive-submission pairs with a logistic mixed-effects model: outcome = later score improved; fixed effects = checked feedback, task difficulty, submission number/order, and ProjectHours; random intercepts for student and task. If the coefficient for 'checked feedback' loses significance or shrinks below the reported 4.69 percentage-point difference, the Fisher test's p=0.0063 is not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Section 5.2, 'Students who check... p = 0.0063') rests on a Fisher's Exact Test applied to pooled submission-pairs. The dataset is hierarchical: each student contributes many non-maximal, non-terminal submissions (mean 66 submissions, sd 72 across five projects), so a student who both checks feedback and improves often can drive the aggregate association. Treating each submission-pair as independent inflates the effective sample size and can produce a spuriously significant p-value even if feedback has no causal effect, because unmeasured student-level traits such as effort, persistence, or ability confound both feedback-checking and score improvement. The paper's own correlation matrix (Figure 7) shows nFeedbackChecks correlates 0.961 with nSubmissions and ProjectHours correlates with ProjectScore (r = 0.34) more strongly than nFeedbackChecks does (r = 0.27), which is direct evidence that engagement/time-on-task is a plausible alternative driver. The authors acknowledge the confound for the correlation analysis but not for the per-submission Fisher test. Also, because consecutive submissions are defined across all tasks and projects, the 'improvement' outcome mixes different tasks and difficulty levels, and no confidence interval or effect-size estimate is reported for the 4.69 percentage-point difference. The claim would be sound only if the comparison were adjusted for student-level confounding, for example via a mixed-effects model with a student random intercept or a within-student comparison that controls for submission order and task difficulty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript analyzes logged interactions from 199 students in an introductory Python course across five U.S. community colleges, using the Sail() platform. It addresses two research questions: whether students consistently check auto-grader feedback, and whether feedback checking is associated with better scores. RQ1 reports a near one-to-one relationship between submissions and feedback checks, with 28% of feedback pages never visited. RQ2 reports positive Pearson correlations between feedback checks and project scores, and a Fisher's Exact Test on pooled non-maximal, non-terminal submission pairs showing that a checked submission is followed by a higher score 38.46% of the time versus 33.77% when not checked (p = 0.0063), interpreted as evidence of positive impact. The authors acknowledge that they cannot know whether feedback was read and that time-on-task may explain the correlation, but the central claim is framed causally in the abstract, introduction, discussion, and conclusions.","tokens_in":8151,"tokens_out":3071,"duration_ms":38878,"significance":"If the association survives proper hierarchical analysis, the study would be a useful observational contribution to the auto-grader feedback literature, because it logs actual feedback-page navigation rather than relying on self-report or treatment-group assignment. The multi-college community college sample and the comparison with Gabbay and Cohen (2022) add practical relevance. However, the central inferential step treats repeated submissions by the same student as independent, and the manuscript's own correlation matrix provides direct evidence that engagement or time-on-task is a plausible alternative explanation. The strength of the paper is its rich behavioral dataset; the weakness is that the analysis does not yet match the causal language used to present it.","major_comments":[{"comment":"The pooled per-submission analysis treats every non-maximal, non-terminal submission pair as an independent observation, but the data are hierarchical: students contribute an average of 66 submissions (σ = 72), so a small number of high-activity students can dominate the 38.46% versus 33.77% difference. Unmeasured student-level traits such as ability, persistence, and effort plausibly influence both feedback checking and score improvement, making the reported p = 0.0063 insufficient to support the stated causal claim. Please re-analyze with a mixed-effects model that includes a student random intercept and fixed effects for project/task, or use a student-level permutation test that respects the nesting.","section":"Section 5.2, Fisher's Exact Test"},{"comment":"The manuscript's own correlation matrix shows that nFeedbackChecks correlates 0.961 with nSubmissions and that ProjectHours correlates with ProjectScore (r = 0.344) more strongly than nFeedbackChecks does (r = 0.273). The submission-pair analysis adjusts for neither time-on-task nor overall submission activity, so the observed association may reflect general engagement rather than feedback content. The text acknowledges this possibility for the correlation analysis but does not apply the same caution to the Fisher test; please add appropriate controls or explicitly restrict the claim to association.","section":"Section 5.2, Figure 7"},{"comment":"Two submissions are considered consecutive even if the student attempted another task or another project between them. Because the outcome is defined as a higher score on the later submission, this mixes comparisons across tasks with different point scales and difficulty levels. Please report an analysis restricted to consecutive submissions to the same task, or justify why cross-task score comparisons are valid, and include a confidence interval or effect-size estimate for the 4.69 percentage-point difference.","section":"Section 5.2, definition of consecutive submissions"},{"comment":"The abstract and conclusions use causal language such as 'impact', 'positive impact', and 'evidence on auto-grader feedback's effectiveness', while the design is observational and the authors explicitly acknowledge that reading behavior and effort are unmeasured. Please revise these passages to state the results in associational terms, reserving causal language for designs that can support it.","section":"Abstract, Sections 6 and 7"}],"minor_comments":[{"comment":"The final sentence contains the typo 'well-prepare'; it should read 'well-prepared'.","section":"Section 8, Future Work"},{"comment":"The phrase 'we could not determine if past a certain threshold students who check feedback more often per submission end up spending less time' is unclear; please specify what threshold is meant.","section":"Section 6, Discussion"},{"comment":"The correlation matrix would be easier to interpret if scatter plots with confidence bands were included in the supplementary material, especially for the nSubmissions-nFeedbackChecks pair.","section":"Figure 7"},{"comment":"Please clarify whether the 66-submission average includes students who attempted only one project, since the earlier Figure 6 shows substantial attrition in later projects; a per-project breakdown would help.","section":"Section 4, Dataset"}],"recommendation":"major_revision","confidential_remarks":"The dataset and research questions are well matched to cs.CY, and the logging-based measurement is a genuine strength. The main barrier is inferential: the central p-value is computed on non-independent units, and the causal framing is not supported by the observational design. These are fixable with a student-level mixed-effects analysis and revised language, so I view this as a major revision rather than a rejection. I do not see concerns about citation practices or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is a solid observational study with an overstated causal frame. The genuinely new thing is the per-submission log of feedback-page views across five community college offerings, which lets the authors separate submissions where students checked feedback from those where they didn't. That is a real improvement over prior work that only knew feedback was available. The conditional-probability result (38.5% vs 33.8% improvement; p=0.0063 Fisher) is worth reporting as a descriptive pattern. The paper also honestly documents the limitation that they don't know if students read the feedback, and it cites the relevant literature without inflating. The soft spots are the usual ones for clickstream studies, but they land on the paper's central claim. The Fisher test treats each submission-pair as independent, but pairs are nested in students. A student who always checks feedback contributes many check pairs; a student who never checks contributes many no-check pairs. The test is therefore, in effect, comparing students who check against students who don't, without adjusting for student-level effort, persistence, or ability. The paper's own correlation matrix shows nFeedbackChecks correlated 0.961 with nSubmissions, and ProjectHours correlated with ProjectScore (r=0.34) slightly stronger than nFeedbackChecks (r=0.27). Those numbers point straight at engagement as the driver. The authors acknowledge the confound for the correlation analysis but not for the per-submission test. They also report no confidence interval or effect-size measure beyond the raw percentage-point gap, and the 'consecutive' definition allows other tasks between the two submissions, so difficulty of the intervening material is uncontrolled. I don't think this is a fatal flaw. The association is real and the measurement is novel. But the title and abstract say 'impacts' and 'effectiveness' where the analysis supports 'is associated with.' A revision that adds a student-level mixed model or within-student comparison, reports CIs, and softens the causal language would make the paper's claims match its evidence. Who's it for: computing-education researchers and instructors using auto-graders; also methodologically interesting for learning analytics because it shows exactly how clickstream confounding can bite. It deserves a serious referee and a conditional decision, not a desk reject. I'd bring it to reading group; the design is a good teaching case.","headline":"A genuinely useful observational study with a new per-submission feedback log, but the causal framing outruns the evidence and the key Fisher test ignores student-level clustering.","tokens_in":636,"tokens_out":1186,"would_cite":true,"duration_ms":44002,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Students who check auto-grader feedback after a non-perfect submission are more likely to improve on their next attempt, the paper reports.","keywords":["auto-grader feedback","programming education","community college","learning outcomes","observational study","submission behavior","Python course","feedback utilization"],"falsifier":"Re-analyze the same logged data while statistically controlling for the number of submissions and estimated project hours; if the 4.69-percentage-point improvement gap shrinks to zero once engagement is held constant, the claim that feedback itself causes the improvement would be falsified. A randomized experiment that hides feedback from a random subset of submissions would settle it directly.","tokens_in":7714,"feed_emoji":"📈","tokens_out":5064,"duration_ms":50003,"temperature":0.7,"pith_summary":"This paper asks whether auto-grader feedback in an introductory programming course actually helps students improve. Using logged navigation and submission data from 199 community-college students across five colleges, it compares submission pairs where a student checked the feedback page with pairs where they did not. The authors find that checking feedback between consecutive submissions is associated with a higher probability of a higher score next time (38.46% vs 33.77%, p = 0.0063). They also find that students who check feedback more often tend to receive higher final project scores. The authors interpret this as evidence that auto-grader feedback has a positive effect, while cautioning that the data cannot prove students actually read the feedback or that the relationship is causal.","feed_headline":"Checking auto-grader feedback lifts odds of a better next score","feed_subtitle":"Community-college students who viewed feedback between submissions improved 4.69 points more often than those who didn't.","key_machinery":"The key mechanism is the platform's logging of student navigation to submission-specific feedback webpages. Because feedback is available nowhere else, a logged visit to the feedback page is treated as the student having checked the feedback. The analysis then focuses on consecutive submissions by the same student to the same task where the first submission scored below maximum and the student later tried again; comparing the next-score outcome between pairs with and without a feedback check isolates the association between feedback use and improvement.","core_discovery":"The central claim is that checking auto-grader feedback after a submission that did not earn full marks increases the likelihood that the student's next submission to the same task will score higher. Across all such non-maximal, non-terminal submission pairs, the probability of improvement is 38.46% when feedback was checked versus 33.77% when it was not, a difference the authors report as significant by Fisher's exact test (p = 0.0063). The paper also reports a moderate positive correlation (r = 0.27, p < 0.001) between how often students check feedback and their final project score. The authors treat these results as evidence that auto-grader feedback supports learning outcomes, while noting that the effect could be confounded by time spent or overall engagement.","pith_inferences":["Editorial inference: The association may partly reflect that checking feedback is a proxy for time on task or persistence; the paper's own correlation matrix leaves this open.","Editorial inference: A randomized A/B test in which feedback pages are withheld for a randomly chosen subset of submissions would directly test whether the improvement gap is causal.","Editorial inference: The 'checked' definition captures a page visit, not comprehension; logging dwell time or scroll depth on the feedback page could sharpen the measure.","Editorial inference: Because the course required successful compilation before submission, these results may not generalize to auto-graders that accept and comment on code that does not compile."],"forward_implications":["If instructors encourage students to check auto-grader feedback after every non-perfect submission, average resubmission scores should improve.","Feedback-checking frequency could serve as an early-warning indicator for students at risk of low project scores.","Course designers should keep feedback immediately accessible and submission-specific, since the measured benefit depends on students visiting those pages.","The observed improvement gap of 4.69 percentage points gives a concrete effect size for decisions about investing in auto-grader feedback.","Because the platform logs each check, feedback utilization can be measured at scale in future studies."],"supporting_citations":[{"why":"Supplies the systematic review showing most feedback-generation tools lack evaluation against learning outcomes, motivating this study.","marker":"Keuning et al. (2018)"},{"why":"Documents that tools from 2017-2021 were mostly evaluated by surveys or human-grader comparison rather than learning outcomes, framing the gap this study addresses.","marker":"Messer et al. (2024)"},{"why":"Provides the comparable 36% resubmission-correction rate that this paper's 38.46% improvement result is measured against.","marker":"Gabbay and Cohen (2022)"},{"why":"Argues for assessing automated tools by grades and learning outcomes, which this study operationalizes with project scores.","marker":"Pettit et al. (2015)"},{"why":"Describes the course the data comes from, supporting the project-based design and platform details.","marker":"Savelka et al. (2023)"}],"fun_headline_variants":["Auto-grader feedback check lifts next submission odds","Feedback checking improves chances of a higher next score","Check feedback, boost next attempt: 38% vs 34%","Frequent feedback checks correlate with higher final scores","Auto-grader feedback use predicts better programming outcomes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that students who check feedback and students who do not are otherwise similar in effort, ability, and persistence; the paper's own correlation table shows that time spent on a project correlates with scores (r = 0.34) more strongly than feedback checks do (r = 0.27), so the apparent benefit could be driven by engagement rather than the feedback itself.","fun_headline_variants_meta":{"raw":{"variants":["Auto-grader feedback check lifts next submission odds","Feedback checking improves chances of a higher next score","Check feedback, boost next attempt: 38% vs 34%","Frequent feedback checks correlate with higher final scores","Auto-grader feedback use predicts better programming outcomes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2931,"prompt_tokens":873,"completion_tokens":2058,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1980}},"tokens_in":489,"tokens_out":2058,"duration_ms":16993,"temperature":1.0,"reasoning_tokens":1980,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:32:45.905880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-analyze the same logged data while statistically controlling for the number of submissions and estimated project hours; if the 4.69-percentage-point improvement gap shrinks to zero once engagement is held constant, the claim that feedback itself causes the improvement would be falsified. A randomized experiment that hides feedback from a random subset of submissions would settle it directly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the systematic review showing most feedback-generation tools lack evaluation against learning outcomes, motivating this study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that tools from 2017-2021 were mostly evaluated by surveys or human-grader comparison rather than learning outcomes, framing the gap this study addresses."},{"cited_title":"and Cohen, A","cited_arxiv_id":null,"evidence_quote":"Provides the comparable 36% resubmission-correction rate that this paper's 38.46% improvement result is measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Argues for assessing automated tools by grades and learning outcomes, which this study operationalizes with project scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the course the data comes from, supporting the project-based design and platform details."}],"review_version":1}