Pith. sign in

REVIEW 4 major objections 4 minor 17 references

Behavior Pattern and Compiled Information Based Performance Prediction in MOOCs

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that 23 compiler-error-count features from programming assignments, plus submission-behavior sequences, let a neural network predict MOOC final-exam pass/fail at 70.49% accuracy.

desk verdict Real data and a sensible feature set, but the headline accuracy is not shown to be out-of-sample; worth refereeing with a demand for proper validation. read the letter →

arxiv 1908.01304 v1 pith:2D3WDFEC submitted 2019-08-04 cs.IR cs.CY

Shaojie Qu , Kan Li , Zheyi Fan , Sisi Wu , Xinyi Liu , Zhiguo Huang This is my paper · ORCID
classification cs.IRcs.CY
keywords MOOCsperformancepredictionsequentialpatternminingcompiledinformationprogrammingassignmentsmultilayerperceptronfeatureselectionplagiarismbehavior
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Using logs from a C programming MOOC with 1,528 students, the paper tries to establish two things. First, students who fail the final exam leave recurring patterns in how they submit assignments—submission times, submission order in the class, and plagiarism—while passing students show no such common pattern. Second, 23 features drawn from the compiler's report (counts of 22 error types plus successful compilations), ranked by random-forest importance and fed to a multilayer perceptron, predict pass/fail at 70.49% accuracy on a held-out 20% of students. This matters because assignment logs are a direct record of how students actually work, unlike the video and forum behaviors used in most prior work, and because compiler error counts are automatically available.

What carries the argument

The work is carried by two objects. One is the standardized feature sequence: for each of 14 weekly assignment groups, a student gets a value for average submission time (mapped to -2,-1,0,1,2 by the difference from the class average), average submission order (binned into 1,2,3), and plagiarism count (0,1,2); a GSP-based algorithm then mines these sequences for patterns that match at least 70% of a performance group and have accuracy and recall above 70%. The other is the compiled-information feature vector: 23 counts (22 compiler-error types and successful compilations), ranked by random-forest importance and then added greedily to a multilayer perceptron until accuracy stops improving.

What would settle it

Rerun the same pipeline with 10-fold cross-validation, performing feature selection and threshold choice inside each training fold, and compare the mean accuracy with a classifier that always predicts the majority class on the same data; if the cross-validated accuracy is at or below that baseline, the central claim that the features carry predictive signal would be refuted.

Watch

Extended reading notes

Core claim

The central claim is the combination of the two results. On the behavior-pattern side, after standardizing each student's submission-time, submission-order, and plagiarism sequences over 14 weekly assignment groups, a GSP-style sequential-pattern search with a 70% matching threshold finds patterns such as 'submits after more than 1000 classmates in at least two groups' and 'commits serious plagiarism in at least one group'; these patterns reach 70–79% accuracy and 70–85% recall among failing students, while passing students have no patterns reaching the threshold. On the prediction side, the single most important compiled-information feature is the count of error-free compilation attempts ('None', importance 0.473), followed by syntax-error count (0.198); using the selected features, a three-hidden-layer multilayer perceptron trained on 80% of the data achieves 70.49% accuracy and 70.16% recall on the remaining 20%, outperforming Naïve Bayes, logistic regression, and support vector machines on the same features.

Load-bearing premise

The load-bearing premise is that the 80/20 split is a genuinely out-of-sample test: the manuscript does not state whether the random-forest feature ranking and the 70% pattern-selection thresholds were computed before or after the split, and it does not report the pass/fail base rate, so the 70.49% accuracy has no stated confidence interval or baseline comparison.

Editorial extensions

If this is right

  • Assignment compilation logs are a usable early-warning signal for MOOC failure, complementing video and forum behavior.
  • A small set of compiler-error counts, led by error-free attempts and syntax errors, can be collected automatically and cheaply during a course.
  • The specific failure patterns—late submission order, high early submission counts followed by low later ones, and plagiarism—point to concrete, automatically triggerable intervention moments.
  • Any future predictor for this course should be compared against the 70.49% accuracy and against the unreported majority-class baseline.
  • Because the features are defined at the compiler level, the approach could in principle be applied wherever programming assignments are graded by compilation, a direction the paper leaves open.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 70.49% is likely an optimistic upper bound: the paper does not describe cross-validation, and if the random-forest feature ranking or the 70% pattern thresholds were chosen using the full dataset, the test accuracy is not a clean out-of-sample estimate.
  • The asymmetry between failing and passing students suggests a one-class 'failure signature' detector rather than a symmetric classifier; combining the pattern scores with the compiled features is a natural next step the paper leaves open.
  • The binning thresholds for submission times and order are arbitrary; a sensitivity check across thresholds would show whether the discovered patterns are stable or artifacts of the discretization.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper analyzes logs from a C programming MOOC with 1,528 students to address two research questions: whether students who pass or fail show recurring assignment-submission behavior patterns, and whether compiler-related features from programming assignments can predict pass/fail. The authors construct three 14-element feature sequences per student (submission times, submission order, plagiarism), apply a GSP-based mining method with 70% matching, accuracy, and recall thresholds, and report that failing students show repeated patterns while passing students do not. They also extract 23 features (22 compiler-error counts and one success count), rank them with random forest importance, and train an MLP with three hidden layers, reporting 70.49% accuracy on an 80/20 split. The conclusion is that performance can be predicted from compiled assignment information.

Significance. If the reported accuracy were a valid out-of-sample estimate, the compiled-information features would be a practically useful, low-cost signal for MOOC performance prediction, complementing the usual video and forum behavior features. The claimed asymmetry between failing and passing students ('all roads lead to Rome, but failure is always the same') is also an interesting educational hypothesis. The paper addresses a real gap by focusing on assignment and compiler data rather than only clickstream behavior. However, the current evaluation does not establish the central quantitative claim: the feature-selection protocol and the pattern-selection thresholds appear to be evaluated on the same data used to make the selections, and the class distribution is never reported. The results are therefore suggestive rather than conclusive, and the manuscript needs a stricter evaluation protocol before the claims can be accepted.

major comments (4)
  1. [IV.C, V.B] The reported 70.49% accuracy is not shown to be an out-of-sample estimate. Section IV.C describes random-forest feature ranking and a greedy feature-addition loop ('When the accuracy is no longer improved, we stop adding features') and only then states that the model was trained on 80% of the data and tested on the remaining 20%. The manuscript does not state that the feature ranking and the stopping criterion were restricted to the training partition. If the held-out 20% influenced feature selection, the accuracy is an in-sample fit. Moreover, the single split gives no confidence interval; with roughly 306 test students, a 70.49% accuracy has a 95% confidence interval of about ±5 percentage points. I ask the authors to use nested cross-validation or a clear separation of feature selection and evaluation, and to report the standard deviation or confidence interval across repeated splits.
  2. [IV.B, V.A, Tables II-IV] The behavior-pattern analysis is circular. The authors mine sequences separately for passed and failed students, then retain only sequences whose accuracy and recall exceed 70% on the same data, and then report those retained accuracy and recall values as evidence that the patterns are predictive. Because the 70% accuracy/recall thresholds are used as selection filters, the reported values in Tables II-IV are guaranteed to be at least 70% by construction; they do not measure predictive validity. The claim that failing students have 'obvious sequence patterns' while passing students do not is therefore a threshold artifact unless the patterns are validated on a holdout set or against a null model (e.g., random permutations of the sequence labels). The authors should also report how many candidate patterns were considered and how many were discarded by the filter.
  3. [V.B, Table VI] The manuscript never reports the pass/fail class distribution, so the gain over a majority-class baseline is unknown. If, for example, 70% of students in the test set failed, then always predicting 'fail' would already achieve roughly 70% accuracy, and the reported MLP accuracy of 70.49% would represent almost no improvement. Table VI compares MLP to Naive Bayes, logistic regression, and SVM, but not to a majority-class predictor or to a no-information baseline. The authors should report the class distribution, the majority-class accuracy, and ideally the confusion matrix, ROC AUC, or F1 score to contextualize the reported accuracy.
  4. [IV.A, Eq. (1)] The sequence standardization in Eq. (1) uses Xj, the average submission time over all students in the same assignment group, before any train/test division. For the descriptive pattern analysis this is acceptable, but if any of the sequence patterns are later used for prediction or are evaluated as generalizable findings, the use of whole-cohort statistics can introduce leakage and overstate the apparent regularity. If the authors intend the pattern results in Section V.A as predictive evidence, they should repeat the standardization and pattern mining within each training fold or report the analysis as purely descriptive, with a clear caveat.
minor comments (4)
  1. [Throughout] There are several typos and stylistic issues: 'multilayer perception' should be 'multilayer perceptron'; 'LogicRegression' in Table VI should be 'Logistic Regression'; the abstract uses '0.7049' while the text uses '70.49%'; and the phrase 'the fourth section introduces the data and describes the methods of the results' is grammatically unclear. A careful language edit is needed.
  2. [Fig. 1] Figure 1 is mentioned as showing the relationship between submission order and final grade, but the text only gives a qualitative summary. The axis labels, units, and the distribution of grades for each submission-order value should be described, and the figure should be referenced with a more substantive interpretation.
  3. [IV.C] The description of the MLP architecture is incomplete: the number of neurons per hidden layer, the activation function, the regularization, the optimization algorithm, and the number of training epochs are not stated. Providing these details would improve reproducibility.
  4. [V.A] The interpretation of the submission-time patterns as evidence of cheating ('the students had cheated and did not try to complete the assignments') is presented without supporting evidence; low submission counts alone do not establish plagiarism or cheating. The authors should either soften the causal language or provide additional data linking the behavior to verified plagiarism.

Circularity Check

2 steps flagged · score 6.0 of 10

Behavior-pattern accuracy is selected by the same 70% thresholds used to report it; MLP feature selection may not be confined to the training split.

  1. fitted input called prediction [Section IV.B (Method for question A) and Section V.A, Tables II–IV]
    "We first exhaust all possible sequences with a length of 1, delete the sequences that do not meet the requirement of 70% matching degree, and then judge whether the accuracy of the screened sequences in predicting student performance is greater than 70%. We retain only the sequences with both an accuracy rate and a recall rate greater than 70%."

    The patterns reported as evidence that failing students have predictive behavioral patterns are exactly those that survived a filter requiring accuracy and recall above 70% on the same data. Table II then reports patterns whose accuracy is 70.01–71.38% and recall 75.78–81.44%, i.e., values that satisfy the Section IV.B thresholds. The reported accuracy/recall are therefore constraints imposed during pattern selection, not independent out-of-sample measurements; the conclusion 'failure is always the same' is an artifact of the 70% selection rule.

  2. fitted input called prediction [Section IV.C and Section V.B (MLP feature selection and accuracy)]
    "After extracting these 23 features, we use the random forest feature selection method to rank the features by importance. Then, we use multilayer perception (MLP) with three hidden layers to predict students’ performance. We trained with 80% of the data and predicted the remaining 20%. ... According to the ranking of feature importance, we add features to the optimal feature set from high importance to low importance and train them with MLP. When the accuracy is no longer improved, we stop adding features to obtain the optimal feature set."

    The paper does not state that the random-forest importance ranking or the greedy 'stop when accuracy no longer improves' loop was restricted to the 80% training partition. As written, the feature set is selected by optimizing the same accuracy value that is later reported as the predictive result. If the remaining 20% informed the ranking or the stopping criterion, the 70.49% accuracy is a fitted selection metric rather than an out-of-sample prediction. This is a leakage risk that is not excluded by the text.

full rationale

The behavior-pattern result in Section V.A is circular by construction: Section IV.B retains only sequences with accuracy and recall above 70%, and Tables II–IV report exactly such sequences as evidence that failing students share predictive patterns. That is a fitted input presented as a finding. For the compiled-information MLP, the raw features are substantive and an 80/20 split is described, but the manuscript never states whether feature ranking and the greedy feature-addition criterion were confined to the training partition; if the test split was used in selection, the headline 70.49% is not a valid out-of-sample estimate. The paper also does not report the pass/fail base rate, so improvement over a majority-class predictor is unknown; that is a correctness limitation, not circularity. I found no load-bearing self-citation or imported uniqueness theorem. Overall score 6 reflects one directly circular pattern-mining evaluation plus a second prediction whose evaluation protocol is unverified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper relies on several hand-chosen preprocessing thresholds, including the standardization bins and the 70% mining/filter thresholds. These choices directly shape the reported sequence patterns. The evaluation also depends on the unstated assumption that a single 80/20 split and the reported pass/fail threshold produce a reliable ground truth. No new entities, particles, or physical mechanisms are introduced.

free parameters (6)
  • Submission-time standardization thresholds = boundaries at -0.5, 0, 0.5; values -2, -1, 0, 1, 2
    Chosen by hand in Eq. (1) of Section IV.B; they define the discrete alphabet of the submission-time sequence and therefore all discovered patterns. No sensitivity analysis is provided.
  • Submission-order bin boundaries = 0-500 to 1, 501-1000 to 2, >1001 to 3
    Chosen by hand in Section IV.B; changing these cutoffs changes the order-sequence alphabet and the resulting pattern-mining output.
  • Plagiarism bin boundaries = 0 to 0, 1-2 to 1, >2 to 2
    Chosen by hand in Section IV.B; affects the plagiarism sequence and the reported plagiarism patterns in Table IV.
  • GSP matching threshold = 70%
    The minimum matching degree required to call a sequence a pattern in Section IV.B; selected without justification and directly determines which patterns are retained.
  • Pattern accuracy/recall filter = 70%
    Section IV.B retains sequences only when both accuracy and recall exceed 70%, and because the evaluation uses the same data, this filter partly forces the reported high values.
  • MLP architecture and feature count = 3 hidden layers; number of selected features not reported
    Section IV.C states a three-hidden-layer MLP but does not report hyperparameters or the final optimal feature set, and the stopping rule for adding features is not fully specified.
assumptions (5)
  • standard math Prefix anti-monotonicity for sequence support: if a full sequence matches 70% of the data, any prefix matches at least 70%.
    Invoked in Section IV.B to prune the GSP-style search; this is a standard support property of sequential pattern mining.
  • domain assumption Final exam score below 60 defines failure.
    Used as the ground-truth label for all prediction and pattern analysis in Sections IV.C and V; the cutoff is not justified.
  • domain assumption Compiler-error categories and submission logs capture learning status and effort.
    The paper's premise in Section IV.A that compiled information reflects learning status; no validation that error counts are not confounded by assignment difficulty or compiler behavior.
  • domain assumption Grouping 69 assignments into 14 weekly groups preserves the behavioral signal.
    Averaging within groups in Section IV.A assumes within-group variation is noise and that weekly grouping is meaningful; no robustness check is provided.
  • domain assumption Plagiarism flags in the log are accurate and meaningful.
    The plagiarism sequence is used in Section IV.B as a proxy for effort or cheating, but the detector's error rate and the definition of plagiarism are not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Behavior Pattern and Compiled Information Based Performance Prediction in MOOCs." pith.science (2026). https://pith.science/paper/2D3WDFEC

@misc{pith2026190801304,
  author       = {Pith},
  title        = {Pith review of: Behavior Pattern and Compiled Information Based Performance Prediction in MOOCs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2D3WDFEC}},
  note         = {Machine review of arXiv:1908.01304}
}
read the original abstract

With the development of MOOCs massive open online courses, increasingly more subjects can be studied online. Researchers currently show growing interest in the field of MOOCs, including dropout prediction, cheating detection and achievement prediction. Previous studies on achievement prediction mainly focused on students' video and forum behaviors, and few researchers have considered how well students perform their assignments. In this paper, we choose a C programming course as the experimental subject, which involved 1528 students. This paper mainly focuses on the students' accomplishment behaviors in programming assignments and compiled information from programming assignments. In this paper, feature sequences are extracted from the logs according to submission times, submission order and plagiarism. The experimental results show that the students who did not pass the exam had obvious sequence patterns but that the students who passed the test did not have an obvious sequence pattern. Then, we extract 23 features from the compiled information of students' programming assignments and select the most distinguishing features to predict the students' performances. The experimental results show that we can obtain an accuracy rate of 0.7049 for predicting students' performances.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

  1. [1]

    Adaptive recommendation for MOOC with collaborative filtering and time series,

    Y. Pang, W. Liu, Y. Jin, H. Peng, T. Xia, and Y. Wu, “Adaptive recommendation for MOOC with collaborative filtering and time series,” Comput. Appl. Eng. Educ.,vol.26,no.6,pp.2071–2083,2018

    work page 2018

  2. [2]

    Detectingandpreventing ‘multiple-account’ cheating in massive open online courses,

    C.G. Northcutt,A. D. Ho, andI. L. Chuang,“Detectingandpreventing ‘multiple-account’ cheating in massive open online courses,” Comput. Educ.,vol.100,pp.71–80,2016

    work page 2016

  3. [3]

    Student success prediction in MOOCs,

    J. Gardner and C. Brooks, “Student success prediction in MOOCs,” User Model. User-Adapt. Interact.,vol.28,no.2,pp.127–203,2018

    work page 2018

  4. [4]

    Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence,

    Z. Papamitsiou and A. A. Economides, “Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence,” J. Educ. Technol. Soc., vol. 17, no. 4, pp. 49– 64,2014

    work page 2014

  5. [5]

    The different relationships between engagement and outcomes across participant subgroups in Massive OpenOnlineCourses,

    L. Qiujie and B. Rachel, “The different relationships between engagement and outcomes across participant subgroups in Massive OpenOnlineCourses,” Comput. Educ.,no.127,pp.41–65,Dec.2018

    work page 2018

  6. [6]

    How automated feedback through text mining changes plagiaristic behavior in online assignments,

    Gökhan Akçapınar, “How automated feedback through text mining changes plagiaristic behavior in online assignments,” Comput. Educ. 87 2015,vol.87,pp.123–130,Sep.2015

    work page 2015

  7. [7]

    Programming Pluralism: Using Learning Analytics to Detect PatternsintheLearningofComputerProgramming,

    P. Blikstein, M. Worsley, C. Piech, M. Sahami, S. Cooper, and D. Koller, “Programming Pluralism: Using Learning Analytics to Detect PatternsintheLearningofComputerProgramming,” J. Learn. Sci.,vol. 23,pp.561–599,2014

    work page 2014

  8. [8]

    Automatically detectable indicators of programming assignment difficulty,

    P. Ihantola, J. Sorva, and A. Vihavainen, “Automatically detectable indicators of programming assignment difficulty,” Acm Sigite Newsl., vol.10,no.2,pp.10–10,2015

    work page 2015

Show all 17 references
  1. [9]

    ACE-RL-Checkers: decision-making adaptability through integration of automatic case elicitation, reinforcementlearning,andsequentialpatternmining,

    H. C. Neto and R. M. S. Julia, “ACE-RL-Checkers: decision-making adaptability through integration of automatic case elicitation, reinforcementlearning,andsequentialpatternmining,” Knowl. Inf. Syst., vol.57,no.3,pp.603–634,2018

    2018

  2. [10]

    Mining Sequential Patterns,

    R. Agrawal and R. Srikant, “Mining Sequential Patterns,” presented at the11thInternationalConferenceonDataEngineering,1995,vol.1,pp. 3–14

    1995

  3. [11]

    Mining Sequential Patterns: Generalizations and Performance Improvements,

    R. Srikant and R. Agrawal, “Mining Sequential Patterns: Generalizations and Performance Improvements,” presented at the 5th Int’lConferenceonEDBT,1996,pp.3–17

    1996

  4. [12]

    SPADE: An Efficient Algorithm for Mining Frequent Sequences,

    M. J. Zaki, “SPADE: An Efficient Algorithm for Mining Frequent Sequences,” Mach. Learn.,vol.42,no.1–2,pp.31–60,2001

    2001

  5. [13]

    SequentialPAtternMining using A Bitmap Representation,

    J.Ayres,J.Flannick,J.Gehrke,andT.Yiu,“SequentialPAtternMining using A Bitmap Representation,” presented at the Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, 2002, pp. 429--435

    2002

  6. [14]

    FreeSpan: Frequent Pattern-projected Sequential Pattern Mining,

    J. Han, J. Pei, B. Mortazavi-Asl, Q. Chen, U. Dayal, and M.-C. Hsu, “FreeSpan: Frequent Pattern-projected Sequential Pattern Mining,” in Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 2000, pp.355–359

    2000

  7. [15]

    Mining sequential patterns by pattern-growth: the PrefixSpan approach,

    J. Pei et al., “Mining sequential patterns by pattern-growth: the PrefixSpan approach,” IEEE Trans. Knowl. Data Eng., vol. 16, no. 11, pp.1424–1440,Nov.2004

    2004

  8. [16]

    BIDE: Efficient mining of frequent closed sequences,

    J. Y. Wang and J. W. Han, “BIDE: Efficient mining of frequent closed sequences,” presented at the International Conference on Data Engineering,Boston,MA,USA,2004,pp.79–90

    2004

  9. [17]

    Multi- dimensional Sequential Pattern Mining,

    H. Pinto, J. Han, J. Pei, K. Wang, Q. Chen, and U. Dayal, “Multi- dimensional Sequential Pattern Mining,” in Proc. of the Tenth International Conference on Information and Knowledge Management (CIKM 2001),Atlanta,Georgia,USA,2001,pp.81–88

    2001

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.