{"id":"5732b240-a2b9-4a73-b6f8-5abe76733f9f","arxiv_id":"1908.08937","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"NMF on weekly log data from Clio Online reveals five student behavior clusters, showing that in-school quiz-taking and reading correlate with higher quiz scores, while at-home quiz use does not.","lead":"This paper applies a clustering technique called non-negative matrix factorization to millions of activity logs from a Danish online learning platform, sorting weekly student behaviors into patterns. It connects patterns like taking quizzes in school or reading at home to quiz performance, which could help teachers and platform designers choose better learning activities.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run random-initialized WNMF with an unspecified k rule is not shown to be stable; the Section 3.1 optimal/sub-optimal behavior labels may be factorization artifacts.","rationale":"The paper is an exploratory application of weighted NMF; the descriptive parts (Bloom levels per subject, system-wide time trends) are plausible and need no strong assumption beyond the factorization itself. The central performance claim, however, is exactly the part that requires the factorization to be a reliable representation of student behavior. The reader's weakest assumption—stability and meaningfulness of a single random-initialized WNMF run—is the load-bearing point; I agree with it. The paper's own Section 2.2 contains an unsupported assertion ('As a consequence clusters will be uncorrelated...') and the k-selection threshold is unspecified. The conclusion section also concedes that individual-level tracking 'was not directly analyzed,' so the abstract's tracking claim is not actually demonstrated. Because these concerns are testable and would determine whether the cluster labels correspond to reproducible patterns, the verdict should remain conditional; I would not move it to accept or reject. The practical recommendations should not be used until stability and out-of-sample checks are run.","tokens_in":7828,"tokens_out":5429,"duration_ms":51017,"concrete_test":"Re-run the Section 2.2 pipeline on the same data 50 times with random initializations for k=4, 5, and 6; align clusters across runs by Hungarian matching of V rows and report pairwise cosine distances and the correlation matrix of U columns. If matched clusters do not reproduce the C1-C5 loadings of Figure 3 within a pre-specified tolerance, or if U columns are substantially correlated at the selected k, the Section 2.2 justification and Figure 3 interpretation fail. As a secondary check, refit with f10 excluded: if the C2/C5 performance statements do not survive, those conclusions are driven by the self-selected quiz-score feature rather than by behavioral dependencies.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in the paper's central claim is the identification of the five columns of V (Section 3.1, Figure 3) with real behavioral-performance dependencies. This step rests on a single WNMF run with random initialization, a stopping threshold that is never specified ('some threshold, which depends on the initial error,' Section 2.2), and a k chosen with no reported validation. NMF is non-convex and, as the authors note, has scaling/rotation ambiguity; random restarts can converge to qualitatively different V matrices. The paper offers no repeated-run stability check, no comparison of k=4 vs k=5 vs k=6, and no out-of-sample or bootstrap validation. The assertion that this procedure 'will' produce student-level uncorrelated clusters is not derived from any stated property of the reconstruction error. The performance interpretation also leans on f10 (average quiz score), which is missing for students who took no quizzes; the WNMF missing-value imputation is a heuristic with an unspecified bound c, so the C2 high-f5/high-f10 pairing may reflect the subpopulation of quiz-takers rather than a dependency valid across students. Without these validations, the labels 'three optimal and two sub-optimal behaviors' and the claim that home use yields no performance gain are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes log data from Clio Online, a Danish online educational platform, using weighted non-negative matrix factorization (WNMF) to obtain soft clusterings of student-week feature vectors. Data from 14,810 students and roughly 3 million sessions over 112 weekly periods are aggregated into 14 features covering time of day, activity type, subject, Bloom's taxonomy level, session length, and quiz performance. The first experiment clusters students on features f1-f10 and interprets five clusters as optimal or sub-optimal behaviors, concluding, for example, that language students benefit most from quizzes during school hours while home use yields no performance gain. The second experiment clusters on subject and Bloom-level features and reports how exercise complexity varies across subject areas. The paper also plots average cluster membership over time to describe system-wide behavioral trends.","tokens_in":8052,"tokens_out":4604,"duration_ms":47376,"significance":"If the behavioral-performance dependencies reported here are reliable, they have practical value for educational intervention design, such as recommending quizzes for language students or encouraging in-school system use. The data set is large and from a real deployed system, and the weighted NMF extension for bounded missing values is a reasonable technical choice. The paper also illustrates a useful way to track soft cluster memberships over time. However, the contribution is primarily an application report, and the central claims currently rest on a single non-convex factorization without stability analysis, statistical validation, or sensitivity analysis. The paper does not ship code or data, so the validity of the conclusions depends entirely on the completeness of the methodological reporting, which is currently insufficient.","major_comments":[{"comment":"The factorization is performed with a single run of randomly initialized WNMF, and the number of clusters k is selected by an incompletely specified rule ('some threshold, which depends on the initial error'). Because NMF is non-convex and the authors themselves note the scaling/rotation ambiguity of U and V, a single run does not justify interpreting the five rows of V in Figure 3 as stable behavioral dependencies. Please report results over multiple random restarts (e.g., distributions of V entries, cophenetic correlation, or cluster agreement measures) and describe the exact k-selection procedure and threshold. Without such evidence, the 'three optimal and two sub-optimal behaviors' conclusion in Section 4 is not secured.","section":"Section 2.2 and Section 3.1"},{"comment":"The statement 'As a consequence clusters will be uncorrelated on a student level, since otherwise we would pick a lower k' is not a logical consequence of the stated k-selection rule. The rule compares reconstruction errors for successive k; it does not by itself constrain the correlations among columns of U. This claim should either be proved from the properties of the weighted multiplicative update or removed and replaced with a direct examination of the empirical correlation structure of U.","section":"Section 2.2"},{"comment":"The handling of missing values for f10 (average quiz score) depends on an unspecified bound c and on a heuristic in which weights are toggled when (UV)_ij crosses c. Since f10 is central to the performance-related conclusions, and since it is missing for students who took no quizzes in a period, the paper should report the fraction of missing entries, test sensitivity of the V matrix to c, and verify that the C2 high-f5/high-f10 association (Section 3.1) is not an artifact of the imputation heuristic or of the subpopulation of quiz-takers only.","section":"Section 2.2 and Table 1"},{"comment":"The manuscript does not state whether the columns of X are normalized or scaled before applying WNMF. The features have very different ranges (e.g., f1 has maximum 31.85 and variance 0.862, while f11 has maximum 2.83 and variance 0.006). Without column scaling, the Frobenius-error objective is dominated by large-magnitude features, which directly affects which loadings appear in V. Please specify the exact preprocessing, or justify the use of raw scales, and ideally include a sensitivity check under alternative scaling choices.","section":"Section 2.1 and Table 1"},{"comment":"The 'optimal' and 'sub-optimal' performance labels are read off from the fitted V matrix, and f10 is one of the features used to construct the factorization. Consequently, stating that a cluster 'has high performance' is in part a restatement of the fitted loadings rather than an independent empirical finding. To support claims such as 'home use yields no performance gain,' the paper needs statistical validation: for example, bootstrap confidence intervals or permutation tests on the differences between loadings, or an out-of-sample check of whether the non-f10 features and cluster memberships predict f10. Without such tests, the causal-sounding performance conclusions are not established.","section":"Section 3.1"}],"minor_comments":[{"comment":"In the C5 description, the phrase 'the high importance of f4 indicates that this most often occur for students working with languages' is confusing because f4 is reading texts, not language subjects; if the intended reference is f6 (language subjects), please align the text with Figure 3.","section":"Section 3.1"},{"comment":"There is a typo in the C1 description: 'These students seem to spent a lot of time reading (f4)' should read 'spend.'","section":"Section 3.1"},{"comment":"The session threshold of 600 seconds is justified by a recommendation from Clio Online, but a brief sensitivity analysis for this threshold would strengthen the reproducibility of the preprocessing.","section":"Section 2.1"},{"comment":"The temporal descriptions of correlation between clusters are qualitative; reporting the actual correlation coefficients of the average membership time series would make the claims in Section 3.1 and 3.2 more precise.","section":"Figures 5 and 8"},{"comment":"The sentence 'the clustering has many high values which is most likely explained by having a teacher who uses the system exclusively in only one of the subjects' is speculative; if it is intended as a finding, it needs supporting evidence.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of an educational data mining venue, and the application is timely, but the current version is methodologically under-supported for its headline claims. The missing stability analysis, unspecified parameters (especially c and the k-selection threshold), and lack of statistical validation of the performance conclusions are all fixable with additional experiments on the same data. Since the data are proprietary and no code is released, the onus is on the authors to provide a much more detailed experimental appendix. I see no grounds for rejection, but I would not recommend acceptance without a substantive revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent exploratory NMF analysis of a large, previously unstudied log dataset from a Danish primary-school platform. The descriptive clusters, especially the Bloom's-taxonomy breakdown by subject, are the real contribution. The performance-related conclusions in Section 3.1 are not established; they are read off a single fitted V matrix with no stability analysis, no validation, and an unspecified k-selection rule.\n\nWhat's new: the dataset (Clio Online, 14,810 students, 112 weeks) and the application to primary school, plus the separation of feature importance by subject and Bloom level. That's a legitimate new application of an existing method, and the second experiment is the more defensible half of the paper. The authors are honest about limitations: proprietary data, no teacher-assignment information, and they explicitly note that individual tracking was not actually done.\n\nSoft spots, in order of importance. First, the performance labels. Features include f10 (average quiz score), so clusters are partly defined by performance, and the paper then reports which behaviors go with high performance. That's descriptive, not predictive. There are no error bars, no bootstrap, no out-of-sample check, and no comparison of k=4 vs k=5 vs k=6. The assertion that the procedure 'will' yield student-level uncorrelated clusters is not derived from anything stated about the reconstruction error. Second, the missing-value heuristic for f10: the bound c is never specified, and the quiz/performance cluster C2 may reflect the subpopulation of quiz-takers. Third, reproducibility: no code, no parameters, proprietary data. That's not a flaw by itself, but it means the results are not independently checkable.\n\nThe stress-test note is right about the load-bearing step. The paper's central claim is the identification of five V columns with behavioral dependencies, and that rests on a single random-initialized run. I don't think the paper is wrong, but the confidence on these findings is unknown. The Bloom taxonomy results are less sensitive to that issue, because they don't depend on f10 and the cluster structure is clear from the plot.\n\nWho should read it: EDM researchers and anyone building teacher-facing dashboards. It deserves a serious referee, but the revision needs to reframe the performance claims as correlational, add a stability analysis, and specify the k rule. I'd accept it for review, with major revision expected.","headline":"Descriptive NMF cluster map of a large Danish primary-school dataset is worth a look, but the performance labels are not backed by stability or validation.","tokens_in":8607,"tokens_out":2402,"would_cite":false,"duration_ms":23079,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Weighted non-negative matrix factorization of weekly student-activity logs separates Danish primary-school students into five behavioral clusters, three tied to higher quiz performance and two tied to lower performance; the same…","keywords":["non-negative matrix factorization","soft clustering","educational data mining","student behavior","Bloom's taxonomy","online learning logs","Clio Online","performance"],"falsifier":"Run the same weighted NMF many times with different random initializations on the same data and compare the cluster matrices V: if the feature loadings change materially across runs, the 'optimal' and 'sub-optimal' labels are artifacts of initialization, not stable properties of student behavior.","tokens_in":7595,"feed_emoji":"📚","tokens_out":5406,"duration_ms":53234,"temperature":0.7,"pith_summary":"This paper analyzes log data from Clio Online, a Danish primary-school online learning platform, covering 14,810 students and three million sessions over two years. Using weighted non-negative matrix factorization on weekly activity features, it derives soft clusters of students that jointly describe time of day, subject, activity type, exercise complexity, and quiz performance. The authors claim the clusters reveal three optimal behaviors and two sub-optimal ones: working during school hours with longer sessions and taking quizzes in language subjects or reading texts in societal and science subjects is associated with better performance, while working mostly outside school hours and taking quizzes at home is not. They further show the method can track behavioral change over time and that system-wide usage is growing, especially in language subjects. A sympathetic reader would care because the result points to concrete, testable recommendations for how schools could nudge students toward more productive use of online learning tools.","feed_headline":"Five student clusters: three lift quiz scores, two don't","feed_subtitle":"Week-by-week clustering of 14,810 Danish students links time of day, subject, and activity type to performance.","key_machinery":"Non-negative matrix factorization (NMF): a factorization X ≈ UV of the non-negative student-period-by-feature matrix into a membership matrix U and a cluster matrix V, computed with multiplicative update rules adapted to missing values by weighting. The rows of V are rescaled to sum to one so clusters are comparable; a threshold on the decrease in reconstruction error selects the number of clusters k. This machinery does the work of turning millions of session logs into a small set of readable behavioral prototypes, and it lets the authors attach each student in each week a membership vector over the prototypes.","core_discovery":"The central discovery is that a low-rank non-negative factorization of a student-by-feature matrix, built from weekly aggregates of session logs, separates student behavior into interpretable clusters whose feature loadings align with performance. In the first experiment, five clusters emerge: science readers, language quiz-takers, societal readers, school-time language workers, and at-home quiz-takers; the first three carry high average quiz scores and long session lengths, while the last two do not. In the second experiment, three clusters separate Bloom's-taxonomy levels by subject: language work concentrates on remember/understand, societal work on analyze/evaluate, and science work uses almost no higher-level exercises. The authors conclude that the dependencies revealed by the factorization identify optimal and sub-optimal behaviors with respect to subject and performance, and that the same decomposition can be used to follow how memberships shift week by week.","pith_inferences":["The same pipeline could be run on other platforms' logs: the feature set is generic enough that the optimal/sub-optimal labels are testable hypotheses elsewhere, provided the factorization's k is validated per dataset.","The paper never analyzes individual students' transitions due to privacy, but its own method implies a natural next test: whether students who move from the at-home cluster to the school-hours clusters subsequently improve quiz scores within a few weeks.","The observed association between long sessions and high performance may reflect that high performers choose to stay longer, not that longer sessions cause performance; an intervention study or instrumental-variable design is needed before recommending longer sessions."],"forward_implications":["For language subjects, recommending quizzes and keeping work within school hours should, if the dependencies are causal, raise average quiz scores more than assigning reading.","For societal and science subjects, reading texts during school hours is the behavior most associated with high performance, so content and assignments should emphasize sustained reading.","Working mostly outside school hours and taking quizzes there is associated with no performance boost, so schools might investigate why home sessions do not translate into quiz scores.","System-wide tracking shows average time spent in the platform rising from about 15 to 25–45 minutes per week per subject, so the cluster structure can serve as a monitoring tool for adoption trends.","Because clusters are uncorrelated at student level while correlated at system level, aggregate dashboards can mask individual behavioral trade-offs; school-level interpretations should rely on membership distributions rather than averages."],"supporting_citations":[{"why":"Supplies the multiplicative update algorithm that the paper adapts to weighted NMF.","marker":"[9]"},{"why":"Provides the weighted-NMF formulation with a binary weight matrix that handles missing quiz scores.","marker":"[6]"},{"why":"Fixes the non-convergence issue in the multiplicative updates and is incorporated into the procedure.","marker":"[11]"},{"why":"Is the prior MOOC study using NMF for weekly soft clustering of student activity, the direct methodological ancestor.","marker":"[3]"},{"why":"Demonstrates NMF as a document-clustering technique, supporting the interpretation of U and V as soft memberships.","marker":"[13]"},{"why":"Surveys NMF clustering and motivates treating clusters as additive latent behaviors.","marker":"[10]"}],"fun_headline_variants":["NMF finds five student clusters, three outperform quiz losers","Behavior clusters from 3M sessions predict quiz gains","Week-by-week NMF tracks student behavior shifts to performance","Danish e-learning logs reveal five clusters with performance links","Three high-scoring clusters emerge from NMF of student logs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single factorization produced by one randomly initialized run, with its chosen number of clusters, is stable and meaningful enough that the feature loadings can be read as real behavioral dependencies.","fun_headline_variants_meta":{"raw":{"variants":["NMF finds five student clusters, three outperform quiz losers","Behavior clusters from 3M sessions predict quiz gains","Week-by-week NMF tracks student behavior shifts to performance","Danish e-learning logs reveal five clusters with performance links","Three high-scoring clusters emerge from NMF of student logs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001574,"raw_usage":{"total_tokens":6241,"prompt_tokens":861,"completion_tokens":5380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":5300}},"tokens_in":477,"tokens_out":5380,"duration_ms":36938,"temperature":1.0,"reasoning_tokens":5300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:54:43.228673+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same weighted NMF many times with different random initializations on the same data and compare the cluster matrices V: if the feature loadings change materially across runs, the 'optimal' and 'sub-optimal' labels are artifacts of initialization, not stable properties of student behavior.","supporting_citations":[{"cited_title":"Semi-Markov model for simulating MOOC students","cited_arxiv_id":null,"evidence_quote":"Supplies the multiplicative update algorithm that the paper adapts to weighted NMF."},{"cited_title":"We have identified three optimal and two sub-optimal behaviors in relation to subject and performance","cited_arxiv_id":null,"evidence_quote":"Provides the weighted-NMF formulation with a binary weight matrix that handles missing quiz scores."},{"cited_title":"Sequence modelling for analysing student interaction with educational systems","cited_arxiv_id":null,"evidence_quote":"Fixes the non-convergence issue in the multiplicative updates and is incorporated into the procedure."},{"cited_title":"Sev- eral of these algorithms have been adapted for the WNMF case, while approaches based on expectation maximization have also been proposed, see [6]","cited_arxiv_id":null,"evidence_quote":"Is the prior MOOC study using NMF for weekly soft clustering of student activity, the direct methodological ancestor."},{"cited_title":"Weighted Nonnegative Matrix Factorization","cited_arxiv_id":null,"evidence_quote":"Demonstrates NMF as a document-clustering technique, supporting the interpretation of U and V as soft memberships."},{"cited_title":"Gelman, Matt Revelle, Carlotta Domeniconi, Kalyan Veeramachaneni, and Aditya Johri","cited_arxiv_id":null,"evidence_quote":"Surveys NMF clustering and motivates treating clusters as additive latent behaviors."}],"review_version":1}