{"id":"19a14808-d1c3-45fe-b634-672727e47635","arxiv_id":"2608.13409","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Transformer that jointly predicts next-semester courses and grades (TRACE) reduces grade MAE by 46% over a grade-only Transformer on a single university's ten-year dataset.","lead":"What if an AI could predict not just the grades a student will get, but which courses they will take in the first place? This paper tests that idea with a Transformer model trained on ten years of university data, and finds that asking the model to predict both together cuts grade error by nearly half compared to predicting grades alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 46.4% grade-MAE improvement rests on an undertrained-looking OnlyGradesTransformer baseline; its MAE (0.2496) is far worse than the LSTM baselines, so the claimed advantage may be a tuning artifact.","rationale":"The reader's weakest assumption was that OnlyGradesTransformer was trained with the same hyperparameters, loss weighting, and convergence conditions as TRACE, and that the large MAE gap is not a tuning artifact. My independent reading agrees: this is the most load-bearing assumption in the paper. The strongest version of the claim, that the auxiliary course-prediction head causes a ~46% grade-MAE improvement, collapses if the baseline is underfit. The empirical red flag is the baseline's absolute performance: a grade-only Transformer with course embeddings and the same architecture should not be 55% worse than an LSTM on the same data. This strongly suggests the comparison is not apples-to-apples. The paper reports no grid search, no learning-rate curves, no repeated runs, and no significance test, so the reader's conditional verdict is appropriate. I do not see a more fundamental flaw: the set-based KL loss is a reasonable multi-label objective; the evaluation on actual courses is a defensible way to isolate grade accuracy; and the single-institution limitation is acknowledged. The Section 4.2 MAE/MSE confusion is real but prose-level, not central to the relative claim. A concrete re-tuning experiment would settle whether the headline gap is genuine; until then, CONDITIONAL remains the right verdict.","tokens_in":13798,"tokens_out":10628,"duration_ms":116873,"concrete_test":"Retrain OnlyGradesTransformer on the same 90/10 split with a hyperparameter search over learning rate (e.g., 1e-5, 3e-5, 1e-4, 3e-4), training length (20, 50, 100 epochs with early stopping), and loss weighting for grade MSE, using at least 5 random seeds. Report the best test MAE and its 95% confidence interval. If the best grade-only MAE stays above 0.20, the baseline is not the source of the gap; if it falls to roughly 0.16 (the LSTM level) or lower, the claimed 46.4% reduction is largely an artifact of an undertrained baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 3.2, Table 1) is that jointly predicting courses and grades cuts grade MAE by 46.4% relative to an otherwise identical grade-only Transformer. For that comparison to be meaningful, OnlyGradesTransformer must be a fairly tuned, converged baseline. The paper fixes 20 epochs for all models, reports no learning-rate schedule details for the baseline, does not document whether omega in Eq. (4) was tuned separately for the grade-only model, and provides no repeated runs or error bars. More tellingly, OnlyGradesTransformer's MAE of 0.2496 is about 55% worse than the two LSTM baselines (0.1608 and 0.1619), which is implausible for a Transformer with the same input, embeddings, and capacity. This pattern suggests the baseline may be severely underfit or that the combined course-KL plus grade-MSE loss in TRACE changes the effective training dynamics in a way that is not controlled for. If a properly tuned grade-only Transformer reaches MAE close to the LSTM level (~0.16), the reported 46.4% improvement would shrink to roughly 15-20%, materially weakening the paper's headline. The absence of multiple seeds also leaves open the possibility that the gap is partly run-to-run variance. Thus the load-bearing assumption is not that joint prediction helps, but that the grade-only baseline is a fair, converged comparator.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TRACE, a Transformer-based encoder-decoder model that jointly predicts the set of courses a student will take in the next semester and the grades they will receive in those courses, given their major and prior course/grade history. Courses taken in the same semester share a positional encoding, making the within-semester representation permutation-invariant, and the training loss combines a KL divergence over the course-set distribution with a grade MSE term. Using ten years of data from a single private university, the authors report that TRACE achieves a grade MAE of 0.1339, a 46.4% reduction relative to a grade-only Transformer baseline, and that it also outperforms LSTM, GNN, and XGBoost baselines. The paper frames the auxiliary course-prediction task as a representation-learning mechanism and releases code and an anonymized data subset.","tokens_in":14083,"tokens_out":6324,"duration_ms":61140,"significance":"If the central comparison is trustworthy, the paper makes a useful contribution to learning analytics: it gives a concrete way to encode concurrent semester enrollments in a Transformer, proposes a set-based KL loss that avoids order artifacts, and provides evidence that an auxiliary course-prediction task can improve grade regression. The evaluation design has a notable strength: grade error is computed only for ground-truth enrolled courses, so the reported grade metrics are not contaminated by course-selection accuracy. The release of training code and anonymized data is also a plus for reproducibility. However, the headline 46.4% improvement rests on a baseline whose training is not documented well enough to be a fair comparator, and the reported differences come from a single 90/10 split without variance or significance testing. These issues must be resolved before the magnitude of the claimed benefit can be accepted.","major_comments":[{"comment":"The central claim of a 46.4% MAE reduction is not supported as reported because the OnlyGradesTransformer baseline is not documented sufficiently to be a fair comparator. The baseline is described in one sentence in Section 3.1, with no learning rate, number of layers or heads, embedding dimension, convergence criterion, or whether the loss weight omega in Eq. (4) was tuned for it; all models are fixed at 20 epochs, and no repeated runs or error bars are given. The baseline's MAE of 0.2496 is roughly 55% worse than the two LSTM baselines (0.1608 and 0.1619) despite using the same input and more capacity, which is consistent with an undertrained or mistuned baseline rather than an architectural limitation. Please provide full baseline details, tune the grade-only Transformer comparably (including omega and learning rate), report results over multiple seeds with mean and standard deviation, and test the MAE/MSE differences for significance; without this, the headline improvement cannot be distinguished from tuning imbalance or run-to-run variance.","section":"Section 3.2 / Table 1"},{"comment":"The one-step-ahead training procedure is inconsistent with the next-semester prediction task defined in Section 2.1. Section 2.1 states that the output is the courses and grades of the immediately following semester, but the example in Eq. (6) sets the target for input (<SOS>, c1) to (c2,...,cn,<EOS>), i.e., all remaining courses and grades across all future semesters. Please clarify whether the model is trained to predict only the next semester or the entire remaining suffix, and describe how the Table 1 grade metrics are restricted to the ground-truth next semester. This is essential for interpreting the reported errors and for comparing TRACE with the baselines, which may use different target horizons.","section":"Section 2.3.2"},{"comment":"The early-alert paragraph misreports the main metric. It states 'Our model achieves a grade prediction MAE of 0.0392 on a [0,1] GPA scale,' but Table 1 lists MSE=0.0392 and MAE=0.1339. The subsequent conversion to a [0,4] scale (0.1568) is therefore also incorrect; the correct conversion of the MAE is 0.1339*4 = 0.5356. Please correct the text and re-assess the claim that predictions are accurate to about half of a +/- grade.","section":"Section 4.2"}],"minor_comments":[{"comment":"The label encoding of course names is said to be in 'random order'; please specify the random seed or state why the ordering cannot affect learned embeddings.","section":"Section 2.2.1"},{"comment":"Eq. (4) uses omega only on the grade loss; please state whether omega was tuned separately for each model (especially OnlyGradesTransformer) and report the selected values, since this affects the comparison.","section":"Section 2.3.1"},{"comment":"The major-removal ablation reports that removing major improves performance, but without repeated runs it is unclear whether the difference (0.1339 vs 0.1327 MAE) is meaningful; a variance estimate would help.","section":"Section 3.2.1"},{"comment":"Please fix typos: 'Kullback-Liebler' should be 'Kullback-Leibler' (Section 2.2.9), 'binary cross entry' should be 'binary cross entropy' (Section 2.3.1), 'and and node embeddings' appears in Section 1.1, and the Table 1 caption has a stray space in 'T able 1'.","section":"Throughout"},{"comment":"The random-selection baseline of 5/360 = 0.014 for course precision is helpful; consider also reporting the course-loss values (KL or BCE) for TRACE and OnlyGradesTransformer to quantify the auxiliary task's fit.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the code/data release is a strength, but the main quantitative claim currently rests on a baseline whose fairness is not established and on a single data split. I would not recommend rejection if the authors can provide the additional experiments and clarifications; the single-institution scope is acceptable given the paper's framing, provided the baseline and statistical issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: TRACE is a reasonable applied contribution. The permutation-invariant semester encoding (same positional vector for all courses in a term) and the KL-based set loss are sensible, and the paper is honestly written with a decent limitations section. But the headline comparison against OnlyGradesTransformer is not trustworthy as reported: that baseline's MAE of 0.2496 is far worse than the LSTM baselines (~0.16), which strongly suggests it was undertrained or mistuned. If a properly tuned grade-only Transformer lands near the LSTM level, the 46.4% advantage shrinks to roughly 15–20%. That still leaves TRACE ahead of the other baselines, but the paper's mechanistic claim—that forcing course prediction is what helps—becomes much weaker.\n\nWhat is genuinely new: the same-position encoding for all courses within a semester, and the normalized KL divergence over course sets as a training target, with a nice ablation showing BCE gives nearly the same grade accuracy. The authors also include a detailed GNN baseline and are transparent about single-institution data and cold-start issues. Code is linked, though the data is mostly private.\n\nSoft spots, in order of severity. First, the grade-only baseline is the load-bearing comparison and it is underdocumented: no learning rate, no omega tuning, no convergence check, no repeated seeds. The LSTM comparison makes this worse, not better, because it implies an identical architecture with less capacity beats the baseline. Second, a single 90/10 split with no confidence intervals; run-to-run variance could explain part of the gap. Third, Section 4.2 calls the MSE (0.0392) the MAE; it is a typo, but it undercuts confidence in the prose. Fourth, the omega weight is set \"near one\" with no tuning details.\n\nNone of this sinks the paper. The method is sensible, the results are directionally consistent across the LSTM and GNN baselines, and the limitations are handled honestly. But the headline number should be treated as an upper bound until the baseline is fixed. This paper is for learning analytics researchers and institutional research offices; a serious referee should be engaged, but the revision needs re-run baselines with proper tuning and report variance. I would bring it to a reading group as a case study in baseline fairness.","headline":"Joint course-grade prediction is a plausible and honest contribution, but the headline 46% gain over the grade-only baseline is likely inflated by an undertrained comparator.","tokens_in":14595,"tokens_out":2329,"would_cite":false,"duration_ms":23884,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TRACE, a Transformer that jointly predicts next-semester courses and grades, cuts grade-prediction mean absolute error by 46.4% compared with an identical grade-only architecture.","keywords":["learning analytics","educational data mining","grade prediction","course prediction","transformer","multi-task learning","set-based loss","course concurrency"],"falsifier":"Run OnlyGradesTransformer and TRACE over multiple random seeds with per-model hyperparameter search, reporting learning rate, layer count, and the weighting omega; if a tuned grade-only model reaches TRACE's mean absolute error of about 0.1339, the claimed benefit of joint course prediction collapses.","tokens_in":13574,"feed_emoji":"🎓","tokens_out":5492,"duration_ms":55215,"temperature":0.7,"pith_summary":"TRACE is a Transformer model that, given a student's major and past semesters of courses and grades, predicts both the set of courses the student will take next semester and the grade in each course. The paper argues that this joint formulation is the key to accurate grade forecasts: an otherwise identical Transformer that predicts grades alone has a mean absolute error of 0.2496, while TRACE reaches 0.1339, a 46.4% reduction, with mean squared error 3.5 times lower. Predicting courses is treated not as an advising feature but as a training mechanism that forces the model to learn meaningful course representations, in the spirit of auxiliary tasks in pretrained language models. The model also outperforms LSTM-based sequence models, a graph neural network, and XGBoost on ten years of institutional data.","feed_headline":"Forecasting the course list cuts next-semester grade error by 46%","feed_subtitle":"Auxiliary course-set prediction, not the transformer alone, drives the accuracy gain in this ten-year institutional study.","key_machinery":"The load-bearing machinery is a semester-positional encoding combined with a set-based multi-task loss. In TRACE, position is the relative semester number, so all courses in the same semester share the same position vector and the model sees the term as an unordered basket, while temporal order across terms is preserved. Embeddings for courses, grades, and major are learned jointly, and a Transformer encoder-decoder with causal masks over semesters produces, for each upcoming semester, a distribution over all course tokens together with a grade for each. The loss uses KL divergence for the course-set distribution, treating the actual course basket as a normalized probability vector, and MSE for grade regression, with a weighting factor $\\omega \\approx 1$; this avoids the order artifacts of token-level cross-entropy and avoids the per-step assignment cost of Hungarian matching. One-step-ahead training turns each student's history into many input-output pairs, and padding masks keep the loss focused on real courses.","core_discovery":"The central claim is that the structural properties of academic history, particularly courses taken concurrently within a semester, can be modeled directly and that predicting the next semester's course set simultaneously with grades is a powerful regularizer for grade prediction. TRACE encodes every course with a positional vector tied to its semester, making all courses in one term permutation-invariant, and uses a loss $\\mathcal{L}_{\\mathrm{total}} = \\mathcal{L}_{\\mathrm{course}} + \\omega \\mathcal{L}_{\\mathrm{grade}}$, where the course term is a KL divergence between the predicted distribution over the course catalog and the normalized one-hot vector of actually taken courses, and the grade term is mean squared error on the grades in those courses. The result on a held-out 10% of students is MAE 0.1339 and MSE 0.0392, versus 0.2496 and 0.1400 for the grade-only Transformer; the same architecture with the course head removed loses most of its accuracy. The grade-only comparison is the paper's central evidence that the auxiliary course-prediction task, not the Transformer architecture alone, drives the improvement.","pith_inferences":["The paper's ablation shows that removing major from the input slightly improves performance; a natural extension would be to test whether the course-prediction head still helps in settings where course choice is highly constrained, such as single-major programs.","A cheap diagnostic would replace the course-set head with a random auxiliary classification task; if grade MAE still drops sharply, the benefit may be generic regularization rather than course-specific representation learning.","The same semester-positional encoding idea could transfer to other domains where events arrive in temporal bundles, such as healthcare procedures or e-commerce baskets, provided the bundle-composition task is predictive of the regression target."],"forward_implications":["If TRACE's result holds, forcing a grade predictor to also forecast the next semester's course basket is a cheap way to cut grade-forecast error roughly in half, with no additional data.","A semester-level concurrency encoding gives institutions a general multi-course model that can be retrained on their own records without hand-built prerequisite rules.","The reported grade MAE of 0.1339 on a [0,1] scale corresponds to about 0.1568 on a 4.0 GPA scale, roughly half of a typical plus/minus grade step, suggesting early-alert systems can use these forecasts for triage.","The KL-based set loss provides a template for any task that predicts a variable-size unordered set alongside regression targets, not only for course baskets.","Because the model learns course embeddings from enrollment data alone, it can be adapted to a new institution by retraining and recalibration, without institution-specific feature engineering."],"supporting_citations":[{"why":"Supplies the Transformer encoder-decoder architecture, positional encodings, and attention masks that TRACE builds on.","marker":"[27]"},{"why":"Provides the multi-task pretraining analogy: an auxiliary task improves downstream representations, motivating course set prediction as a secondary objective.","marker":"[20]"},{"why":"Serves as the general multiset-prediction loss framework that the KL-based set loss adapts for semester course baskets.","marker":"[34]"},{"why":"Prior work incorporating co-taken courses into grade prediction, which TRACE extends by modeling concurrency at the semester level.","marker":"[10]"},{"why":"A context-aware attentive grade prediction model that motivates the joint course-grade representation.","marker":"[11]"},{"why":"A review documenting the field's reliance on course-specific models and motivating a general multi-course prediction model.","marker":"[25]"},{"why":"A degree-planning model that recommends courses separately from grade prediction, providing the contrast for the joint formulation.","marker":"[13]"}],"fun_headline_variants":["Joint course-grade Transformer reduces grade MAE by 46%","Predicting courses and grades together cuts error nearly half","Course concurrency modeling improves next-semester grade forecasts","Transformer with course-set loss beats LSTM and GNN baselines","Course-aware Transformer halves grade prediction error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The grade-only Transformer baseline was trained under the same hyperparameters and stopping conditions as TRACE, so the large error gap is attributed to the course-prediction head rather than to a tuning imbalance.","fun_headline_variants_meta":{"raw":{"variants":["Joint course-grade Transformer reduces grade MAE by 46%","Predicting courses and grades together cuts error nearly half","Course concurrency modeling improves next-semester grade forecasts","Transformer with course-set loss beats LSTM and GNN baselines","Course-aware Transformer halves grade prediction error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3075,"prompt_tokens":1011,"completion_tokens":2064,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1986}},"tokens_in":627,"tokens_out":2064,"duration_ms":14229,"temperature":1.0,"reasoning_tokens":1986,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:34:34.486350+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OnlyGradesTransformer and TRACE over multiple random seeds with per-model hyperparameter search, reporting learning rate, layer count, and the weighting omega; if a tuned grade-only model reaches TRACE's mean absolute error of about 0.1339, the claimed benefit of joint course prediction collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-task pretraining analogy: an auxiliary task improves downstream representations, motivating course set prediction as a secondary objective."},{"cited_title":"Advances in Neural Information Processing Systems31 (2018)","cited_arxiv_id":null,"evidence_quote":"Serves as the general multiset-prediction loss framework that the KL-based set loss adapts for semester course baskets."},{"cited_title":"International Educational Data Mining Society (2019)","cited_arxiv_id":null,"evidence_quote":"Prior work incorporating co-taken courses into grade prediction, which TRACE extends by modeling concurrency at the semester level."},{"cited_title":"Context-aware Non-linear and Neural Attentive Knowledge-based Models for Grade Prediction","cited_arxiv_id":"2003.05063","evidence_quote":"A context-aware attentive grade prediction model that motivates the joint course-grade representation."},{"cited_title":"Information and Learning Sciences 120(3/4), 208–227 (2019)","cited_arxiv_id":null,"evidence_quote":"A review documenting the field's reliance on course-specific models and motivating a general multi-course prediction model."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence, vol","cited_arxiv_id":null,"evidence_quote":"A degree-planning model that recommends courses separately from grade prediction, providing the contrast for the joint formulation."}],"review_version":1}