Sentiment Analysis in Learning Management Systems Understanding Student Feedback at Scale

Mohammed Almutairi

REVIEW 4 major objections 6 minor 18 references

Sentiment Analysis in Learning Management Systems Understanding Student Feedback at Scale

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims an attention-based LSTM classifies the sentiment of student course comments more accurately than a logistic regression baseline—80% vs 77%—because the network reads feedback sequentially and contextually.

desk verdict Standard LSTM+attention recipe on RateMyProfessor data, undermined by an ambiguous ground-truth label and inconsistent confusion matrices. read the letter →

arxiv 2506.05490 v1 pith:45S7WTRQ submitted 2025-06-05 cs.HC cs.AIcs.LG

Mohammed Almutairi This is my paper

classification cs.HCcs.AIcs.LG

keywords sentimentanalysisstudentfeedbacklearningmanagementsystemsLSTMattentionmechanismlogisticregressiononlineeducationopinionmining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that automated sentiment analysis can be embedded in a learning management system to recover some of the emotional signal instructors lose when teaching moves online and non-verbal cues disappear. It reports a concrete pipeline built from public student-comment data, text preprocessing, class balancing, and a deep neural network, and shows that the network outperforms a logistic regression baseline on accuracy, precision, recall, F1, and AUC. The practical stakes are that educators could receive continuous, aggregated readings of whether student feedback is positive or negative, instead of relying on sparse verbal communication. The paper also identifies a boundary of the model: both tested classifiers misread explicit negation, so the proposed method is not a complete substitute for human reading.

What carries the argument

The machinery is the proposed deep network, called the RNN model in the paper: a word-embedding layer that turns words into dense vectors, a bidirectional LSTM that reads each comment forward and backward to capture long-range dependencies, an attention layer that up-weights sentiment-bearing words, and a sigmoid output that converts scores to positive/negative probabilities. Around it sits a preprocessing pipeline—TF-IDF for vectorization, chi-squared statistics to select a 5,000-word vocabulary from the full 17,583-word vocabulary, SMOTE to balance classes, plus stopword removal and lemmatization. The baseline, logistic regression on the same features, is the counterfactual: its word-independent weighting is what the RNN is claimed to improve upon.

What would settle it

Inspect the prepared dataset and compare each comment's assigned sentiment label to the per-comment rating recorded with that same comment; if labels were instead assigned from the professor-level average rating, many individual comments will be mislabeled. Re-running the same train/test split with labels derived from each comment's own rating would settle whether the reported 80% vs 77% accuracy gap is real.

Watch

Extended reading notes

Core claim

The central claim is that an RNN-based sentiment classifier—implemented as a word embedding layer, a bidirectional LSTM, an attention mechanism, and a sigmoid output—understands the sentiment of student course feedback better than a TF-IDF logistic regression baseline, and is a viable component to embed in a learning management system. On the public professor-review dataset, the paper reports the RNN outperforming the baseline on every metric: accuracy 80% vs 77%, precision 83% vs 77%, recall 85% vs 76%, F1 0.84 vs 0.77, and AUC 0.88 vs 0.86. The paper attributes the gap to sequential and contextual processing: the RNN interprets words in relation to their neighbors, while logistic regression treats each word largely independently. A qualitative sensitivity analysis on constructed sentences adds a second claim: the RNN tracks mixed sentiments better than the baseline, yet neither model correctly handles negation such as "not engaging and informative."

Load-bearing premise

The load-bearing premise is that each comment's sentiment label is derived from the rating attached to that individual comment, not from the professor's overall average rating; the paper never says which field generated the labels, and if the average was used, individual comments are mislabeled and the evaluation is invalid.

Editorial extensions

If this is right

Embedded in an LMS, the model could produce course-level sentiment summaries for instructors in near real time, turning thousands of comments into an interpretable signal.
The reported gains across precision, recall, F1, and AUC imply that sequence-aware classifiers are worth their added complexity for this task, at least on public professor-review feedback.
The qualitative failures on negation imply that a deployed system should either flag sentences containing negations for human review or be paired with a model that handles them.
The paper presents the public multi-institution data as a proxy that makes the approach transferable to LMS settings, where feedback is often private and restricted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the labeling ambiguity is resolved and each comment is labeled by its own rating, a natural next experiment would be to fine-tune the same architecture on actual LMS comments, where feedback is formative and about course logistics rather than professor reputation; the current dataset is retrospective and professor-focused.
The failure on negation suggests a cheap diagnostic for any future sentiment system in education: test with minimal pairs like "engaging and informative" versus "not engaging and informative" before trusting the model with student feedback.
The paper proposes transformer-based contextual embeddings as future work; a direct test of that suggestion is to swap the embedding layer for a contextual encoder and compare on the same qualitative sentences, especially the one where both models misread negation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Reading this one was quick. The genuinely new thing is the exact combination: a BiLSTM+attention model on RateMyProfessor comments, compared against logistic regression. Nothing else is new; the recipe is textbook TF-IDF, chi-square selection, SMOTE, BiLSTM. The reported headline (80% vs 77% accuracy) would be a modest but believable result if the evaluation were clean. To the paper's credit, it is clearly written, uses an external public dataset, and describes preprocessing steps concretely enough to reproduce. That is real evidence of effort.

The soft spots are real and one is load-bearing. The paper never states whether the ground-truth label comes from `star_rating`, the professor-level average, or `students_star`, the per-comment rating. Section 2.1 defines both, Table 1 says star_rating is the average over all comments, and Section 2.3 only says "the star rating." If the aggregate average was used, every comment for a professor gets the same label, which is systematic correlated label noise; the model could be learning professor reputation rather than comment sentiment, and the RNN-vs-LR gap becomes uninterpretable. That alone is enough to invalidate the central claim as submitted. Adding to it, the two confusion matrices imply test sets of size 4002 (LR) and 3996 (RNN) under the same stated 80/20 split, with identical false positives (422). Those numbers cannot both be right. There are also no error bars or significance tests, so even ignoring the matrix inconsistency, a 3-point accuracy gap is not established.

The conclusion overstates by claiming the model was integrated into an LMS; no integration is described anywhere in the methods. The qualitative analysis is eight hand-built sentences, which is fine for illustrating behavior but is not evidence.

Who is this for? Someone wanting a worked example of a standard sentiment-analysis pipeline on RMP data. A serious referee could ask for a corrected re-analysis, so I wouldn't desk-reject the idea. But in this version the numbers do not reliably support the headline. I'd send it back for major revision requiring: (1) explicit definition of the label field and re-analysis with per-comment labels; (2) consistent confusion matrices; (3) error bars or bootstrap CIs; and (4) removal of the LMS-integration claim. If the author does that, the comparison could become publishable as an application note. As is, not enough.

Referee Report

4 major / 6 minor

Summary. The paper proposes a sentiment analysis pipeline for student feedback in learning management systems, using the RateMyProfessor dataset. It applies TF-IDF vectorization, chi-squared feature selection, SMOTE balancing, and compares a logistic regression baseline with an LSTM plus attention deep model. The reported results claim the RNN is superior across all metrics, with 80% accuracy versus 77% for logistic regression, and the paper concludes that the deep model better captures sequential and contextual sentiment in student comments.

Significance. If the evaluation were sound, the paper would offer a modest, reproducible comparison of a deep sequence model against a linear baseline on a public educational-feedback dataset, with a useful qualitative sensitivity analysis on constructed sentences. The use of a public dataset and a clear baseline is a strength. However, the central claim of RNN superiority is currently unsupported because of an ambiguous ground-truth label definition, an inconsistency in the reported test-set sizes of the two confusion matrices, and a lack of any statistical significance testing. The paper also frames the work as LMS integration while using RateMyProfessor data, so the ecological validity of the 'LMS' claim is limited.

major comments (4)

[Section 2.1, Table 1, Section 2.3] The ground-truth label source is never specified. Table 1 distinguishes 'star_rating' (the professor-level average overall quality) from 'students_star' (the rating given by each individual student). Section 2.3 refers only to 'the star rating' as the indicator of dataset distribution. If the aggregate 'star_rating' is used to label each individual comment, then all comments for a given professor share the same label, and the model may be learning professor reputation rather than the sentiment of the specific comment. This is systematic, text-correlated label noise that would invalidate the comparison in Section 3.1. The authors must state explicitly which field generated the 'sentiment' labels and, if the aggregate field was used, rerun the evaluation using per-comment 'students_star' labels or provide a justified alternative.
[Section 3.1, Figures 7 and 8] The reported confusion matrices are internally inconsistent with the stated 80/20 split. The logistic regression matrix totals 4002 instances (1387 + 1718 + 422 + 475), while the RNN matrix totals 3996 instances (2184 + 1009 + 422 + 381), and both report exactly 422 false positives. Under the same test-set fraction, both models should be evaluated on the same number of test instances. This indicates the two evaluations were not performed on the same, consistently defined test set, so the accuracy, precision, recall, and F1 values in Table 2 are not directly comparable. The authors must rerun both models on the same held-out test set and report consistent totals.
[Section 2.4 and Section 2.5] The paper does not clarify whether SMOTE is applied before or after the train/test split. Section 2.4 describes applying SMOTE to the dataset and Figure 3 shows class distributions before and after SMOTE, while Section 2.5 describes the 80/20 split only afterward. If SMOTE is applied to the full dataset prior to splitting, synthetic minority-class examples will appear in the test set, artificially inflating the reported metrics. The authors must state explicitly that SMOTE was applied only to the training portion and that the test set consists exclusively of real, held-out examples.
[Section 2.5 and Table 2] No error bars, confidence intervals, or significance tests are reported for any metric. The central claim of Section 3.1 that the RNN is 'superior across all metrics' rests on a single 80/20 split, where the accuracy difference is only 3 percentage points on a test set of roughly 4000 instances. Such a difference can easily arise from sampling noise. The authors should report multiple runs or cross-validation and apply a paired significance test, such as McNemar's test or a bootstrap interval, to support the claim of superiority.

minor comments (6)

[Abstract] Several grammatical errors and typos appear in the abstract, such as 'in person learning traditional', 'The absent of non-verbal communication', and 'has impacted the teacher-student especially in non-verbal communication'. These should be corrected throughout.
[Section 2.2] The paper states that the vocabulary consists of 17,583 unique words and the 'comment' feature contains 19,993 sentences, but it does not report the total number of comments or the class distribution counts corresponding to Figure 3. This information is needed to interpret the SMOTE balancing and the reported test-set sizes.
[Section 2.4] The oversampling ratio or target class distribution after SMOTE is not specified. Readers cannot assess how much synthetic data was added or whether the resulting distribution is sensible.
[Section 2.7] Hyperparameters for the deep model are not reported, including embedding dimension, number of LSTM units, dropout, learning rate, batch size, and number of training epochs. Without these, the comparison to logistic regression is not reproducible.
[Section 3.2] In the qualitative analysis, the paper claims both models failed to classify sentence 8 as negative despite 'not', but Table 3 lists sentence 8 as 'The lecture was not engaging and informative.' The discussion would be clearer if the exact intended sentiment of each constructed sentence were explicitly stated before interpreting model outputs.
[References] The reference list contains inconsistent formatting, including missing venue information for reference [1] and a dissertation reference [14] that appears unrelated to the cited claim about sentiment analysis in education. These should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RNN-vs-LR comparison is a standard supervised-learning evaluation on an external dataset.

full rationale

The paper's central claim—that an LSTM-with-attention model outperforms logistic regression on RateMyProfessor comment sentiment—is supported by a held-out test evaluation. The sentiment labels come from the external RateMyProfessor star ratings, not from the model being evaluated; the text features are TF-IDF and chi-square-selected word representations; and the 80/20 train/test split means the reported accuracy, precision, recall, F1, and AUC are out-of-sample predictions. No equation in the paper defines the target metric in terms of a fitted parameter, and no load-bearing premise is imported from a self-citation; the citations to Lipton and Molnar are background explanations of RNN capabilities and interpretability rather than evidence that the evaluation is true. The paper's ambiguities—such as not stating whether the aggregate 'star_rating' or per-comment 'students_star' is the ground-truth label and the inconsistent confusion-matrix totals between the two models—are correctness and validity concerns, not circularity, because they do not make the comparison true by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central comparison rests on two unverified domain assumptions: that star ratings correctly encode comment sentiment, and that RateMyProfessor comments proxy LMS feedback. It also introduces several hand-picked choices (vocabulary size, SMOTE ratio, hyperparameters) whose values are not reported. No new entities are postulated.

free parameters (4)

star-rating to sentiment mapping
Section 2.1 and 2.3: the paper mentions positive/neutral/negative but does not give the exact star thresholds used to label each comment; this mapping determines the ground truth for every metric.
chi-square vocabulary size = 5000 words
Section 2.2: the vocabulary is truncated to 5000 words based on chi-square scores; the threshold is chosen by hand and affects model input.
SMOTE oversampling ratio
Section 2.4: SMOTE is applied to balance classes, but the target ratio and the feature space where resampling happens are not stated.
Deep network hyperparameters
Section 2.7: embedding dimension, LSTM hidden size, number of layers, attention variant, learning rate, epochs, and batch size are all unspecified, yet they determine the reported 80% vs 77% comparison.

assumptions (4)

domain assumption RateMyProfessor comments are a valid proxy for LMS feedback
Section 2.1 states the dataset 'serves our purpose' as a proxy for broader student experiences; the LMS integration claim in the conclusion depends on this transfer.
domain assumption Star ratings accurately reflect the sentiment of the comment text
Comments are labeled via ratings; the paper does not validate the rating-text sentiment alignment.
domain assumption The binary positive/negative setup is consistent with the three-class (positive/neutral/negative) framing
Section 2.3 introduces a neutral class, but Section 2.7 describes only a sigmoid binary output; no explanation of how neutral comments are handled.
standard math Standard feature engineering (TF-IDF, chi-square, SMOTE) preserves sentiment signal
These are standard methods; no guarantee is required, but the paper relies on them without validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Sentiment Analysis in Learning Management Systems Understanding Student Feedback at Scale." pith.science (2026). https://pith.science/paper/45S7WTRQ

@misc{pith2026250605490,
  author       = {Pith},
  title        = {Pith review of: Sentiment Analysis in Learning Management Systems Understanding Student Feedback at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/45S7WTRQ}},
  note         = {Machine review of arXiv:2506.05490}
}

read the original abstract

During the wake of the Covid-19 pandemic, the educational paradigm has experienced a major change from in person learning traditional to online platforms. The change of learning convention has impacted the teacher-student especially in non-verbal communication. The absent of non-verbal communication has led to a reliance on verbal feedback which diminished the efficacy of the educational experience. This paper explores the integration of sentiment analysis into learning management systems (LMS) to bridge the student-teacher's gap by offering an alternative approach to interpreting student feedback beyond its verbal context. The research involves data preparation, feature selection, and the development of a deep neural network model encompassing word embedding, LSTM, and attention mechanisms. This model is compared against a logistic regression baseline to evaluate its efficacy in understanding student feedback. The study aims to bridge the communication gap between instructors and students in online learning environments, offering insights into the emotional context of student feedback and ultimately improving the quality of online education.

Figures

Figures reproduced from arXiv: 2506.05490 by the authors.

**Figure 1.** Bridging the Non-Verbal Communication Gap in Online Education through Sentiment Analysis Integration in Learning [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 3.** Distribution of the dataset’s classes before and after ap [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Architecture of the Proposed Sentiment Analysis Model. This figure illustrates our deep neural network design, highlighting [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: ROC curve displaying the performance of the sentiment [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Confusion matrix illustrating the performance of the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 9.** Figure 9: Sensitivity analysis of sentence variations using the Recur [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

[1]

Felipe Child, Marcus Frank, and Jonat Law. 2023. What do higher education students want from online learning? (2023)

work page 2023
[2]

Chaitali Choudhary, Inder Singh, and Manoj Kumar. 2023. SARWAS: Deep ensemble learning techniques for sentiment based recommendation system. Expert Systems with Applications 216 (2023), 119420

work page 2023
[3]

Yuhuan Cui, Zezhong Ma, Liya Wang, Aimin Yang, Qiumei Liu, Shanshan Kong, and Huifang Wang. 2023. A survey on big data-enabled innovative online education systems during the COVID-19 pandemic. Journal of Innovation & Knowledge 8, 1 (2023), 100295

work page 2023
[4]

Simon L Evans, Rosalind Jones, Erkan Alkan, Jaime Simão Sichman, Amanul Haque, Francisco Bráulio Silva de Oliveira, Davoud Mougouei, et al

work page
[5]

Kutay Güler. 2023. Structuring knowledge-building in online design education. International Journal of Technology and Design Education 33, 3 (2023), 1055–1086

work page 2023
[6]

Jibo He. 2020. Big Data Set from RateMyProfessor.com for Professors’ Teaching Evaluation. Mendeley Data. https://doi.org/10.17632/fvtfjyvw7d.2

work page doi:10.17632/fvtfjyvw7d.2 2020
[7]

Sarah Hodges. 2023. The Eyes Have It: The importance of eye contact in education and strategies to improve teaching in the virtual environment. eLearn 2023, 2 Special Issue (2023)

work page 2023
[8]

Zachary Chase Lipton. 2015. A Critical Review of Recurrent Neural Networks for Sequence Learning. CoRR abs/1506.00019 (2015). arXiv:1506.00019 http://arxiv.org/abs/1506.00019

arXiv 2015

Show all 18 references

[9]

Qi Liu, Matt J Kusner, and Phil Blunsom. 2020. A survey on contextual embeddings. arXiv preprint arXiv:2003.07278 (2020)

2020 arXiv
[10]

Rommel Pascual Manzon. 2023. SENTIMENTS ANALYSIS ON THE PERFORMANCE OF SECONDARY SCHOOL ENGLISH TEACHERS DURING ONLINE CLASSES. EPRA International Journal of Research and Development (IJRD) 8, 6 (2023), 27–30

2023
[11]

Christoph Molnar. [n. d.]. Interpretable machine learning: A guide for making black box models explainable
[12]

Natural Language Toolkit. 2023. NLTK WordNet Lemmatizer Source Code. Available: https://www.nltk.org/_modules/nltk/stem/wordnet.html

2023
[13]

Natural Language Toolkit. 2023. stopwords. Available: https://www.nltk.org/search.html?q=stopwords

2023
[14]

NIDHI NIRBHIK. 2023. INTEGRATING TEXT AND EMOTICONS FOR DETECTING EXTREMIST AFFILIATIONS ON TWITTER USING DEEP LEARNING . Ph. D. Dissertation

2023
[15]

Ž Vujović et al. 2021. Classification model evaluation metrics. International Journal of Advanced Computer Science and Applications 12, 6 (2021), 599–606

2021
[16]

Chengcheng Xu, Xiabing Zheng, and Feng Yang. 2023. Examining the effects of negative emotions on review helpfulness: The moderating role of product price. Computers in Human Behavior 139 (2023), 107501

2023
[17]

Tan Zi Yi, Kazuya Shide, Hirotake Kanisawa, Naoto Mine, Kazuki Otsu, Yohei Koga, and Shunsuke Someya. 2023. A study on the effectiveness of non-verbal teaching materials in construction accidents prevention education: through an International comparative experiment. Journal of...

2023
[2023]

Human Behavior and Emerging Technologies 2023 (2023)

The emotional impact of COVID-19 news reporting: A longitudinal study using natural language processing. Human Behavior and Emerging Technologies 2023 (2023)

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Felipe Child, Marcus Frank, and Jonat Law. 2023. What do higher education students want from online learning? (2023)

work page 2023

[2] [2]

Chaitali Choudhary, Inder Singh, and Manoj Kumar. 2023. SARWAS: Deep ensemble learning techniques for sentiment based recommendation system. Expert Systems with Applications 216 (2023), 119420

work page 2023

[3] [3]

Yuhuan Cui, Zezhong Ma, Liya Wang, Aimin Yang, Qiumei Liu, Shanshan Kong, and Huifang Wang. 2023. A survey on big data-enabled innovative online education systems during the COVID-19 pandemic. Journal of Innovation & Knowledge 8, 1 (2023), 100295

work page 2023

[4] [4]

Simon L Evans, Rosalind Jones, Erkan Alkan, Jaime Simão Sichman, Amanul Haque, Francisco Bráulio Silva de Oliveira, Davoud Mougouei, et al

work page

[5] [5]

Kutay Güler. 2023. Structuring knowledge-building in online design education. International Journal of Technology and Design Education 33, 3 (2023), 1055–1086

work page 2023

[6] [6]

Jibo He. 2020. Big Data Set from RateMyProfessor.com for Professors’ Teaching Evaluation. Mendeley Data. https://doi.org/10.17632/fvtfjyvw7d.2

work page doi:10.17632/fvtfjyvw7d.2 2020

[7] [7]

Sarah Hodges. 2023. The Eyes Have It: The importance of eye contact in education and strategies to improve teaching in the virtual environment. eLearn 2023, 2 Special Issue (2023)

work page 2023

[8] [8]

Zachary Chase Lipton. 2015. A Critical Review of Recurrent Neural Networks for Sequence Learning. CoRR abs/1506.00019 (2015). arXiv:1506.00019 http://arxiv.org/abs/1506.00019

arXiv 2015