Virtual teaching assistant for undergraduate students using natural language processing & deep learning

Annajiat Alim Rasel; Baktiar Kabir Joy; Md. Nuruzzaman; Sadman Jashim Sakib; Zahin Rydha

REVIEW 4 major objections 4 minor 20 references

Virtual teaching assistant for undergraduate students using natural language processing & deep learning

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VTA-bot is a virtual teaching assistant that classifies first-year Python course questions by intent and returns curated responses; the paper reports its trained model reaching 95% accuracy.

desk verdict The 95% accuracy is a training-set number, so the central claim of a working VTA-bot is unsupported; the dataset description is the only useful part. read the letter →

arxiv 2411.09001 v1 pith:J2CD52R6 submitted 2024-11-13 cs.CY

Sadman Jashim Sakib , Baktiar Kabir Joy , Zahin Rydha , Md. Nuruzzaman , Annajiat Alim Rasel This is my paper · ORCID

classification cs.CY

keywords virtualteachingassistantchatbotintentclassificationnaturallanguageprocessingdeeplearningPythonprogrammingeducationundergraduatestudents

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces VTA-bot, a virtual teaching assistant aimed at first-year university students learning the basics of Python. The authors' central claim is that a chatbot trained on a purpose-built dataset of 1,006 student questions, organized into 214 intent tags, can automatically answer a variety of fundamental programming inquiries by classifying each question's intent and retrieving a matching response. The motivation is that many institutions' online education systems leave new programming students without timely help, hurting retention. The paper reports that its feed-forward neural network reaches 95% accuracy and argues the same architecture can be retrained for other course material. If this holds, students would get round-the-clock assistance and instructors would be freed from repetitive questions.

What carries the argument

The load-bearing mechanism is an intent-classification feed-forward neural network with two hidden layers and a Softmax output, fed by a bag-of-words representation of a student's question. The bag of words is as long as the vocabulary collected from all training patterns and marks each word as present (1) or absent (0) after tokenization and stemming. The dataset is organized as a hierarchy of intent, topic, tag, question, and response, so a predicted intent directly retrieves a stored answer; this fixed mapping is what lets the bot answer without generating text. The network's training loop over 1,000 epochs with batch size 8 is the part that is claimed to reach 95% accuracy.

What would settle it

Give the trained VTA-bot 200 fresh Python questions written by first-year students who did not take part in dataset construction, have a human judge mark each returned response as on- or off-intent, and check whether the correct-intent rate approaches 95% rather than the much lower rates the paper's baseline classifiers show on the full dataset.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a supervised intent-classification model can serve as the core of a virtual teaching assistant for a first-year Python course. Student questions are tagged by intent, and each intent tag holds similar question patterns plus one curated response. After preprocessing (case folding, punctuation and stopword removal, stemming, lemmatization) each pattern is turned into a bag of words; the neural network maps that bag to a probability distribution over intents, and the highest-probability intent selects the response. The authors report that this pipeline, trained on their new dataset, drives training accuracy to 95% and loss down to 0.0007, and they position the system as a first step toward a fuller virtual teaching assistant that can handle course-related concerns in real time.

Load-bearing premise

The central claim depends on the assumption that the reported 95% training accuracy also holds for new student questions the model has never seen.

Editorial extensions

If this is right

First-year students can receive immediate answers to common Python questions without waiting for an instructor or teaching assistant.
Course instructors can offload repetitive questions and spend more time on students' harder, individual problems.
The same neural network pipeline can be pointed at a new dataset of patterns and responses to serve a different course or topic.
The released primary dataset gives other groups a starting point for building or benchmarking programming-education chatbots.
Future work can add more patterns per tag to make the intent classes more balanced, which the paper's classifier comparisons suggest raises accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A deployment-oriented reading implies the 95% accuracy figure must be tested on questions the model has not seen during training; the paper reports accuracy over its training run and does not describe a held-out test set.
The same intent-classification pipeline could generalize to other first-year programming courses or other STEM subjects if comparable pattern-response datasets are assembled.
The dataset's long tail (18.7% of tags have two or fewer questions) is the natural place to add more phrasings; the paper's own classifier comparisons suggest accuracy rises with more patterns per tag.
One editorial note: the preprocessing section cites a reference marker [34] that is not present in the reference list, so the pipeline description is missing one provenance link.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the 95% accuracy in Fig. 10 is a training-loop number, not a test-set result, so the paper's core claim that VTA-bot can answer unseen student questions is unsupported. The authors do some things well—the dataset is described carefully, with a sensible hierarchy and TA validation, and the system architecture is clear—but the evaluation is in-sample and the 'refactored datasets' only make selection bias worse.

The dataset itself may be the most useful artifact, but it isn't released, so we're left with a description. The literature review is honest about prior chatbots (PythonBot, ProgBot), so the novelty claim is appropriately modest: this is an incremental application with one new dataset. Fine as a systems report, not as a research claim.

The serious problem is in the RESULT section: loss and accuracy are computed 'in each 100 epochs' and the text says 'loss is the error over the training set.' Both curves are training curves. The paper even says two data sets are required—training and test—but never reports a test result. With only 1,006 questions across 214 tags, the model can easily memorize bag-of-words patterns; 95% training accuracy tells us nothing about generalization. The earlier classifier comparison is also done on the entire dataset, and the 'refactoring' step drops low-frequency tags, which inflates apparent accuracy. This is not harmless; it directly undermines the abstract's claim that the bot 'can automatically respond' to student inquiries.

That said, the paper is readable, the authors are transparent about their pipeline, and the problem they address is real. But as a result, it doesn't hold up. I'd treat it as a preliminary systems note, not a peer-reviewed research contribution.

For you: skim it if you're building a similar chatbot and want an example of what not to do in evaluation. Otherwise skip. I would not send it to a serious referee as-is; it needs a proper held-out evaluation and dataset release before it's worth anyone's time.

Referee Report

4 major / 4 minor

Summary. The paper proposes VTA-bot, a virtual teaching assistant chatbot for a first-year undergraduate Python programming course. The authors construct a labeled dataset of 1,006 questions with 214 intent tags and 254 responses, and build a feed-forward neural network with bag-of-words features to classify student queries into intents. The central claim is that the trained model reaches 95% accuracy and can automatically respond to a variety of student inquiries. The paper also compares four classical classifiers on the full dataset and on progressively 'refactored' subsets, and describes a Flask-based web integration.

Significance. The problem addressed is real: first-year programming students in online or blended courses often lack timely help, and an automated teaching assistant could improve retention. The authors have put in tangible effort in collecting and tagging a domain-specific dataset and in building a working prototype with a web interface. If the 95% accuracy were demonstrated on unseen student queries, the paper would be a useful practical contribution. However, the evaluation is in-sample: the neural-network accuracy is measured on the training data, and the classifier comparisons use the entire dataset without any train/test split. The dataset itself is not made available, and the 'refactored' subsets are created by dropping low-support tags, which inflates accuracy. As presented, the manuscript does not establish that VTA-bot generalizes to new student questions, so the central claim is unsupported.

major comments (4)

[Result, Fig.10] The reported 95% accuracy is a training-set number, not a measure of generalization. The Result section states that loss and accuracy were computed 'in each 100 epochs' during training and that 'throughout the training loop... the accuracy is increasing'; Fig.10 is therefore the training accuracy curve. No held-out test set, validation split, or cross-validation is described anywhere in the paper, even though the Prototype Development section correctly notes that both a training set and a test set are required. The abstract's claim that the VTA-bot 'can automatically respond to a variety of inquiries' is a claim about unseen student questions, and the 95% figure does not support it. This is the central load-bearing result of the paper and it is currently an in-sample fit.
[Fitting dataset on basic classifiers, Fig.4] The classifier comparison in Fig.4 is performed without any train/test split. The text says 'After training and evaluating the models with the entire dataset' and then repeats the same procedure on datasets created by 'omitting tags with less training patterns' and using a 'more refactored dataset.' The accuracy improvements from 21.42% (Naive Bayes) and 62.24% (Logistic Regression) on the full data to 73.91% and 82.60% on the 'more refactored' data are exactly what is expected when the smallest and hardest classes are removed and when the same data are used for both fitting and scoring. These numbers therefore do not establish that the dataset is 'balanced' or that the neural model will reach 'more than 90 percent' accuracy on new student queries.
[Dataset for VTA-bot, Data Hierarchy and Distribution] The dataset has only 1,006 questions across 214 tags, and 59.3% of tags contain four or fewer questions (18.7% with two or fewer, 18.2% with three, 22.4% with four). With a bag-of-words representation and a feed-forward network, the model can easily memorize these small tag-specific vocabularies; the reported training loss of 0.0007 and 95% training accuracy are fully consistent with memorization. The paper provides no analysis, such as a held-out evaluation or an error analysis on genuinely unseen questions, to show that the model generalizes beyond exact keyword matches. Without such evidence, the accuracy claim cannot be credited.
[Result, early stopping] The Result section states that 'The most typical strategy for avoiding over-fitting is early stopping' and describes separating data into training and validation sets and computing the validation error regularly, but no validation split or validation error is reported anywhere in the paper. This paragraph appears to describe a textbook procedure rather than the experiment actually performed. The reader cannot tell whether early stopping was applied, and the 95% accuracy figure is not tied to any validation or test criterion.

minor comments (4)

[Fitting dataset on basic classifiers] There is a stray, incomplete sentence in this section: 'In important evaluation metrics. It elegantly summarizes the predictive performance of a model by combining precision and recall.' This appears to be a fragment from a discussion of F1-score and should be rewritten as a complete, coherent paragraph.
[References] The reference numbering is inconsistent: the text cites [34] for NLTK, but the reference list contains only 20 numbered entries. Please renumber all citations and ensure every in-text citation has a corresponding entry.
[Dataset for VTA-bot] The paper says it offers a primary dataset, but no data availability statement, repository link, or supplementary file is provided. Without access to the dataset, independent replication and reuse are impossible.
[Prototype Development, Methodology] Figure 6, described as 'VTA-bot assisting a student,' is referenced but the figure itself is not visible in the text; please confirm all figures are included and correctly reproduced.

Circularity Check

2 steps flagged · score 6.0 of 10

The 95% accuracy is a training-loop number; without a held-out test split the generalization claim reduces to an in-sample fit.

fitted input called prediction [RESULT section (Fig. 10 and Fig. 11)]
"We set the batch size to 8 and the number of epochs to 1,000. Then we calculated the loss and accuracy in each 100 epochs. ... Loss is the error over the training set. ... Throughout the training loop, in every epoch the loss is decreasing and the accuracy is increasing. In the Fig.10, we can see the accuracy increased to 95%."

The 95% figure is computed inside the training loop on the same data that fit the network, as the paper's own definition ('Loss is the error over the training set') and the training-loop timing ('calculated the loss and accuracy in each 100 epochs') make clear. The abstract claims the chatbot 'can automatically respond to a variety of inquiries,' which is a generalization claim about unseen student questions, but no held-out test set or cross-validation accuracy is reported. Therefore the headline accuracy is a fitted value (training error) renamed as system performance; it reduces by construction to a measure of memorization, not prediction.
fitted input called prediction [Fitting dataset on basic classifiers section (Fig. 4)]
"After training and evaluating the models with the entire dataset, the accuracy for Naive Bayes Classifier is 21.42 percent, Decision Tree Classifier is 58.67 percent, Linear Support Vector Machine is 50 percent, and Logistic Regression is 62.24 percent. ... we omitted certain tags with less training patterns and refactored data while preserving the patterns of tags with at least 10 patterns. On rerunning these classifiers on the refactored data, the accuracy of each classifier improves."

The classifiers are 'trained and evaluat[ed] with the entire dataset'—no train/test split—and the refactoring step removes tags with few patterns, which are precisely the hard, sparse classes. The resulting accuracy is therefore an in-sample fit on a selected subset, and the paper explicitly uses it to predict the neural net ('we can anticipate that the neural model of our prototype will have an accuracy of more than 90 percent'). The 'prediction' is forced by the same in-sample evaluation; it contains no independent test information.

full rationale

The derivation chain for the central performance claim is circular in its evaluation: the reported 95% accuracy is a training-set metric, not a measure of performance on unseen student questions. The RESULT section explicitly defines loss as 'the error over the training set' and describes measuring accuracy 'in each 100 epochs' throughout the training loop, while providing no validation or test accuracy. The classifier comparison in the earlier section is likewise conducted 'with the entire dataset' and further refines the dataset by dropping low-pattern tags, making the reported accuracies in-sample fits on an easier, selected subset. The paper's own methodology even states that a training set and a set to be predicted (test set) are both required, but no test result is ever supplied. There is no self-citation or imported uniqueness problem here; the dataset itself is a genuine contribution. Nevertheless, the abstract's claim that VTA-bot 'can automatically respond to a variety of inquiries' depends on generalization, and the only quantitative support offered is a training-loop number that reduces by construction to a measure of fit, not prediction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The only explicit free parameter is the post-hoc refactoring threshold. The axioms are the standard assumptions of bag-of-words intent classification and the unvalidated representativeness of the hand-built dataset.

free parameters (1)

refactoring threshold (minimum patterns per tag) = 10 patterns
Tags with fewer than 10 patterns were removed to increase classifier accuracy; this choice is post hoc and directly inflates reported performance.

assumptions (3)

domain assumption Bag-of-words representation preserves enough information for intent classification.
The model converts sentences to binary word vectors, discarding word order; this standard assumption underlies the entire NLP pipeline (Fig.8).
domain assumption Intent classification is a supervised problem with a fixed finite intent set.
The paper defines intents I = {greetings, method, class, ...} and learns a mapping from question patterns to intents (PROBLEM STATEMENT).
domain assumption The hand-labeled dataset accurately reflects real student queries.
The dataset was created by the authors with some TA feedback, but no inter-annotator agreement or external validation is reported; the chatbot's usefulness depends on this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Virtual teaching assistant for undergraduate students using natural language processing & deep learning." pith.science (2026). https://pith.science/paper/J2CD52R6

@misc{pith2026241109001,
  author       = {Pith},
  title        = {Pith review of: Virtual teaching assistant for undergraduate students using natural language processing & deep learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2CD52R6}},
  note         = {Machine review of arXiv:2411.09001}
}

read the original abstract

Online education's popularity has been continuously increasing over the past few years. Many universities were forced to switch to online education as a result of COVID-19. In many cases, even after more than two years of online instruction, colleges were unable to resume their traditional classroom programs. A growing number of institutions are considering blended learning with some parts in-person and the rest of the learning taking place online. Nevertheless, many online education systems are inefficient, and this results in a poor rate of student retention. In this paper, we are offering a primary dataset, the initial implementation of a virtual teaching assistant named VTA-bot, and its system architecture. Our primary implementation of the suggested system consists of a chatbot that can be queried about the content and topics of the fundamental python programming language course. Students in their first year of university will be benefited from this strategy, which aims to increase student participation and involvement in online education.

Figures

Figures reproduced from arXiv: 2411.09001 by the authors.

**Figure 3.** FIGURE 3 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** FIGURE 4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: FIGURE 5 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: FIGURE 6 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: FIGURE 7 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: FIGURE 8 [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

**Figure 9.** Figure 9: FIGURE 9 [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

[1]

Python-bot: A chatbot for teaching python programming.,

C. W. Okonkwo and A. Ade -Ibijola, “Python-bot: A chatbot for teaching python programming.,” Engineering Letters, vol. 29, no. 1, 2020

work page 2020
[2]

Elabmate: A tool for delivering programming courses effectively,

R. Ahmed, “Elabmate: A tool for delivering programming courses effectively,” International Journal of Advanced Corporate Learning (iJAC), vol. 5, no. 3, pp. 6–11, 2012

work page 2012
[3]

A Virtual Teaching Assistant for Personalized Learning

L. Benedetto, P. Cremonesi, and M. Parenti, “A virtual teaching assistant for personalized learning,” arXiv preprint arXiv:1902.09289, 2019

work page Pith review arXiv 1902
[4]

Eliza—a computer program for the study of natural language communication between man and machine,

J. Weizenbaum, “Eliza—a computer program for the study of natural language communication between man and machine,” Communications of the ACM, vol. 9, no. 1, pp. 36–45, 1966

work page 1966
[5]

Using chatbots as smart teaching assistants for first-year engineering students,

S. Abdelhamid and A. Katz, “Using chatbots as smart teaching assistants for first-year engineering students,” in 2020 First-Year Engineering Experience, 2020

work page 2020
[6]

Intelligent chat r obot in digital campus based on deep learning,

Y. Liang, Y. Yu, and W. Ouyang, “Intelligent chat r obot in digital campus based on deep learning,” in Journal of Physics: Conference Series, IOP Publishing, vol. 1629, 2020, p. 012 079

work page 2020
[7]

Framework for educational domain -based multichatbot communication system,

Z. Memon, H. Aghian, M. S. Sarfraz, et al., “Framework for educational domain -based multichatbot communication system,” Scientific Programming, vol. 2021, 2021

work page 2021
[8]

Implementation of chatbot that teach programming language,

M. A. Jayakumar Sadhasivam, A. B. Amitava Mazumdar, B. J.M, and V. Kumar, “Implementation of chatbot that teach programming language,” Xi’an University of Architecture & Technology, vol. 12, 2020

work page 2020

Show all 20 references

[9]

Artificial intelligence in mathematics education: An empirical study of using chatbot in teaching and learning mathematics at vietnamese high schools,

N. N. P. Anh and H. T. Ngan, “Artificial intelligence in mathematics education: An empirical study of using chatbot in teaching and learning mathematics at vietnamese high schools,” 5th ASIA PACIFIC International Modern Sciences Congress, 2021. FIGURE 10: Accuracy Curve FIGURE...

2021
[10]

Educational assistance bot,

Roy, D. Singh, and S. Sahana, “Educational assistance bot,” in Journal of Physics: Conference Series, IOP Publishing, vol. 1797, 2021, p. 012 062

2021
[11]

Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assistant,

Q. Wang, K. Saha, E. Gregori, D. Joyner, and A. Goel, “Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assistant,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021...

2021
[12]

Jill watson: A virtual teaching assistant for online education,

A. K. Goel and L. Polepeddi, “Jill watson: A virtual teaching assistant for online education,” Georgia Ins titute of Technology, Tech. Rep., 2016

2016
[13]

Complementary machine intelligence and human intelligence in virtual teaching assistant for tutoring program tracing,

C.-Y. Chou, B.-H. Huang, and C.-J. Lin, “Complementary machine intelligence and human intelligence in virtual teaching assistant for tutoring program tracing,” Computers & Education, vol. 57, no. 4, pp. 2303–2312, 2011

2011
[14]

Nlp -based personal learning assistant for school education.,

A. N. Mathew, J. Paulose, et al., “Nlp -based personal learning assistant for school education.,” International Journal of Electrical & Computer Engineering (2088-8708), vol. 11, no. 5, 2021

2021
[15]

Intelligent education assistant powered by chatbots,

V. Fernoagă, G.-A. Stelea, C. Gavrilă, and F. Sandu, “Intelligent education assistant powered by chatbots,” in The International Scientific Conference eLearning and Software for Education,” Carol I” National Defence University, vol. 2, 2018, pp. 376–383

2018
[16]

Say hello to ‘coding tutor’! design and evaluation of a chatbot based learning system supporting students to learn to program,

S. Hobert, “Say hello to ‘coding tutor’! design and evaluation of a chatbot based learning system supporting students to learn to program,” 2019

2019
[17]

Ejava chatbot for learning programming language: A post - pandemic alternative virtual tutor,

S. H. M. Daud, N. H. I. Teo, and N. H. M. Zain, “Ejava chatbot for learning programming language: A post - pandemic alternative virtual tutor,” International Journal, vol. 8, no. 7, pp. 3290–3298, 2020

2020
[18]

Intents

G. Cloud. “Intents.” (), [Online]. Available: https://cloud.google.com/dialogflow/es/docs/intents-overview. (accessed: 15.10.2021)

2021
[19]

M. L. Hetland, Beginning Python: from novice to professional. Apress, 2008

2008
[20]

Chun, Core python programming

W. Chun, Core python programming. Prentice Hall Professional, 2007. 030042-11 25 February 2024 05:24:11

2007

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Python-bot: A chatbot for teaching python programming.,

C. W. Okonkwo and A. Ade -Ibijola, “Python-bot: A chatbot for teaching python programming.,” Engineering Letters, vol. 29, no. 1, 2020

work page 2020

[2] [2]

Elabmate: A tool for delivering programming courses effectively,

R. Ahmed, “Elabmate: A tool for delivering programming courses effectively,” International Journal of Advanced Corporate Learning (iJAC), vol. 5, no. 3, pp. 6–11, 2012

work page 2012

[3] [3]

A Virtual Teaching Assistant for Personalized Learning

L. Benedetto, P. Cremonesi, and M. Parenti, “A virtual teaching assistant for personalized learning,” arXiv preprint arXiv:1902.09289, 2019

work page Pith review arXiv 1902

[4] [4]

Eliza—a computer program for the study of natural language communication between man and machine,

J. Weizenbaum, “Eliza—a computer program for the study of natural language communication between man and machine,” Communications of the ACM, vol. 9, no. 1, pp. 36–45, 1966

work page 1966

[5] [5]

Using chatbots as smart teaching assistants for first-year engineering students,

S. Abdelhamid and A. Katz, “Using chatbots as smart teaching assistants for first-year engineering students,” in 2020 First-Year Engineering Experience, 2020

work page 2020

[6] [6]

Intelligent chat r obot in digital campus based on deep learning,

Y. Liang, Y. Yu, and W. Ouyang, “Intelligent chat r obot in digital campus based on deep learning,” in Journal of Physics: Conference Series, IOP Publishing, vol. 1629, 2020, p. 012 079

work page 2020

[7] [7]

Framework for educational domain -based multichatbot communication system,

Z. Memon, H. Aghian, M. S. Sarfraz, et al., “Framework for educational domain -based multichatbot communication system,” Scientific Programming, vol. 2021, 2021

work page 2021

[8] [8]

Implementation of chatbot that teach programming language,

M. A. Jayakumar Sadhasivam, A. B. Amitava Mazumdar, B. J.M, and V. Kumar, “Implementation of chatbot that teach programming language,” Xi’an University of Architecture & Technology, vol. 12, 2020

work page 2020