pith. sign in

arxiv: 2605.09393 · v3 · pith:VYV6JCWDnew · submitted 2026-05-10 · 💻 cs.SE

Prediction Model of Motivators and Demotivators of Integrating Large Language Models in Software Engineering Education: An Empirical Study

Maryam Khan , Muhammad Azeem Akbar , Jussi Kasurinen , Estefan\'ia Mart\'in-Barroso This is my paper

Pith reviewed 2026-05-20 23:12 UTC · model grok-4.3

classification 💻 cs.SE
keywords large language modelssoftware engineering educationmotivators and demotivatorsprediction modelgenetic algorithm optimizationstakeholder surveycost-aware integrationgovernance safeguards
0
0 comments X

The pith

Stakeholder surveys combined with probabilistic modeling and genetic algorithm optimization can identify cost-efficient paths for integrating large language models into software engineering education.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a prediction model that turns survey answers about what encourages or discourages the use of large language models into estimates of how familiar stakeholders are likely to become with the technology. Those estimates then feed an optimization routine that weighs the expected familiarity gains against the money and effort required to implement various integration steps. A sympathetic reader would care because the resulting framework gives institutions a concrete way to plan staged rollouts that favor high-benefit areas while respecting limited resources and addressing risks such as plagiarism and loss of critical thinking skills. The work shows that ethical and governance safeguards emerge as high-priority items when costs are constrained.

Core claim

The study claims that operationalizing nineteen factors into a survey of 126 stakeholders, training Naive Bayes and Logistic Regression models to predict the probability of high LLM familiarity from Likert responses, and embedding those probabilities in a Genetic Algorithm that minimizes implementation cost produces an optimization-informed decision support framework capable of recommending staged, cost-aware integration strategies for large language models in software engineering education, with particular emphasis on governance mechanisms such as integrity and ethical safeguards.

What carries the argument

The Genetic Algorithm optimization layer that trades off predicted familiarity probabilities against implementation costs at both global and category-specific levels.

If this is right

  • Governance mechanisms focused on integrity and ethical safeguards should receive priority when budgets are limited.
  • Programming assistance, debugging support, and personalized adaptive learning are perceived as the strongest benefits.
  • Plagiarism concerns, over-reliance risks, and potential reductions in critical thinking require explicit mitigation in any rollout plan.
  • Decisions can be made separately at the overall institutional level and at the level of individual educational categories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same survey-plus-optimization structure could be reused for other classroom technologies by substituting new factors while keeping the modeling steps intact.
  • Institutions might begin with low-cost pilots in programming assistance to raise familiarity before expanding into more complex uses.
  • The cost-awareness built into the model implies a need for periodic re-surveying to check whether actual familiarity matches the original predictions.

Load-bearing premise

The nineteen factors taken from earlier literature taxonomies capture the essential motivators and demotivators, and the Likert-scale answers collected from the 126 stakeholders supply reliable training data for the probabilistic models and the subsequent optimization.

What would settle it

A follow-up study that collects new survey responses from a comparable or larger group and produces materially different optimization priorities, such as down-ranking governance safeguards, would falsify the claim that the current model reliably identifies cost-efficient integration strategies.

Figures

Figures reproduced from arXiv: 2605.09393 by Estefan\'ia Mart\'in-Barroso, Jussi Kasurinen, Maryam Khan, Muhammad Azeem Akbar.

Figure 1
Figure 1. Figure 1: Research Methodology Process and Cognitive Load, and Integration and Practical Implementation. Across these categories, ten sub￾themes were identified: Plagiarism and Intellectual Property Concerns, Over-Reliance on AI in Learning, Reduced Critical Thinking and Problem-Solving, Ethical Concerns in AI-Assisted Learning, Challenges in Evaluating Learning Outcomes, Security, Privacy, and Data Integrity Issues… view at source ↗
Figure 1
Figure 1. Figure 1: Research Methodology Process • Phase 3: Data Preprocessing and Model Training – Preprocessing of the collected survey data and training of predictive models based on the validated responses. • Phase 4: Probabilistic Cost and Effort Prediction Modeling – Development of a proba￾bilistic cost and effort prediction model grounded in the taxonomies (Phase 1) and the empirically collected and analysed data (Phas… view at source ↗
Figure 2
Figure 2. Figure 2: Demographic characteristics of the survey respondents ( [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Demographic characteristics of the survey respondents ( [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prediction Model ø Final Insight from Factor-Level Optimization Factor-level optimization indicates that effective LLM integration is not achieved by uniformly addressing all factors in a specific category but by selectively prioritizing those that deliver the highest impact relative to effort. The results show that stronger outcomes are associated with investments in deep learning processes, structured fe… view at source ↗
Figure 3
Figure 3. Figure 3: Prediction Model only an automation tool [49]. In risk-oriented domains, the model emphasizes Bias and Hallucination in LLM Outputs, Security, Privacy, and Data Integrity Issues, and Reduced Critical Thinking and Problem￾Solving. The third layer, represented through category-level efficiency gains (∆Fitness), supports staged im￾plementation planning. Pedagogical categories such as Collaboration and Peer Le… view at source ↗
read the original abstract

Context: Large Language Models (LLMs) are increasingly influencing software engineering practice and education. While prior studies examine their technical performance and classroom use, limited research provides cost-aware and empirically grounded models for systematic institutional integration. Objective: This study develops and validates a prediction model to identify cost-efficient strategies for integrating LLMs into software engineering education using motivating and demotivating factors. Method: Based on our previously developed literature survey taxonomies [1], we operationalized 19 validated factors (9 motivators and 10 demotivators) into a structured survey completed by 126 stakeholders from multiple countries. Likert-scale responses were encoded and used to train probabilistic models (Naive Bayes and Logistic Regression) to estimate the likelihood of high LLM familiarity. The probability estimates were integrated into a Genetic Algorithm (GA)-based optimization framework to model trade-offs between predicted familiarity and implementation cost at global and category levels. Results: Respondents perceived strong benefits in Programming Assistance and Debugging Support and Personalized and Adaptive Learning. Major concerns included Plagiarism and Intellectual Property Concerns, Over-Reliance on AI in Learning, and Reduced Critical Thinking and Problem Solving. Optimization results indicate that governance-related mechanisms, particularly integrity and ethical safeguards, should be prioritized under cost constraints. Conclusions: The study introduces an optimization-informed decision support framework linking stakeholder perceptions with probabilistic modeling and cost-effort analysis. The model supports staged and cost-aware LLM integration grounded in governance stability and pedagogically meaningful development.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The paper develops a prediction model for motivators and demotivators of LLM integration in software engineering education. It operationalizes 19 factors (9 motivators, 10 demotivators) drawn from the authors' prior taxonomy into a survey completed by 126 stakeholders across countries. Likert responses are encoded to train Naive Bayes and Logistic Regression models estimating the probability of high LLM familiarity; these probabilities feed a Genetic Algorithm that optimizes trade-offs between predicted familiarity and implementation costs at global and category levels. Results emphasize benefits in programming assistance and debugging while highlighting concerns over plagiarism, over-reliance, and reduced critical thinking, with optimization favoring governance mechanisms such as integrity and ethical safeguards.

Significance. If the probabilistic models are validated and cost assignments are made transparent and reproducible, the work could supply a practical, optimization-based decision-support framework that connects stakeholder perceptions to staged, cost-aware LLM integration strategies. The combination of empirical survey data, probabilistic modeling, and GA-driven trade-off analysis extends prior taxonomies and offers falsifiable outputs (e.g., prioritized factor lists under varying cost constraints) that institutions could test.

major comments (2)
  1. [Abstract / Method] Abstract and Method: No performance metrics (accuracy, AUC, calibration, or cross-validation results) are reported for the Naive Bayes and Logistic Regression models. Because the probability estimates P(high familiarity) are the direct inputs to the Genetic Algorithm, the lack of validation leaves the optimization outputs sensitive to unexamined model quality rather than demonstrably supported by the 126 responses.
  2. [Method] Method: The procedure for deriving or assigning concrete implementation cost values to the 19 factors (including governance safeguards) is not described. Without explicit cost quantification, weighting scheme, or sensitivity analysis, the GA results that prioritize integrity/ethics under cost constraints rest on unspecified assumptions and cannot be reproduced or stress-tested from the stakeholder data alone.
minor comments (3)
  1. [Method] Clarify the exact encoding scheme used to convert Likert-scale responses into features for the probabilistic models.
  2. [Method] Report the specific GA hyperparameters, population size, number of generations, and fitness function formulation.
  3. [Discussion / Limitations] Add a limitations subsection discussing response bias, sample representativeness across countries, and dependence on the authors' earlier taxonomy.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback, which identifies key areas for strengthening the methodological rigor and reproducibility of our work. We address each major comment point by point below, indicating the revisions planned for the next version of the manuscript.

read point-by-point responses

  1. Referee: [Abstract / Method] Abstract and Method: No performance metrics (accuracy, AUC, calibration, or cross-validation results) are reported for the Naive Bayes and Logistic Regression models. Because the probability estimates P(high familiarity) are the direct inputs to the Genetic Algorithm, the lack of validation leaves the optimization outputs sensitive to unexamined model quality rather than demonstrably supported by the 126 responses.

    Authors: We acknowledge the validity of this observation. The manuscript presents the application of the trained models to generate probability estimates for the Genetic Algorithm but does not include explicit performance evaluation. In the revised manuscript, we will add a dedicated subsection in the Method section reporting 5-fold cross-validation results, including accuracy, AUC-ROC, precision, recall, F1-score, and Brier score for calibration, for both Naive Bayes and Logistic Regression. We will also discuss the implications of these metrics for the reliability of the P(high familiarity) inputs to the optimization framework. revision: yes

  2. Referee: [Method] Method: The procedure for deriving or assigning concrete implementation cost values to the 19 factors (including governance safeguards) is not described. Without explicit cost quantification, weighting scheme, or sensitivity analysis, the GA results that prioritize integrity/ethics under cost constraints rest on unspecified assumptions and cannot be reproduced or stress-tested from the stakeholder data alone.

    Authors: This comment correctly highlights a transparency gap. The current text describes the integration of costs into the GA but does not specify their derivation. We will revise the Method section to include an explicit description of the cost assignment process: the numerical values assigned to each of the 19 factors, the basis for those values (expert estimation informed by educational resource literature), the weighting scheme applied at global and category levels, and a sensitivity analysis that varies cost parameters to demonstrate the robustness of the prioritization of integrity and ethical safeguards. These additions will enable full reproducibility and stress-testing. revision: yes

Circularity Check

1 steps flagged

Self-citation for factor taxonomy but independent survey data and modeling provide new content

specific steps
  1. self citation load bearing [Abstract / Method]
    "Based on our previously developed literature survey taxonomies [1], we operationalized 19 validated factors (9 motivators and 10 demotivators) into a structured survey completed by 126 stakeholders from multiple countries. Likert-scale responses were encoded and used to train probabilistic models (Naive Bayes and Logistic Regression) to estimate the likelihood of high LLM familiarity."

    The selection and operationalization of the 19 factors that define the entire input space for the probabilistic models and GA optimization rests solely on the authors' overlapping prior publication [1]; while new survey responses are collected, the factor structure itself is not re-derived or externally validated within this paper and therefore carries the self-citation into the central modeling pipeline.

full rationale

The paper selects its 19 motivators and demotivators from the authors' own prior literature survey taxonomy [1] and then collects fresh Likert-scale responses from 126 stakeholders to train Naive Bayes and Logistic Regression models whose outputs feed a Genetic Algorithm. This self-citation defines the input structure but does not make the subsequent probability estimates or optimization results equivalent to the prior taxonomy by construction; the empirical data and fitted models introduce independent content. No equations reduce predictions to inputs, no uniqueness claims are imported, and no ansatz is smuggled. The central decision-support framework therefore retains non-circular empirical grounding despite the self-referential starting taxonomy.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The framework rests on the assumption that the prior taxonomy factors are complete and that survey data can be directly mapped to cost-benefit optimization without additional validation steps.

free parameters (2)
  • Logistic regression and Naive Bayes model parameters
    Fitted to the 126 survey responses to predict high LLM familiarity likelihood.
  • Genetic algorithm hyperparameters and cost weights
    Chosen to balance predicted familiarity against unspecified implementation costs at global and category levels.
axioms (2)
  • domain assumption The 19 factors from the authors' prior literature survey accurately represent stakeholder motivators and demotivators.
    Invoked when operationalizing factors into the survey instrument.
  • standard math Likert-scale responses provide valid quantitative input for probabilistic modeling.
    Standard assumption when encoding survey answers for Naive Bayes and logistic regression.

pith-pipeline@v0.9.0 · 5819 in / 1495 out tokens · 34970 ms · 2026-05-20T23:12:32.883003+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.