AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study

Nesma Ragab Nasr; Samaa Haniya; Shahin Hossain; Shapla Khanam

REVIEW 5 major objections 6 minor 8 references

AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 318-student survey links LLM use to perceived literacy gains in the US and Bangladesh.

desk verdict A useful but over-sold cross-national survey: the US-Bangladesh data are a real contribution, but the abstract claims causal effects the design cannot support. read the letter →

arxiv 2507.03020 v2 pith:DH26KTXN submitted 2025-07-02 cs.CY

Shahin Hossain , Shapla Khanam , Samaa Haniya , Nesma Ragab Nasr This is my paper

classification cs.CY

keywords LargeLanguageModelsAIliteracyHighereducationExpectancy-ValueTheory3PModelCross-nationalstudyStudentsurveyAcademicmotivation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a cross-national survey of 318 university students in the United States and Bangladesh asking how they use large language models, how literate they feel about AI, what motivates them, and what outcomes they perceive. Its central claim is that LLM use is associated with perceived benefits: students who use LLMs more often report stronger literacy gains and more optimism, with a correlation of r = .59 between use frequency and perceived literacy impact. The study also finds that U.S. students and STEM majors use LLMs more frequently than Bangladeshi students and non-STEM majors, while Bangladeshi students report larger literacy benefits. Low formal AI literacy scores lead the authors to argue that universities should build ethically grounded, culturally adapted AI literacy curricula.

What carries the argument

Three guiding models carry the argument. The AI Literacy Framework divides readiness into knowledge, skills, and dispositions; expectancy-value theory explains engagement through expected success, task value, and perceived costs; the 3P model treats learning as presage (prior ability and motivation), process (LLM strategies), and product (outcomes). The empirical work is a 21-item online survey whose Likert scales are scored into five constructs, checked with Cronbach's alpha (above .70), then analyzed with Pearson correlations and group comparisons (t-tests and ANOVAs). The correlation matrix and group comparisons are what carry the conclusions.

What would settle it

Take a comparable student cohort, log their actual LLM interactions through institutional tools, and measure literacy gains with pre/post writing assessments; if heavy users show no larger objective gains than light users, or show lower-quality writing, the paper's central correlation would not survive.

Watch

Extended reading notes

Core claim

The paper establishes, on its own terms, that among 318 students at one U.S. and one Bangladeshi university, frequency of LLM use is positively associated with perceived literacy impact (r = .59, p < .001) and optimism (r = .41, p < .001), that U.S. and STEM students report higher usage than Bangladeshi and non-STEM students, and that Bangladeshi students nevertheless perceive larger literacy benefits. It reads these patterns through an integrated set of theoretical lenses—AI literacy, expectancy-value motivation, and the 3P model of learning—concluding that perceived usefulness and motivational beliefs, not formal training, are the immediate drivers of engagement while formal AI literacy remains low (M = 1.86).

Load-bearing premise

The conclusions depend on self-reported survey answers accurately reflecting actual LLM use, literacy gains, and academic outcomes.

Editorial extensions

If this is right

Universities that want higher LLM engagement can target perceived usefulness and confidence, since these motivational beliefs correlate with use frequency.
Formal AI literacy training is the clearest measured gap, so curricula teaching knowledge, skills, and ethical dispositions would address a documented deficit.
Non-STEM students report meaningfully lower use, so discipline-specific support in non-STEM courses could narrow the gap.
Cross-national differences in optimism and perceived literacy impact imply that blanket AI policies will fit neither context and need local calibration.
Because ethical concern correlates weakly and negatively with AI literacy, raising literacy may also reduce unfocused ethical anxiety, though the effect is small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The strong use–literacy correlation is untested against objective outcomes; an independent replication with writing-quality or grade data could change the size or sign of the effect.
Bangladeshi students' higher perceived literacy impact at lower usage suggests perceived benefit may reflect learning context or task demands rather than amount of use; comparing task types across countries would clarify this.
The low AI literacy score (M = 1.86) combined with moderate-to-high use suggests students learn by doing, so just-in-time training embedded in assignments may beat prerequisite AI courses.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know: this paper has a genuinely new dataset, but it overclaims what that dataset can show. The abstract says LLMs 'enhance access to information, improve writing, and boost academic performance,' while the methods section and limitations explicitly say the design is non-experimental, correlational, and precludes causal inference. That is an internal contradiction, and it is the main thing to flag.

Credit where due: the cross-national comparison between US and Bangladesh is rare; Bangladesh is understudied in this literature. The authors integrate three established frameworks (AI Literacy, Expectancy-Value, 3P) without forcing them into a derivative model. They report reliability coefficients, effect sizes, and a power analysis. The finding that Bangladeshi students perceive higher literacy benefits while US students are more optimistic is a concrete, context-specific result that future work can build on.

Soft spots. First, the causal language in the abstract and discussion should be fixed. The r = .59 is between two self-report scales; it supports 'associated with,' not 'enhances.' Second, the demographic table is internally inconsistent: the race/ethnicity counts sum to 295 out of 318, and the Asian count (171) exactly equals the US count (171). That looks like a recoding or data entry error, and it needs to be corrected before anyone uses these numbers. Third, no adjustment for multiple comparisons across the many t-tests/ANOVAs; not fatal, but it should be acknowledged. Fourth, the data, survey instrument, and analysis code are not posted, so claims of transparency are unverifiable. Fifth, all outcomes are self-perceptions; there are no behavioral or performance logs.

These are all fixable. The core of the paper is a descriptive survey, and as a descriptive survey it is reasonably careful. The problem is that it keeps being sold as something more.

Who should read it: anyone working on AI literacy policy, especially in South Asia; it is a useful data point even with the caveats.

Recommendation: send it to peer review, but require revision. A serious referee can get this into publishable shape by deleting the causal verbs, fixing the table, and adding the data and instrument. I would not cite it in its current form, but I would not desk-reject it either.

Referee Report

5 major / 6 minor

Summary. This manuscript reports a cross-national survey study of 318 university students (171 in the United States, 147 in Bangladesh) examining AI literacy, familiarity with and use of large language models (LLMs), motivational beliefs, and self-reported academic and ethical outcomes. The authors use descriptive statistics, Pearson correlations, t-tests, and ANOVAs, interpreting results through the AI Literacy Framework, Expectancy-Value Theory, and Biggs' 3P Model. The headline findings are a correlation between LLM use frequency and perceived literacy impact (r = .59, p < .001), a correlation with optimism (r = .41, p < .001), and group differences in use by country, gender, and major. The abstract claims that LLMs enhance access to information, improve writing, and boost academic performance.

Significance. If its claims were restricted to descriptive associations, this would be a modest but useful contribution: the study provides a cross-national comparison of LLM engagement in higher education, applies established theoretical frameworks, and reports effect sizes and a power analysis. It also adapts validated instruments and obtained IRB approval. However, the paper's central causal language is not supported by its non-experimental, cross-sectional design, and the demographic table contains serious internal inconsistencies. Because these issues affect the headline conclusions, the current version requires substantial revision before the empirical contribution can be fairly assessed.

major comments (5)

[Abstract / Research Design] The abstract states that 'LLMs enhance access to information, improve writing, and boost academic performance,' but the Research Methodology section explicitly describes the design as 'non-experimental and correlational in nature, aiming to test associations and group differences rather than causal relationships,' and the Limitations section states that 'a cross-sectional design also precludes causal inference.' No analysis reported in the Results section supports causal claims; the correlation r = .59 is between two self-reported measures collected at one time point. The causal verbs must be removed from the abstract and Discussion unless the authors add a longitudinal or experimental design.
[Table 1 / Demographic Characteristics] The demographic tabulation is internally inconsistent. The race/ethnicity counts (Asian 171, White 73, Black/African American 21, Hispanic/Latine 19, Mixed Race 11) sum to 295, not the reported N = 318. The text states that 61% of the sample is Asian, but 171/318 = 54%. The Asian count exactly equals the U.S. count (171), which raises the possibility of a data coding or reporting error. Please verify the raw data and correct the table, the percentages, and the narrative description of sample diversity.
[Results / Correlational Analysis] The abstract's claim that LLMs 'boost academic performance' is not supported by any reported inferential statistic. The 'Grade Impact' variable (M = 4.08, SD = 0.49) appears only in the descriptive statistics table; no correlation, t-test, or ANOVA involving Grade Impact is reported, and the correlation matrix in Figure 6 is not described for this variable. Thus the only evidence bearing on academic performance is a single self-reported item, with no statistical link to LLM use. Either report the relevant analysis or remove the performance claim.
[Results / Group Comparisons] Multiple inferential tests are reported without any control for the family-wise error rate. The paper presents t-tests and ANOVAs for gender, country, major, and several outcome variables (LLM use, perceived literacy impact, optimism, AI literacy, AI familiarity, grade impact, ethical concern). With roughly a dozen tests, some p-values below .05 are expected to arise by chance. Please apply a correction (e.g., Bonferroni or Benjamini-Hochberg) or pre-specify the primary analysis plan, and indicate which findings survive.
[Instrumentation and Data Analysis] The paper states that Cronbach's alpha coefficients for each construct were calculated and that all exceeded .70, but only two alphas are reported (AI Literacy α = .77; Motivational Beliefs α = .81). The remaining constructs (LLM Use Frequency, Perceived Literacy Impact, Grade Impact, Ethical Concern, Optimism) lack reported reliability coefficients. Please provide the full set of alphas or explain why reliability is not assessed for these constructs.

minor comments (6)

[Limitations and Bias] The sentence 'A cross-sectional design also precludes causal inference' is repeated twice within the same paragraph; please remove the duplicate.
[Results / Group Comparisons] The gender comparison reports t(298), but the subgroup sizes (men n = 163, women n = 143) imply df = 304, and the treatment of the 12 non-binary respondents is not described. Please clarify the df and the handling of non-binary participants.
[Introduction] The duplicated citation '(Ennion & McLellan, 2025; Cotton et al., 2024; Chiu, 2024)' appears twice consecutively; please delete the duplicate.
[Instrumentation] The paper mentions that the AI familiarity scale was rescaled from a 10-point continuum, but the rescaling method is not described; please specify the transformation.
[Data Availability] No data availability statement or survey instrument is included; appending the 21-item instrument and a data-sharing statement would improve transparency and reproducibility.
[Figures] Figures 5-8 are referenced in the text but not described in enough detail to verify the plotted values; please add captions or text that explain what each figure shows and ensure the figures are legible in grayscale.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the correlations and group differences are direct empirical summaries of survey responses, not outputs of a fitted model or of a self-citation chain.

full rationale

This paper contains no derivation chain that could collapse into its inputs. The central results are Pearson correlations and ANOVA/t-test comparisons computed directly from Likert-scale survey responses; no parameter is fitted to one subset of data and then reported as a prediction of a closely related quantity. The constructs (AI literacy, motivational beliefs, perceived literacy impact, grade impact) are operationalized through survey items adapted from independently published instruments, and the theoretical frameworks (AI Literacy Framework, Expectancy-Value Theory, Biggs’ 3P Model) are used ex post to interpret associations rather than to force them. There are no load-bearing self-citations: none of the cited sources are works by the present authors, and no uniqueness theorem or prior author-derived ansatz is invoked to rule out alternatives. The paper’s own Limitations section states that data are self-reported and that the cross-sectional design precludes causal inference. The abstract’s causal phrasing ('LLMs enhance access to information, improve writing, and boost academic performance') is inconsistent with the non-experimental, correlational design, but that is an internal-validity and framing problem, not circularity, because no quantitative result is equivalent by construction to any input. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is an empirical survey study, not a derivation, so there are no fitted parameters or invented entities. The main implicit axioms are the validity of self-reports and the applicability of the chosen theoretical frameworks.

assumptions (2)

domain assumption Self-reported survey responses are valid indicators of actual LLM use, literacy, and academic outcomes.
The central correlations and group differences rely on the assumption that Likert-scale self-reports reflect real behavior and outcomes, acknowledged as a limitation but still load-bearing.
domain assumption The AI Literacy Framework, Expectancy-Value Theory, and Biggs' 3P Model are appropriate and mutually compatible lenses for operationalizing the constructs.
The survey constructs are derived from these theories without empirical validation of their integration in this context.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study." pith.science (2026). https://pith.science/paper/DH26KTXN

@misc{pith2026250703020,
  author       = {Pith},
  title        = {Pith review of: AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DH26KTXN}},
  note         = {Machine review of arXiv:2507.03020}
}

read the original abstract

This study presents a cross-national quantitative analysis of how university students in the United States and Bangladesh interact with Large Language Models (LLMs). Based on an online survey of 318 students, results show that LLMs enhance access to information, improve writing, and boost academic performance. However, concerns about overreliance, ethical risks, and critical thinking persist. Guided by the AI Literacy Framework, Expectancy-Value Theory, and Biggs' 3P Model, the study finds that motivational beliefs and technical competencies shape LLM engagement. Significant correlations were found between LLM use and perceived literacy benefits (r = .59, p < .001) and optimism (r = .41, p < .001). ANOVA results showed more frequent use among U.S. students (F = 7.92, p = .005) and STEM majors (F = 18.11, p < .001). Findings support the development of ethical, inclusive, and pedagogically sound frameworks for integrating LLMs in higher education.

Figures

Figures reproduced from arXiv: 2507.03020 by the authors.

**Figure 2.** Expectancy Value Theory and LLMs [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** LLMs through John Bigg’s 3P Model [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Mean and Standard Deviation of Key LLM Variables Correlational Analysis Pearson correlation coefficients among the key study variables are depicted in [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages

[1]

AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study

AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study Shahin Hossain University of Maryland Baltimore County Baltimore, MD, United States Email: shahinh1@umbc.edu Shapla Khanam Washington University of Science and Technology, VA, United States Email: s.khanam@research.wust.edu Samaa Haniya Pepperdine University, Graduate ...

work page Pith review arXiv doi:10.48550/arxiv.2507.03020 2025
[2]

Expectancy-Value Theory complements the AI Literacy Framework by offering a robust account of why students choose to engage with LLMs at different levels

Expectancy Value Theory and LLMs Figure 2 visualizes how expectancies, value appraisals, and perceived costs converge to influence students’ motivation to engage with LLMs. Expectancy-Value Theory complements the AI Literacy Framework by offering a robust account of why students choose to engage with LLMs at different levels. However, it does not address ...

work page 1999
[3]

LLMs through John Bigg’s 3P Model Figure 3 shows how learner characteristics interact with engagement strategies to produce varying academic outcomes. Biggs’ model is indispensable for understanding how students engage with LLMs in authentic academic settings something neither the AI Literacy Framework nor Expectancy-Value Theory fully accounts for. Integ...

work page 2024
[8]

Non-STEM Comparison of Mean Scores Collectively, these descriptive, correlational, and comparative findings provide a comprehensive empirical foundation

STEM vs. Non-STEM Comparison of Mean Scores Collectively, these descriptive, correlational, and comparative findings provide a comprehensive empirical foundation. They underscore how demographic and disciplinary factors shape students’ engagement with LLMs, highlighting key motivational, ethical, and literacy- 19 related trends. These results lay a robust...

work page arXiv 2025
[467]

doi: 10.3390/bs15040467

work page doi:10.3390/bs15040467
[2022]

offers a lens for understanding how background factors and learning strategies mediate the use of LLMs and their outcomes. By bringing together robust statistical analyses with cross-country perspectives, this work establishes evidence-based insights to inform curricula for educating in AI literacy, support equity-informed interventions, and discipline-sp...

work page 2025
[2024]

point out students’ concerns regarding heavy reliance, privacy, ethical accountability, and academic integrity, despite recognition of LLMs’ educational advantages that LLMs make the academic tasks easier. Current studies reinforce these apprehensions, notably Acut et al.’s (2025) book chapter, which emphasizes growing unease among educators regarding LLM...

work page 2025
[2025]

strongly disagree

The survey instrument comprised 21 closed questions that were divided into five primary constructs: (1) AI literacy (knowledge, skills, and disposition); (2) familiarity and frequency of LLM use; (3) motivational beliefs (expectancies, task value, and perceived cost); (4) perceived academic impact; and (5) ethical concerns. All items, except the familiari...

work page 2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study

AI Literacy and LLM Engagement in Higher Education: A Cross-National Quantitative Study Shahin Hossain University of Maryland Baltimore County Baltimore, MD, United States Email: shahinh1@umbc.edu Shapla Khanam Washington University of Science and Technology, VA, United States Email: s.khanam@research.wust.edu Samaa Haniya Pepperdine University, Graduate ...

work page Pith review arXiv doi:10.48550/arxiv.2507.03020 2025

[2] [2]

Expectancy-Value Theory complements the AI Literacy Framework by offering a robust account of why students choose to engage with LLMs at different levels

Expectancy Value Theory and LLMs Figure 2 visualizes how expectancies, value appraisals, and perceived costs converge to influence students’ motivation to engage with LLMs. Expectancy-Value Theory complements the AI Literacy Framework by offering a robust account of why students choose to engage with LLMs at different levels. However, it does not address ...

work page 1999

[3] [3]

LLMs through John Bigg’s 3P Model Figure 3 shows how learner characteristics interact with engagement strategies to produce varying academic outcomes. Biggs’ model is indispensable for understanding how students engage with LLMs in authentic academic settings something neither the AI Literacy Framework nor Expectancy-Value Theory fully accounts for. Integ...

work page 2024

[4] [8]

Non-STEM Comparison of Mean Scores Collectively, these descriptive, correlational, and comparative findings provide a comprehensive empirical foundation

STEM vs. Non-STEM Comparison of Mean Scores Collectively, these descriptive, correlational, and comparative findings provide a comprehensive empirical foundation. They underscore how demographic and disciplinary factors shape students’ engagement with LLMs, highlighting key motivational, ethical, and literacy- 19 related trends. These results lay a robust...

work page arXiv 2025

[5] [467]

doi: 10.3390/bs15040467

work page doi:10.3390/bs15040467

[6] [2022]

offers a lens for understanding how background factors and learning strategies mediate the use of LLMs and their outcomes. By bringing together robust statistical analyses with cross-country perspectives, this work establishes evidence-based insights to inform curricula for educating in AI literacy, support equity-informed interventions, and discipline-sp...

work page 2025

[7] [2024]

point out students’ concerns regarding heavy reliance, privacy, ethical accountability, and academic integrity, despite recognition of LLMs’ educational advantages that LLMs make the academic tasks easier. Current studies reinforce these apprehensions, notably Acut et al.’s (2025) book chapter, which emphasizes growing unease among educators regarding LLM...

work page 2025

[8] [2025]

strongly disagree

The survey instrument comprised 21 closed questions that were divided into five primary constructs: (1) AI literacy (knowledge, skills, and disposition); (2) familiarity and frequency of LLM use; (3) motivational beliefs (expectancies, task value, and perceived cost); (4) perceived academic impact; and (5) ethical concerns. All items, except the familiari...

work page 2023