The Essentials of AI for Life and Society: An AI Literacy Course for the University Community

Don Fussell; Joydeep Biswas; Kristen Procko; Kristin Patterson; Lea Sabatini; Peter Stone; Zifan Xu

REVIEW 3 major objections 6 minor 17 references

The Essentials of AI for Life and Society: An AI Literacy Course for the University Community

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A ten-question retrospective survey found that participants rated their AI literacy higher after the course on every item, with gains from 0.97 to 1.37 points on a five-point scale.

desk verdict Readable case study of a broad-audience AI literacy course; the only evidence for learning gains is retrospective self-report, and a within-study check for response-shift was collected but never reported. read the letter →

arxiv 2501.07392 v1 pith:RWDICHZO submitted 2025-01-13 cs.AI cs.CY

Joydeep Biswas , Don Fussell , Peter Stone , Kristin Patterson , Kristen Procko , Lea Sabatini , Zifan Xu This is my paper · ORCID

classification cs.AIcs.CY

keywords AIliteracycoursedesignhighereducationretrospectivepre-postsurveyinterdisciplinaryteachinglargelanguagemodelsseminarprogramevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports on the rapid design and delivery of a one-credit online AI literacy course open to all members of a large public university, including students, faculty, staff, and community members. The authors claim that participants' AI literacy improved: on a ten-question retrospective pre/post survey, every measure rose by between 0.97 and 1.37 points on a five-point Likert scale, with each gain statistically significant at p < 0.01. The course paired lectures on AI fundamentals with interdisciplinary talks on societal impacts, and weekly reflections plus a final survey guided the design of a follow-up three-credit version. The significance is a tested template for bringing AI literacy to non-technical audiences quickly and for gathering actionable feedback that can shape later offerings.

What carries the argument

The central mechanism is the retrospective pre/post self-assessment: a ten-item Likert questionnaire administered at the end of the course in which participants rate their AI literacy both before and after the course in a single sitting. This design is intended to control for response-shift bias, in which a participant's internal standard for 'literate' changes as they learn, and the authors explicitly ground the method in the response-shift literature. The survey, together with weekly reflection prompts and thematically coded open-ended responses, carries the argument that the course improved literacy and identifies the design lessons.

What would settle it

Conduct the same ten-item instrument as a true pre-test at the start of the course and again at the end, with a no-course control group; if the course group shows no larger gain than the control, or if the true pre-test ratings already match the retrospective 'before' ratings, the reported improvement is largely a measurement artifact.

Watch

Extended reading notes

Core claim

The course's central finding is that a broad-audience AI literacy course can move self-assessed literacy substantially in a single semester. In the final survey, 145 respondents rated their agreement with ten statements about their understanding of AI twice: as they recalled their level before the course and as they judged it afterward. Every question showed a statistically significant gain, with the largest increases in the ability to list examples of AI, to discuss AI with an appropriate vocabulary, and to be literate about the technical components of AI. The authors take this as evidence that the course achieved its primary learning objective, and they use the participant feedback to identify what worked (varied speakers, concrete examples) and what did not (challenging readings, disconnected fundamentals).

Load-bearing premise

The paper's central claim rests on retrospective self-reports: participants rated their own literacy before and after the course in a single sitting, with no control group or objective knowledge test to confirm that self-assessed gains correspond to real learning.

Editorial extensions

If this is right

The same 14-week lecture structure with interdisciplinary speakers can be re-deployed quickly at other institutions, since the paper shows it can be assembled in about six weeks.
A three-credit expansion of the course, using the same topic list with more interactive components, is already justified by the feedback and was offered in fall 2024.
The ten survey items provide a reusable, statistically significant outcome measure for evaluating future AI literacy courses.
The finding that lectures were rated easier than readings suggests future iterations should keep lecture-based fundamentals and replace or supplement technical readings with more accessible journalism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next experiment is to administer the same ten items as a true pre-test at the start of the course; comparing those baseline scores with the retrospective 'before' ratings would directly quantify response-shift bias.
The course's structure—short lectures by rotating experts plus weekly reflections—could transfer to workplace continuing-education or public-library settings, where AI literacy gaps are similar but credit and grading are absent.
If the reported gains reflect genuine learning, the ten-item retrospective instrument could serve as a lightweight evaluation tool for other institutions, but only after being validated against an objective knowledge measure.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The paper is a straightforward experience report on a one-credit AI literacy course offered to a broad university audience. Read it for the course design and the institutional context; don't read it for evidence that the course improved AI literacy, because that evidence isn't there.

What the paper does well: it gives a clear picture of the course—14 weekly lectures, interdisciplinary speakers, readings, quizzes, weekly reflections, and a final survey. The enrollment data showing all 17 colleges represented and the breakdown by role is useful. The qualitative coding of open-ended responses is reasonable, and the authors are transparent about several limitations, including the lack of baselines for the detailed evaluation. The abstract is carefully worded: "attendees reported gains." That is true.

The soft spot is the central evaluation claim. The ten AI-literacy questions are retrospective pre/post: participants rated their skill "before" after already completing the course. That design invites response-shift bias, and the authors cite Howard and Dailey but don't address it. The stress-test note is right that the weekly reflections asked for a contemporaneous "prior understanding" rating each week. If those weekly ratings were higher than the retrospective "before" ratings, that would be direct evidence of response-shift. The paper doesn't report that comparison, even though it collected the data. That's a missed opportunity and an actionable fix.

The sample is also narrow: 145 or 151 respondents out of 788 enrollees, with 70% undergraduate and 54% from Natural Sciences. The positive results could easily be selection. There is a small internal inconsistency—the summary says 131 students and 584 auditors while the enrollment section says 132 undergraduates, 631 auditors, and 25 external participants; the survey text says 145 participants while Figure 2 says n=151. Minor, but it should be cleaned up.

The bigger issue is language. In "Lessons Learned" the authors write that the audience "improved their AI literacy." That overstates what the design can support. The abstract's phrasing, "reported gains," is the right one, and the lessons-learned section should match it.

Bottom line: this is a useful case study for instructors building similar courses, with honest discussion of practical challenges. It deserves a serious referee, but the referee should require a revised claim about learning gains and a response to the response-shift check. I'd bring it to a reading group mainly to discuss retrospective pre/post measurement in education.

Referee Report

3 major / 6 minor

Summary. The paper reports on the design, delivery, and retrospective evaluation of a one-credit, online AI literacy course offered at the University of Texas at Austin in Fall 2023 to a broad audience of students, faculty, staff, and community members. It describes the course structure, the interdisciplinary lecture schedule, enrollment demographics, weekly reflection data, and a final course survey. The key quantitative evidence consists of ten retrospective pre/post self-assessment items in which participants rated themselves higher 'now' than they recalled 'before' the course, with average gains between +0.97 and +1.37 on a five-point Likert scale, each reported as statistically significant at p < 0.01. The authors conclude that the course improved participants' AI literacy and use the feedback to design a subsequent three-credit course.

Significance. If the effectiveness claims were treated only as participants' self-reported impressions, the paper provides a useful and detailed blueprint for a broadly accessible AI literacy course, including the lecture schedule, institutional support structures, and openly available course materials. The enrollment data across all UT Austin colleges and the candid reporting of challenges with readings and audience heterogeneity are valuable for instructors designing similar courses. The paper is not a rigorous effectiveness study: it uses no control group, no objective literacy measure, and its sole outcome measure is a retrospective self-assessment collected after the intervention. The authors are appropriately cautious in the abstract ('reported gains') but overstate the conclusion in Lessons Learned. The paper's main contribution is as a course design and lessons-learned narrative, and its conclusions should be scaled back accordingly.

major comments (3)

[Overall Course Survey, Figure 2] The central claim that the course improved participants' AI literacy rests entirely on retrospective pre/post self-ratings, in which the 'before' ratings were collected at the end of the course. The authors cite Howard and Dailey (1979) and Geldhof et al. (2018), both of which are foundational references for response-shift bias, yet the paper never addresses this threat. Response-shift bias predicts exactly the observed pattern: participants recalibrate their recollection of their prior knowledge after being exposed to course content. Without a contemporaneous pre-course questionnaire, a control group, or an objective knowledge test, the gains of +0.97 to +1.37 show only that retrospective recollections differ from current self-assessments, not that literacy changed. The Lessons Learned statement that 'the audience that participated in the final course survey improved their AI literacy' is therefore not supported by the evidence presented; the abstract's 'reported gains' is the defensible phrasing.
[Response to Weekly Surveys] The weekly reflections included a rating of 'prior understanding' of each week's topic, collected contemporaneously before or during the course. This provides a within-study check on response-shift bias that the authors do not report: if the average of the weekly prior-understanding ratings is systematically lower than the retrospective 'before course' ratings on comparable questions, that would directly evidence response-shift. Conversely, if the weekly prior-understanding ratings are similar to or higher than the retrospective 'before' ratings, the retrospective gains would be more credible. The authors should report this comparison, or explain why the two measures are not comparable. As it stands, the only internal validity check available in the data is omitted.
[Overall Course Survey, enrollment demographics] The survey sample is 145 respondents out of 788 enrollees, and the authors report that 70 percent were undergraduates and 54 percent were affiliated with the College of Natural Sciences. This is a heavily self-selected subsample that overrepresents the most engaged and technically oriented participants. The paper does not report response rates by enrollment category (students versus auditors, or by college), nor does it discuss how selection might bias the reported retrospective gains. Given that auditors were found to be less engaged than students, the absence of this analysis weakens the generalization of the reported improvements to the full enrolled population and should be addressed explicitly.

minor comments (6)

[Related Work] The word 'assesments' appears twice in the discussion of Williams (2023) and should be corrected to 'assessments'.
[Table 1] The entry 'Elargethical Datasets' appears to be a typographical error for 'larger ethical datasets' and should be corrected.
[Future Plans] The word 'asyncrhonous' should be 'asynchronous'.
[Relation to Previous Work] The phrase 'aimed at abroad audience' should be 'aimed at a broad audience'.
[Overall Course Survey, Figure 2] The text states that gains are 'reported' in parentheses following each question, but the term should be 'in parentheses'; additionally, Figure 2 would be more informative with error bars, per-item p-values, and a statement of the statistical test used (e.g., paired t-test or Wilcoxon signed-rank).
[Overall Course Survey] The survey description reports n = 151 for all questions except Q5 with n = 150; the paper should explain the single missing response for Q5.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper makes no first-principles derivation; its effectiveness claim rests on self-reported survey gains, a methodological limitation but not a circular reduction.

full rationale

This is a course-report paper, not a derivation, so most circularity patterns do not apply. The central empirical claim—that attendees reported gains in AI literacy—is directly evidenced by the retrospective pre/post survey (Figure 2), and the abstract's phrasing 'reported gains' is consistent with the instrument. The stronger Lessons Learned phrasing 'improved their AI literacy' is an interpretive step that assumes retrospective self-ratings track actual learning; the paper itself cites the response-shift bias literature (Howard and Dailey 1979; Geldhof et al. 2018) and notes there are 'no baselines to compare the detailed evaluation results to.' These are validity and overclaiming concerns, not circularity: the survey is not defined in terms of the conclusion, no parameter is fitted and then called a prediction, and no load-bearing claim is justified by a self-citation chain. Vekhter and Biswas (2023) is a related-work citation only. Accordingly, the derivation chain is self-contained in the limited sense that the outcome is measured, not derived, and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No mathematical model or fitted parameters appear. The central claims rest on survey design assumptions rather than formal derivations.

assumptions (3)

domain assumption Retrospective self-report accurately measures change in AI literacy.
The central evaluation (Figure 2) asks participants to rate their own ability before and after; no objective measure is used. The paper cites response-shift bias literature but does not control for it.
domain assumption Survey respondents are representative of the 788 enrollees.
Only 145 of 788 participants responded, 70% undergraduates and 54% from Natural Sciences, while Table 2 shows a broader enrollment distribution; the paper does not weight or otherwise account for non-response.
domain assumption Participants understood the Likert scale and literacy statements consistently.
Survey items like 'I am literate about the technical components of AI' are vague, and no validation of the instrument is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Essentials of AI for Life and Society: An AI Literacy Course for the University Community." pith.science (2026). https://pith.science/paper/RWDICHZO

@misc{pith2026250107392,
  author       = {Pith},
  title        = {Pith review of: The Essentials of AI for Life and Society: An AI Literacy Course for the University Community},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RWDICHZO}},
  note         = {Machine review of arXiv:2501.07392}
}

read the original abstract

We describe the development of a one-credit course to promote AI literacy at The University of Texas at Austin. In response to a call for the rapid deployment of class to serve a broad audience in Fall of 2023, we designed a 14-week seminar-style course that incorporated an interdisciplinary group of speakers who lectured on topics ranging from the fundamentals of AI to societal concerns including disinformation and employment. University students, faculty, and staff, and even community members outside of the University, were invited to enroll in this online offering: The Essentials of AI for Life and Society. We collected feedback from course participants through weekly reflections and a final survey. Satisfyingly, we found that attendees reported gains in their AI literacy. We sought critical feedback through quantitative and qualitative analysis, which uncovered challenges in designing a course for this general audience. We utilized the course feedback to design a three-credit version of the course that is being offered in Fall of 2024. The lessons we learned and our plans for this new iteration may serve as a guide to instructors designing AI courses for a broad audience.

Figures

Figures reproduced from arXiv: 2501.07392 by the authors.

**Figure 1.** Summary of survey responses on engagement and interest, difficulty of course materials, and overall understanding. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Retrospective pre-/post-survey questions. Each re [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Auerbach, C.; and Silverstein, L. B. 2003. Qualitative data: An introduction to coding and analysis, volume 21

work page 2003
[4]

Carnegie Mellon University . 2024. B.S. in Artificial Intelligence. https://www.cs.cmu.edu/bs-in-artificial-intelligence/. Accessed: 2024-09-16

work page 2024
[5]

J.; Warner, D

Geldhof, G. J.; Warner, D. A.; Finders, J. K.; Thogmartin, A. A.; Clark, A.; and Longway, K. A. 2018. Revisiting the utility of retrospective pre-post designs: the need for mixed-method pilot data. Evaluation and program planning, 83--89

work page 2018
[6]

S.; and Dailey, P

Howard, G. S.; and Dailey, P. R. 1979. Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64(2): 144

work page 1979
[7]

M.-Y.; and Zhang, G

Kong, S.-C.; Cheung, W. M.-Y.; and Zhang, G. 2021. Evaluation of an artificial intelligence literacy course for university students with diverse study backgrounds. Computers and Education: Artificial Intelligence

work page 2021
[8]

Gathering Strength, Gathering Storms: The One Hundred Year Study on Artificial Intelligence (AI100) 2021 Study Panel Report

Littman, M. L.; Ajunwa, I.; Berger, G.; Boutilier, C.; Currie, M.; Doshi-Velez, F.; Hadfield, G.; Horowitz, M. C.; Isbell, C.; Kitano, H.; Levy, K.; Lyons, T.; Mitchell, M.; Shah, J.; Sloman, S.; Vallor, S.; and Walsh, T. 2022. Gathering Strength, Gathering Storms: The One Hundred Year Study on Artificial Intelligence (AI100) 2021 Study Panel Report. arXi...

work page Pith review arXiv 2022

Show all 17 references

[9]

Ng, D. T. K.; Lee, M.; Tan, R. J. Y.; Hu, X.; Downie, J. S.; and Chu, S. K. W. 2023. A review of AI teaching and learning from 2000 to 2020. Education and Information Technologies, 28(7): 8445--8501

2023
[10]

Ng, D. T. K.; Leung, J. K. L.; Chu, K. W. S.; and Qiao, M. S. 2021 a . AI literacy: Definition, teaching, evaluation and ethical issues. Proceedings of the Association for Information Science and Technology, 58(1): 504--509

2021
[11]

Ng, D. T. K.; Leung, J. K. L.; Chu, S. K. W.; and Qiao, M. S. 2021 b . Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence

2021
[12]

Salda \ n a, J. 2021. The coding manual for qualitative researchers

2021
[13]

Saltz, J.; Skirpan, M.; Fiesler, C.; Gorelick, M.; Yeh, T.; Heckman, R.; Dewar, N.; and Beard, N. 2019. Integrating Ethics within Machine Learning Courses. ACM Trans. Comput. Educ., 19(4)

2019
[14]

Scassellati, B. 2023. AI for Future Presidents (CS170). https://zoo.cs.yale.edu/dsac/blog/2023/12/19/cpsc-170/. Accessed: 2024-09-16

2023
[15]

University of Texas at Austin . 2024. M.S. in Artificial Intelligence. https://cdso.utexas.edu/msai. Accessed: 2024-09-16

2024
[16]

Vekhter, J.; and Biswas, J. 2023. Responsible robotics: a socio-ethical addition to robotics courses. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 15877--15885

2023
[17]

Williams, R. 2023. A Review of Assessments in K-12 AI Literacy Curricula . https://randi-c-dubs.github.io/K12-AI-ed/Constructionist_AI_Assessments.pdf. Accessed: 2024-09-16

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Auerbach, C.; and Silverstein, L. B. 2003. Qualitative data: An introduction to coding and analysis, volume 21

work page 2003

[4] [4]

Carnegie Mellon University . 2024. B.S. in Artificial Intelligence. https://www.cs.cmu.edu/bs-in-artificial-intelligence/. Accessed: 2024-09-16

work page 2024

[5] [5]

J.; Warner, D

Geldhof, G. J.; Warner, D. A.; Finders, J. K.; Thogmartin, A. A.; Clark, A.; and Longway, K. A. 2018. Revisiting the utility of retrospective pre-post designs: the need for mixed-method pilot data. Evaluation and program planning, 83--89

work page 2018

[6] [6]

S.; and Dailey, P

Howard, G. S.; and Dailey, P. R. 1979. Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64(2): 144

work page 1979

[7] [7]

M.-Y.; and Zhang, G

Kong, S.-C.; Cheung, W. M.-Y.; and Zhang, G. 2021. Evaluation of an artificial intelligence literacy course for university students with diverse study backgrounds. Computers and Education: Artificial Intelligence

work page 2021

[8] [8]

Gathering Strength, Gathering Storms: The One Hundred Year Study on Artificial Intelligence (AI100) 2021 Study Panel Report

Littman, M. L.; Ajunwa, I.; Berger, G.; Boutilier, C.; Currie, M.; Doshi-Velez, F.; Hadfield, G.; Horowitz, M. C.; Isbell, C.; Kitano, H.; Levy, K.; Lyons, T.; Mitchell, M.; Shah, J.; Sloman, S.; Vallor, S.; and Walsh, T. 2022. Gathering Strength, Gathering Storms: The One Hundred Year Study on Artificial Intelligence (AI100) 2021 Study Panel Report. arXi...

work page Pith review arXiv 2022