Pith. sign in

REVIEW 4 major objections 6 minor 37 references

What Can Youth Learn About Artificial Intelligence and Machine Learning in One Hour? Examining How Hour of Code Activities Address the Five Big Ideas of AI

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Hour of Code AI activities cluster on perception, skip reasoning, and tell more than they do.

desk verdict Useful, clearly-scoped content analysis with credible topic-distribution findings, but the central 'telling vs. doing' claim needs explicit numbers and the coding artifacts should be shared. read the letter →

arxiv 2412.11911 v2 pith:OY6CQMTC submitted 2024-12-16 cs.CY

classification cs.CY
keywords HourofCodeAIliteracyK-12educationmachinelearningfivebigideascontentanalysissocietalimpactcomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper analyses all 47 Hour of Code activities that introduce artificial intelligence or machine learning to middle and high school students. It finds that the portfolio is lopsided: perception appears in 83% of the activities that engage with at least one big idea, learning in 75%, natural interaction and societal impact in about 42% each, and representation and reasoning in only 14%. Only two activities touch all five big ideas, and many activities labeled as AI contain no AI content at all. On pedagogy, the authors find far more 'telling' than 'doing': ideas are usually conveyed through videos and teacher guides rather than through learners building, training, or experimenting with models. The paper argues this matters because these short activities are many students' first and possibly only formal contact with AI.

What carries the argument

The analytical machinery is the five big ideas of AI—perception, learning, natural interaction, representation and reasoning, societal impact—used as a content taxonomy, together with a five-aspect breakdown of machine learning (defining ML, how learning algorithms work, the role of training data, data bias, and learning-versus-application phases). The authors apply this taxonomy deductively to the 47 activities, add inductive codes for societal-impact topics and instructional mode (telling vs. hands-on), and code collaboratively for unanimous consensus. The taxonomy is what turns a collection of tutorials into comparable measures of content coverage and pedagogical engagement.

What would settle it

Re-code the 47 activities with independent coders who do not discuss their judgments, and observe a sample of classroom implementations; if independent coding or classroom observation shows far more hands-on model building than the consensus coding captured, the central 'mostly telling' claim would be undercut.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Hour of Code's AI and machine learning offerings, as a portfolio in 2023–2024, are narrow in content and shallow in hands-on engagement. Using the five big ideas of AI as a coding scheme, the authors report that of the 37 activities addressing at least one big idea, perception is most common (83.33%), followed by learning (75%), natural interaction (41.67%) and societal impact (41.67%), with representation and reasoning last (13.89%). A deeper look at machine learning shows that while 17 activities ascribe model behavior to training data, only nine address learning algorithms, and ten of the 27 ML-related activities never define machine learning. On the instructional side, the authors note that ideas are often integrated through videos or teacher explanations without opportunities for hands-on engagement, and that several activities labeled as AI-related by the platform do not engage with any big idea. The increase in attention to societal impact—15 of 37 activities—is described as a surprising and welcome shift compared with earlier Hour of Code offerings.

Load-bearing premise

The study assumes that the public descriptions, videos, and tutorials of each activity, as read and coded by three researchers in consensus, faithfully represent what learners actually experience and can learn in the hour.

Editorial extensions

If this is right

  • If the analysis holds, the Hour of Code's flagship AI offerings give millions of students a lopsided first view of AI: heavy on perception and data-centric machine learning, light on reasoning and representation.
  • Some activities labeled as AI by the platform teach no AI at all, which risks misleading students, educators, and parents about what AI is.
  • The jump from six AI activities in 2021 to 47 by April 2024 shows fast growth, but the concentration of content means scale has not yet brought breadth.
  • The rise in societal-impact coverage, from under 2% in earlier Hour of Code activities to more than 40% of AI activities, suggests critical perspectives are becoming part of introductory AI education.
  • The authors' recommendations imply that a single hour can be better used with unplugged activities, collaborative data work, and novice tools for actually building models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the 'telling vs. doing' finding is to measure learning outcomes: for example, whether students who train a model with Teachable Machine for five minutes can explain how training data shapes behavior better than students who only watch a video.
  • Because the authors coded public activity content rather than classroom implementations, their percentages may overstate actual engagement; observing real classrooms could push the balance even further toward 'telling.'
  • The finding that only two activities integrate all five big ideas suggests a design challenge worth addressing directly: a one-hour activity that touches all five ideas may need to be a structured game or unplugged simulation rather than a screen-based tutorial.
  • A follow-up design study could test whether activities that foreground the energy cost of training models—currently absent from the portfolio—change students' attitudes toward AI's environmental impact.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a content analysis of Hour of Code (HoC) activities related to artificial intelligence and machine learning (AI/ML), focusing on beginner activities for middle and high school students. Using the "five big ideas" framework of Touretzky et al. (2019) and a companion framework for machine learning (Touretzky et al., 2023), the authors coded 47 AI-related activities identified from the HoC website in December 2023 and April 2024. They report that 37 activities engaged with at least one big idea, with perception (83.33%) and learning (75%) most common, followed by natural interaction (41.67%) and societal impact (41.67%), and representation and reasoning least common (13.89%). The paper also describes how ML is introduced (e.g., definitions, training data, algorithms, bias, training/test phases), discusses societal and environmental impact topics, and claims that instructional approaches were much more "telling" than "doing" (hands-on). The discussion offers design recommendations for future introductory AI/ML activities.

Significance. If the findings are accurate, this is a useful and timely empirical map of a widely used outreach program, contributing to the growing literature on K–12 AI literacy. The use of an external framework (Touretzky et al.'s five big ideas) rather than an author-defined scheme is a strength, and the authors are appropriately careful to describe the analysis as exploratory and descriptive rather than causal. The increased attention to societal impact compared with earlier HoC analyses is a notable and policy-relevant observation. However, the paper's central quantitative claims are currently undermined by an internal denominator inconsistency and by the absence of any reported data behind the "telling vs. doing" finding, which limits the verifiability of the main conclusions.

major comments (4)
  1. [Findings, 'How Did AI HoC Activities Address the Five Big Ideas of AI?'] The percentages reported in this section appear to use a denominator of 36, not the stated 37 activities. For example, 30 activities addressing perception would be 81.08% of 37 but 83.33% of 36; similarly 27/36 = 75%, 15/36 = 41.67%, and 5/36 = 13.89%. The manuscript should either correct the number of activities (e.g., 36) or report the correct percentages for 37, because these percentages are the central descriptive result of the paper.
  2. [Analysis and Findings, 'telling vs. doing'] The claim that activities are much more "telling" than "doing" is stated in the Findings, abstract, and Discussion, but the manuscript never defines the categories "hands-on" and "telling," does not specify the unit of coding (e.g., activity, segment, or topic), and provides no counts, table, or per-activity breakdown. As a result, a key finding cannot be verified from the reported evidence. The authors should add an operational definition of these categories, describe how they were applied, and report the number of activities (or idea-instances) coded as telling, hands-on, or both, ideally broken down by big idea.
  3. [Data Collection] The procedure for identifying AI/ML activities from the large HoC repository is underspecified. The text states that the authors "reviewed the short description of each activity to select those mentioning AI/ML," but it does not describe the search or screening criteria, how the 557 (or 542) descriptions were processed, or the specific method by which nine unlabeled activities were found. This lack of detail limits reproducibility and makes it difficult to assess potential selection bias in the resulting 47-activity corpus.
  4. [Discussion, 'The Why'] The statement that "nearly one-third" of AI/ML-related HoC activities addressed societal impact contradicts the reported figure of 41.67% (15 of 36 activities). The text should be corrected to "over two-fifths" or the percentage should be revised.
minor comments (6)
  1. [Title and Abstract] The title asks "What Can Youth Learn ... in One Hour?" but the study analyzes activity materials, not actual learning outcomes. The authors do acknowledge this limitation in the Discussion, but the title and abstract could be more precise (e.g., "What Do Hour of Code Activities Aim to Teach...").
  2. [Findings, first paragraph] The relationship among the numbers 47, 38, 28, 10, and 9 is confusing. The paper initially says 47 AI-related activities were selected, then later reports 38 labeled AI activities (28 with big ideas, 10 without) and 9 unlabeled activities with big ideas (28+9=37). Please clarify the arithmetic and explain why the total corpus is 47 while the analysis of big ideas is based on 37 (or 36).
  3. [Discussion, first paragraph] The phrase "Using the Touretzky and colleagues' (2023) five big ideas" is a citation error: the five big ideas are from Touretzky et al. (2019), while the 2023 reference is the companion paper on machine learning. Please correct this citation.
  4. [Figure 3 and Figure 5] These figures use colors to indicate whether big ideas were incorporated into hands-on activities, telling activities, or both, but the figure captions do not explain the color legend. Since the text refers to these colors, the captions should include an explicit legend or refer to a common color key.
  5. [Discussion, 'Offering Unplugged and Collaborative AI/ML Activities'] The claim that "all the HoC activities were individual in nature" is made without presenting any coding or analysis that would support it. If this was part of the inductive coding, the relevant category and findings should be reported; otherwise, the statement should be softened or removed.
  6. [Discussion, 'The Why'] The comparison between the 41.67% of AI-specific activities addressing societal impact and the less than 2% of all HoC activities in earlier years is not apples-to-apples, because the earlier figure applies to the general HoC catalogue, not to AI-related activities. The authors should note this difference in scope when interpreting the improvement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the content analysis is coded against an external framework, and the authors' self-citations serve only as prior baselines, not as load-bearing inputs.

full rationale

This paper is an empirical content analysis rather than a derivation, so the circularity patterns (fitted input called prediction, ansatz smuggled via citation, etc.) do not apply. The coding scheme is imported from Touretzky et al. (2019) and Touretzky et al. (2023) as external frameworks; the paper's counts (perception 83.33%, learning 75%, natural interaction 41.67%, representation and reasoning 13.89%) are the output of applying that scheme to publicly available activity materials, not quantities defined by the paper's own conclusions. The authors' self-citations (Morales-Navarro et al. 2021a, 2022a, 2022b) are used only as prior baselines for comparison (e.g., less than 2% critical engagement in earlier HoC activities) and as sources of design recommendations, not as the evidence for the current percentages; that prior work is a separate published empirical study and therefore an independent, externally checkable datum. The main vulnerability—that the 'telling vs. doing' finding is asserted without reported counts, an operational definition, or per-activity tallies in the manuscript—is a verifiability and completeness problem, not a circularity problem, because the claim is a summary of the authors' own inductive coding rather than a quantity derived from itself. A separate numerical inconsistency (the percentages imply a denominator of 36 while the text says 37 activities) likewise affects correctness reporting, not circularity. No step in the paper's argument reduces to its own inputs by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numeric free parameters are used; this is a qualitative content analysis. The key presuppositions are the validity of the five big ideas framework, the fidelity of public activity content to the real learning experience, and the accuracy of the collaborative consensus coding. The study does not introduce any new theoretical entities.

assumptions (3)
  • domain assumption The five big ideas framework (Touretzky et al., 2019) is a valid and complete lens for characterizing AI literacy content in learning activities.
    The paper uses this framework deductively to code all activities in the Methods/Analysis section. If a different framework were used, the distribution of topics could look different.
  • domain assumption The publicly available content of each activity (videos, tutorials, teacher guides) is an accurate representation of what learners experience and can learn.
    The coders analyzed activities without observing implementations. This assumption is acknowledged in the Discussion as a limitation, but it is structurally necessary for the content analysis to support claims about what youth can learn.
  • domain assumption The consensus achieved by three researchers collaboratively coding the activities is a reliable measure of the activities' content.
    The authors state they prioritized establishing unanimous agreement on all coding instead of computing inter-rater reliability. This assumes that collaborative discussion effectively removes individual coding errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Can Youth Learn About Artificial Intelligence and Machine Learning in One Hour? Examining How Hour of Code Activities Address the Five Big Ideas of AI." pith.science (2026). https://pith.science/paper/OY6CQMTC

@misc{pith2026241211911,
  author       = {Pith},
  title        = {Pith review of: What Can Youth Learn About Artificial Intelligence and Machine Learning in One Hour? Examining How Hour of Code Activities Address the Five Big Ideas of AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OY6CQMTC}},
  note         = {Machine review of arXiv:2412.11911}
}
read the original abstract

The prominence of artificial intelligence and machine learning in everyday life has led to efforts to foster AI literacy for all K-12 students. In this paper, we review how Hour of Code activities engage with the five big ideas of AI, in particular with machine learning and societal impact. We found that a large majority of activities focus on perception and machine learning, with little attention paid to representation and other topics. A surprising finding was the increased attention paid to critical aspects of computing. However, we also observed a limited engagement with hands-on activities. In the discussion, we address how future introductory activities could be designed to offer a broader array of topics, including the development of tools to introduce novices to artificial intelligence and machine learning and the design of more unplugged and collaborative activities.

Figures

Figures reproduced from arXiv: 2412.11911 by the authors.

Figure 1
Figure 1. Diagram of the 5 big ideas in AI education pro [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Number of AI-related activities in HoC by date. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Distribution of activities by number of big ideas [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: How HoC activities addressed different aspects of [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Machine Vision in Robotics (Virtual) uses bees and [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Data labeling interface in AI for Oceans [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 32 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    H.; Williams, R.; Park, H

    Ali, S.; Payne, B. H.; Williams, R.; Park, H. W.; and Breazeal, C. 2019. Constructionism, ethics, and creativity: Developing primary and middle school artificial intelligence education. In International workshop on education in artificial intelligence k-12 (eduai’19), volume 2, 1--4. mit media lab Palo Alto, California

  4. [4]

    Breazeal, C.; Du, X.; Abelson, H.; Klopfer, E.; and Park, H. W. 2023. Day of AI: Innovating Pedagogical Practices to Bring AI Literacy to Classrooms at Scale. In International Conference on Artificial Intelligence in Education Technology, 267--281. Springer

  5. [5]

    Broll, B.; and Grover, S. 2023. Beyond black-boxes: Teaching complex machine learning ideas through scaffolded interactive activities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 15990--15998

  6. [6]

    Carney, M.; Webster, B.; Alvarado, I.; Phillips, K.; Howell, N.; Griffith, J.; Jongejan, J.; Pitaru, A.; and Chen, A. 2020. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI EA '20, 1–8. New York, NY, USA: Association for...

  7. [7]

    Code.org. 2024. Hour of Code 2023 Live Stream with Hadi Partovi

  8. [8]

    Costanza-Chock, S. 2020. Design justice: Community-led practices to build the worlds we need. The MIT Press

Show all 37 references
  1. [9]

    C.; Pacheco, F

    Gresse von Wangenheim, C.; Hauck, J. C.; Pacheco, F. S.; and Bertonceli Bueno, M. F. 2021. Visual tools for teaching machine learning in K-12: A ten-year systematic mapping. Education and Information Technologies, 26(5): 5733--5778

  2. [10]

    Hardy, L.; Dixon, C.; and Hsi, S. 2020. From data collectors to data producers: Shifting students’ relationship to data. Journal of the Learning Sciences, 29(1): 104--126

  3. [11]

    Hollands, F.; and Breazeal, C. 2024. Establishing AI Literacy before Adopting AI. The Science Teacher, 91(2): 35--42

  4. [12]

    Hornberger, M.; Bewersdorff, A.; and Nerdel, C. 2023. What do university students know about Artificial Intelligence? Development and validation of an AI literacy test. Computers and Education: Artificial Intelligence, 5: 100165

  5. [13]

    Judd, S. 2020. Activities for building understanding: How AI4ALL teaches AI to diverse high school students. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, 633--634

  6. [14]

    Kafai, Y.; and Harel, I. 1991. Constructionism, 111--140. Praeger

  7. [15]

    Lim, H.; Min, W.; Vandenberg, J.; Catet \'e , V.; and Mott, B. 2024. Unplugged K-12 AI Learning: Exploring Representation and Reasoning with a Facial Recognition Game. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 23285--23293

  8. [16]

    Lindner, A.; Seegerer, S.; and Romeike, R. 2019. Unplugged Activities in the Context of AI. In Informatics in Schools. New Ideas in School Informatics: 12th International Conference on Informatics in Schools: Situation, Evolution, and Perspectives, ISSEP 2019, Larnaca, Cyprus,...

  9. [17]

    Long, D.; Blunt, T.; and Magerko, B. 2021. Co-designing AI literacy exhibits for informal learning spaces. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2): 1--35

  10. [18]

    Long, D.; and Magerko, B. 2020. What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI conference on human factors in computing systems, 1--16

  11. [19]

    S.; Viguier, S.; and Ligozat, A.-L

    Luccioni, A. S.; Viguier, S.; and Ligozat, A.-L. 2023. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253): 1--15

  12. [20]

    McDonald, N.; Schoenebeck, S.; and Forte, A. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proc. ACM Hum.-Comput. Interact., 3(CSCW)

  13. [21]

    Morales-Navarro, L.; and Kafai, Y. 2024. Unpacking Approaches to Learning and Teaching Machine Learning in K-12 Education: Transparency, Ethics, and Design Activities. In Proceedings of the 19th Workshop in Primary and Secondary Computing Education, 1--6

  14. [22]

    Morales-Navarro, L.; Kafai, Y.; Jayathirtha, G.; and Shaw, M. 2021 a . Investigating Creative and Critical Engagement with Computing in the Hour of Code (Practical Report). In Proceedings of the 16th Workshop in Primary and Secondary Computing Education, WiPSCE '21. New York, ...

  15. [23]

    Morales-Navarro, L.; Kafai, Y.; Jayathirtha, G.; and Shaw, M. 2021 b . Investigating creative and critical engagement with computing in the Hour of Code (Practical report). In Proceedings of the 16th Workshop in Primary and Secondary Computing Education, 1--6

  16. [24]

    B.; and Gregory, K

    Morales-Navarro, L.; Kafai, Y. B.; and Gregory, K. T. 2022 a . Is Computational Empowerment Promoted in the Hour of Code? Analyzing Opportunities for Conceptual, Creative, and Critical Engagement in the Design of Introductory Computing Activities. In Proceedings of the 22nd Ko...

  17. [25]

    B.; and Gregory, K

    Morales-Navarro, L.; Kafai, Y. B.; and Gregory, K. T. 2022 b . Is computational empowerment promoted in the hour of code? Analyzing opportunities for conceptual, creative, and critical engagement in the design of introductory computing activities. In Proceedings of the 22nd Ko...

  18. [26]

    Oleson, A.; Wortzman, B.; and Ko, A. J. 2021. On the Role of Design in K-12 Computing Education. ACM Trans. Comput. Educ., 21(1)

  19. [27]

    Salda \ n a, J. 2021. The coding manual for qualitative researchers. The Coding Manual for Qualitative Researchers

  20. [28]

    Skjuve, M.; F lstad, A.; and Brandtzaeg, P. B. 2023. The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users. In Proceedings of the 5th International Conference on Conversational User Interfaces, CUI '23. New York, NY, USA: Association for Computing ...

  21. [29]

    Solyst, J.; Yang, E.; Xie, S.; Hammer, J.; Ogan, A.; and Eslami, M. 2024. Children's Overtrust and Shifting Perspectives of Generative AI. arXiv preprint arXiv:2404.14511

  22. [30]

    Tedre, M.; Denning, P.; and Toivonen, T. 2021. CT 2.0. In Proceedings of the 21st Koli Calling International Conference on Computing Education Research, 1--8

  23. [31]

    Touretzky, D.; Gardner-McCune, C.; Martin, F.; and Seehorn, D. 2019. Envisioning AI for K-12: What should every child know about AI? In Proceedings of the AAAI conference on artificial intelligence, volume 33, 9795--9799

  24. [32]

    Touretzky, D.; Gardner-McCune, C.; and Seehorn, D. 2023. Machine learning and the five big ideas in AI. International Journal of Artificial Intelligence in Education, 33(2): 233--266

  25. [33]

    J.; Morales-Navarro, L.; Chen, J

    Tseng, T.; Davidson, M. J.; Morales-Navarro, L.; Chen, J. K.; Delaney, V.; Leibowitz, M.; Beason, J.; and Shapiro, R. B. 2024. Co-ML: Collaborative machine learning model building for developing dataset design practices. ACM Transactions on Computing Education, 24(2): 1--37

  26. [34]

    Vee, A. 2017. Coding literacy: How computer programming is changing writing. Mit Press

  27. [35]

    Waite, J.; and Sentance, S. 2021. Teaching programming in schools: A review of approaches and strategies. Raspberry Pi Foundation

  28. [36]

    W.; and Breazeal, C

    Williams, R.; Park, H. W.; and Breazeal, C. 2019. A is for Artificial Intelligence: The Impact of Artificial Intelligence Activities on Young Children's Perceptions of Robots. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, 1–11. New Y...

  29. [37]

    Hour of Code

    Yauney, J.; Bartholomew, S. R.; and Rich, P. 2023. A systematic review of “Hour of Code” research. Computer Science Education, 33(4): 512--544

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.