pith. sign in

arxiv: 2603.26906 · v2 · pith:77OI6OBKnew · submitted 2026-03-27 · 💻 cs.HC

KI-Adventskalender: An Informal Learning Intervention for Data & AI Literacy

Rahul Sharma , Lars Henrich , Larisa Ivanova , Arsalan Karimzadmotallebiazar , Annette Bieniusa , Leo Van Waveren , Sebastian Vollmer This is my paper

Pith reviewed 2026-05-25 06:35 UTC · model grok-4.3

classification 💻 cs.HC
keywords AI literacydata literacyinformal learningadvent calendarsecondary educationmicro-challengesstudent engagementextracurricular activity
0
0 comments X

The pith

A December calendar of short daily AI micro-challenges grew participation among secondary students and showed most who passed the midpoint finishing the full set.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents KI-Adventskalender, a free web tool that releases one short guided task each day in December on topics such as data quality and how modeling choices shape AI outputs. It reports that the second year saw substantially higher sign-ups than the first, that drop-off was heaviest early on, and that more than three-quarters of those who reached day 12 completed every remaining task. Revision counts rose alongside pass rates across competence areas, which the authors read as a sign of continued effort rather than one-time attempts. These platform patterns are offered as the basis for arguing that future work needs embedded assessments and mixed-method designs to move beyond counts of activity to evidence of actual learning.

Core claim

The authors built and ran two annual versions of a 24-day online advent calendar containing curated micro-challenges on data-centric AI competencies and socio-technical themes. Participation rose markedly in the second year. Among 2025 users who reached day 12, over 75 percent completed the calendar. Shifts appeared in how different competence clusters performed, and higher revision activity coincided with stronger pass rates, taken as an indicator of sustained engagement. The observations are used to motivate tighter task instrumentation, embedded micro-assessments, and mixed-method studies that can separate persistence from conceptual uptake.

What carries the argument

KI-Adventskalender, a sequence of 24 short daily micro-challenges released online in December that target interconnected data quality, evaluation, and modeling concepts in AI systems.

If this is right

  • The daily micro-challenge format can attract growing numbers of secondary students to data and AI topics over successive years.
  • Most attrition occurs before the midpoint, after which completion becomes the norm for those who continue.
  • Revision activity that tracks with higher pass rates can be read as evidence of repeated effort rather than superficial attempts.
  • Platform traces alone leave open whether observed activity produces lasting changes in how students reason about data and AI.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same daily-release structure might be adapted to other domains such as statistics or environmental data to test whether the engagement pattern repeats.
  • Embedding a single quick question at the end of each task could give an immediate signal of whether users understood the intended concept.
  • Schools could experiment with assigning the calendar as an optional December activity and track whether voluntary participation predicts later interest in formal AI courses.

Load-bearing premise

Platform records of who started, revised, and passed tasks can serve as reliable signs of sustained engagement and actual understanding without separate checks such as interviews or controlled tests.

What would settle it

A controlled follow-up that compares pre- and post-test scores or interview responses between calendar completers and a matched group of non-participants and finds no measurable difference in grasp of data-evaluation concepts would show the participation patterns do not reflect learning gains.

Figures

Figures reproduced from arXiv: 2603.26906 by Annette Bieniusa, Arsalan Karimzadmotallebiazar, Larisa Ivanova, Lars Henrich, Leo Van Waveren, Rahul Sharma, Sebastian Vollmer.

Figure 1
Figure 1. Figure 1: KI-Adventskalender, 2025 Secondary school students increasingly encounter AI systems whose outputs depend on data quality, evaluation choices and modeling assumptions. To provide accessible entry points to these interconnected concepts, we developed KI-Adventskalender, a free web￾based extracurricular initiative with 24 didactically curated, short, guided micro-challenges released daily in December, target… view at source ↗
Figure 2
Figure 2. Figure 2: Cumulative relative active users(a) and progression (b) for both runs of KI-Adventskalender [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean attempts (a) and rate of passing the challenge (b) across competence clusters for both runs of KI-Adventskalender [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Complexity vs. Success Matrix: Comparison of weighted mean attempts and pass rates by topic category for the 2024 and 2025 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
read the original abstract

Secondary school students increasingly encounter AI systems whose outputs depend on data quality, evaluation choices and modeling assumptions. To provide accessible entry points to these interconnected concepts, we developed KI-Adventskalender, a free web-based extracurricular initiative with 24 didactically curated, short, guided micro-challenges released daily in December, targeting data-centric competencies and socio-technical themes that shape how data are interpreted in practice. Drawing on two annual iterations, we report aggregate platform traces characterizing participation and task-level engagement. Participation increased substantially in 2025, but early attrition persists. Progression stabilized after midpoint: among users reaching Day 12 in 2025, more than 75% completed the calendar. Competence cluster performance shifted across years; higher revision rates co-occurred with strong pass rates, suggesting sustained engagement. We use these observations to motivate a next-step measurement agenda: tighter task instrumentation, embedded micro-assessments and mixed-method evaluation designs that can distinguish persistence from conceptual uptake, knowledge progression and durable learning outcomes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript describes KI-Adventskalender, a free web-based extracurricular intervention consisting of 24 daily guided micro-challenges targeting data-centric competencies and socio-technical themes in AI for secondary school students. Drawing on usage logs from two annual iterations, it reports increased participation in 2025, persistent early attrition, stabilization of progression after the midpoint (with >75% of users reaching Day 12 completing the calendar in 2025), shifts in competence cluster performance across years, and the co-occurrence of higher revision rates with strong pass rates as suggestive of sustained engagement. These observations are presented to motivate a future measurement agenda involving tighter instrumentation, embedded micro-assessments, and mixed-method designs to distinguish persistence from conceptual uptake.

Significance. If the reported platform traces are taken at face value as descriptive observations, the work supplies baseline engagement data on an informal AI literacy intervention. This can inform the design of similar micro-challenge formats in HCI and educational technology, particularly the noted attrition patterns and the call for distinguishing persistence metrics from learning outcomes. The explicit framing as observational rather than validated proxy evidence is a strength.

major comments (2)
  1. [Abstract and results section] Abstract and results section: the statement that 'higher revision rates co-occurred with strong pass rates, suggesting sustained engagement' is presented without any operational definition of revision rate, pass rate, or the criterion for 'strong' performance, nor any quantitative measure of co-occurrence (e.g., correlation coefficient or contingency table). This interpretive step is load-bearing for the engagement claim even though the paper ultimately motivates better measurement.
  2. [Methods or results section] Methods or results section: no information is given on how users are counted (e.g., unique accounts vs. sessions), exclusion criteria for incomplete logs, or the precise definition of 'completion' and 'reaching Day 12.' These definitions are required to interpret the attrition and progression statistics that form the core descriptive claims.
minor comments (2)
  1. [Results] A summary table comparing key metrics (participation counts, completion percentages, revision rates) across the two iterations would improve readability and allow direct year-over-year comparison.
  2. [Abstract and introduction] The manuscript should explicitly state the calendar years of the two iterations (presumably 2024 and 2025) in the abstract and introduction for clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments and the recommendation of minor revision. The feedback highlights areas where additional methodological clarity will strengthen the descriptive claims. We address each point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract and results section] Abstract and results section: the statement that 'higher revision rates co-occurred with strong pass rates, suggesting sustained engagement' is presented without any operational definition of revision rate, pass rate, or the criterion for 'strong' performance, nor any quantitative measure of co-occurrence (e.g., correlation coefficient or contingency table). This interpretive step is load-bearing for the engagement claim even though the paper ultimately motivates better measurement.

    Authors: We agree that the current phrasing lacks explicit operational definitions and quantitative support. In the revised manuscript we will add precise definitions (revision rate as the average number of resubmissions per task; pass rate as the proportion of tasks passed on the first attempt; 'strong' performance as pass rates above 80%) and include a quantitative measure of co-occurrence, such as a Pearson correlation or contingency table between revision counts and pass rates. This will be placed in the results section while preserving the observational framing and the call for improved future instrumentation. revision: yes

  2. Referee: [Methods or results section] Methods or results section: no information is given on how users are counted (e.g., unique accounts vs. sessions), exclusion criteria for incomplete logs, or the precise definition of 'completion' and 'reaching Day 12.' These definitions are required to interpret the attrition and progression statistics that form the core descriptive claims.

    Authors: We acknowledge the omission of these operational details. The revised Methods section will include a new subsection specifying: (1) user counting via unique registered accounts (sessions without login are excluded); (2) exclusion criteria (logs with zero task attempts are removed); (3) 'completion' as having submitted a response for all 24 days; and (4) 'reaching Day 12' as having submitted at least one response on or before Day 12. These clarifications will be added without altering the reported statistics. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper is a purely observational descriptive report of aggregate platform traces (participation rates, attrition, revision counts, pass rates) from two iterations of an educational intervention. It contains no equations, fitted parameters, derivations, or predictions that reduce to inputs by construction. The central claims are limited to reporting observed patterns and explicitly motivating a future measurement agenda to distinguish persistence from conceptual uptake, without asserting that the traces serve as validated proxies. No self-citations are load-bearing for any premise, and the analysis does not rename known results or smuggle in ansatzes.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The reported observations rest on the assumption that digital engagement metrics reflect meaningful educational processes; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Platform usage logs accurately reflect user engagement and learning progress
    The abstract uses completion rates, revision rates, and pass rates to infer sustained engagement without additional validation.

pith-pipeline@v0.9.0 · 5729 in / 1220 out tokens · 43230 ms · 2026-05-25T06:35:32.810697+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages

  1. [1]

    Laura Kristen Allen and Panayiota Kendeou. 2024. ED-AI Lit: An interdisciplinary framework for AI literacy in education.Policy Insights from the Behavioral and Brain Sciences11, 1 (2024), 3–10

    work page 2024

  2. [2]

    Maalvika Bhat and Duri Long. 2024. Designing Interactive Explainable AI Tools for Algorithmic Literacy and Transparency. InProceedings of the 2024 ACM Designing Interactive Systems Conference(Copenhagen, Denmark)(DIS ’24). Association for Computing Machinery, New York, NY, USA, 939–957. doi:10.1145/3643834.3660722

    work page doi:10.1145/3643834.3660722 2024

  3. [3]

    Rensink, Enrico Bertini, and Jean-Daniel Fekete

    Jeremy Boy, Ronald A. Rensink, Enrico Bertini, and Jean-Daniel Fekete. 2014. A Principled Way of Assessing Visualization Literacy.IEEE Transactions on Visualization and Computer Graphics20, 12 (2014), 1963–1972. doi:10.1109/TVCG.2014.2346984

    work page doi:10.1109/tvcg.2014.2346984 2014

  4. [4]

    Erik Brockbank, Arnav Verma, Hannah Lloyd, Holly Huey, Lace Padilla, and Judith E. Fan. 2025. Evaluating convergence between two data visualization literacy assessments.Cognitive Research: Principles and Implications10, 1 (April 2025). doi:10.1186/s41235-025-00622-9

    work page doi:10.1186/s41235-025-00622-9 2025

  5. [5]

    Lorena Casal-Otero, Alejandro Catala, Carmen Fernández-Morante, Maria Taboada, Beatriz Cebreiro, and Senén Barro. 2023. AI literacy in K-12: a systematic literature review.International Journal of STEM Education10, 1 (2023), 29

    work page 2023

  6. [6]

    Ge, Yiren Ding, Fumeng Yang, Lane Harrison, and Matthew Kay

    Yuan Cui, Lily W. Ge, Yiren Ding, Fumeng Yang, Lane Harrison, and Matthew Kay. 2024. Adaptive Assessment of Visualization Literacy.IEEE Transactions on Visualization and Computer Graphics30, 1 (2024), 628–637. doi:10.1109/TVCG.2023.3327165

    work page doi:10.1109/tvcg.2023.3327165 2024

  7. [7]

    Kientz, Jason Yip, and Caroline Pitt

    Aayushi Dangol, Michele Newman, Robert Wolfe, Jin Ha Lee, Julie A. Kientz, Jason Yip, and Caroline Pitt. 2024. Mediating Culture: Cultivating Socio-cultural Understanding of AI in Children through Participatory Design. InProceedings of the 2024 ACM Designing Interactive Systems Conference (Copenhagen, Denmark)(DIS ’24). Association for Computing Machinery...

    work page doi:10.1145/3643834.3661515 2024

  8. [8]

    Payne, and Cynthia Breazeal

    Daniella DiPaola, Blakeley H. Payne, and Cynthia Breazeal. 2020. Decoding design agendas: an ethical design activity for middle school students. In Proceedings of the Interaction Design and Children Conference(London, United Kingdom)(IDC ’20). Association for Computing Machinery, New York, NY, USA, 1–10. doi:10.1145/3392063.3394396

    work page doi:10.1145/3392063.3394396 2020

  9. [9]

    Stefania Druga, Jason Yip, Michael Preston, and Devin Dillon. 2021. The 4As: Ask, adapt, author, analyze-AI literacy framework for families. Algorithmic rights and protections for children(2021)

    work page 2021

  10. [10]

    Ge, Yuan Cui, and Matthew Kay

    Lily W. Ge, Yuan Cui, and Matthew Kay. 2023. CALVI: Critical Thinking Assessment for Literacy in Visualizations. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 815, 18 pages. doi:10.1145/3544548.3581406 Manuscript submitted to ACM KI...

    work page doi:10.1145/3544548.3581406 2023

  11. [11]

    Robert Gould. 2017. Data literacy is statistical literacy.Statistics Education Research Journal16, 1 (2017), 22–25

    work page 2017

  12. [12]

    Maryam Hedayati and Matthew Kay. 2025. What University Students Learn In Visualization Classes.IEEE Transactions on Visualization and Computer Graphics31, 1 (2025), 1072–1082. doi:10.1109/TVCG.2024.3456291

    work page doi:10.1109/tvcg.2024.3456291 2025

  13. [13]

    Holly Huey, Lauren A Oey, Hannah Lloyd, and Judith E Fan. 2023. How do communicative goals guide which data visualizations people think are effective?. InProceedings of the annual meeting of the cognitive science society, Vol. 45

    work page 2023

  14. [14]

    Sushmita Khan, Mehtab Iqbal, Oluwafemi Osho, Khushbu Singh, Kyra Derrick, Philip Nelson, Lingyuan Li, Emily Sidnam-Mauch, Nicole Bannister, Kelly Caine, and Bart Knijnenburg. 2024. Teaching Middle Schoolers about the Privacy Threats of Tracking and Pervasive Personalization: A Classroom Intervention Using Design-Based Research. InProceedings of the 2024 C...

    work page doi:10.1145/3613904.3642460 2024

  15. [15]

    Matthias Carl Laupichler, Alexandra Aster, Jana Schirch, and Tobias Raupach. 2022. Artificial intelligence literacy in higher and adult education: A scoping literature review.Computers and Education: Artificial Intelligence3 (2022), 100101. doi:10.1016/j.caeai.2022.100101

    work page doi:10.1016/j.caeai.2022.100101 2022

  16. [16]

    Irene Lee, Safinah Ali, Helen Zhang, Daniella DiPaola, and Cynthia Breazeal. 2021. Developing Middle School Students’ AI Literacy. InProceedings of the 52nd ACM Technical Symposium on Computer Science Education(Virtual Event, USA)(SIGCSE ’21). Association for Computing Machinery, New York, NY, USA, 191–197. doi:10.1145/3408877.3432513

    work page doi:10.1145/3408877.3432513 2021

  17. [17]

    Tomáš Lintner. 2024. A systematic review of AI literacy scales.npj Science of Learning9, 1 (2024), 50

    work page 2024

  18. [18]

    Duri Long and Brian Magerko. 2020. What is AI Literacy? Competencies and Design Considerations. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–16. doi:10.1145/3313831.3376727

    work page doi:10.1145/3313831.3376727 2020

  19. [19]

    Mladen Raković, Dragan Gašević, Saeed Ul Hassan, José A Ruipérez Valiente, Naif Aljohani, and Sandra Milligan. 2023. Learning analytics and assessment: Emerging research trends, promises and future opportunities.British Journal of Educational Technology(2023)

    work page 2023

  20. [20]

    Chantel Ridsdale, James Rothwell, Mike Smit, Hossam Ali-Hassan, Michael Bliemel, Dean Irvine, Daniel Kelley, Stan Matwin, and Brad Wuetherick

    work page

  21. [21]

    Strategies and best practices for data literacy education.Knowledge synthesis report(2015)

    work page 2015

  22. [22]

    Iris Heung Yue Yim. 2024. A critical review of teaching and learning artificial intelligence (AI) literacy: Developing an intelligence-based AI literacy framework for primary school education.Computers and Education: Artificial Intelligence7 (2024), 100319. doi:10.1016/j.caeai.2024.100319 Manuscript submitted to ACM

    work page doi:10.1016/j.caeai.2024.100319 2024