pith. sign in

arxiv: 2606.25358 · v1 · pith:5NIYFI2Mnew · submitted 2026-06-24 · 💻 cs.AI · cs.MA

Agentic Knowledge Tracing: A Multi-Agent LLM Architecture for Stealth Assessment of Financial Literacy in Serious Games

Gabriel Santos , Rita Julia , Marcelo Nascimento This is my paper

Pith reviewed 2026-06-25 21:21 UTC · model grok-4.3

classification 💻 cs.AI cs.MA
keywords financial literacyserious gamesstealth assessmentmulti-agent LLMBayesian Knowledge TracingOECD/INFE frameworkeducational games
0
0 comments X

The pith

A multi-agent LLM pipeline triples the predictive validity of assessing financial literacy from open-ended gameplay compared to single-model baselines.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents the Agentic BKT pipeline, a four-phase multi-agent architecture that logs player decisions in a 2D platformer game, classifies actions on a financial literacy rubric, routes them to domain-specific agents, applies Bayesian knowledge tracing per competency, and synthesizes an overall mastery score. Evaluated on 193 K-12 students across 264 sessions, the estimates correlate with learning gains and post-test scores but not pre-tests. The multi-agent design with session-level reasoning approximately triples the correlation strength achieved by a single LLM baseline. This shows that decomposing financial literacy into risk, investing, spending, and credit domains enables stealth assessment that captures its multidimensional structure from gameplay traces.

Core claim

The Agentic BKT pipeline processes structured event logs from gameplay through an LLM classifier (Fleiss kappa 0.624 against experts), four domain agents performing session-level reasoning, per-competency Bayesian Knowledge Tracing, and a judge agent; the resulting mastery scores from 193 participants correlate with learning gain (r=0.276) and post-test scores (r=0.333) while showing no pre-test correlation, tripling the validity of a single-LLM baseline (r=0.095, nonsignificant) and demonstrating the value of domain decomposition for multidimensional competencies.

What carries the argument

The Agentic BKT pipeline: an LLM event classifier feeding four domain-specific agents (risk mitigation, investing, spending, credit management) that perform session-level reasoning into per-competency Bayesian Knowledge Tracing, followed by a judge agent for overall synthesis.

Load-bearing premise

The LLM-generated labels on the four-point rubric accurately reflect the OECD/INFE financial competencies when applied to open-ended gameplay events.

What would settle it

If independent human experts re-label the same gameplay events on the identical rubric and the resulting mastery estimates lose significant correlation with post-test scores, the claim that the pipeline validly measures financial competencies would not hold.

Figures

Figures reproduced from arXiv: 2606.25358 by Gabriel Santos, Marcelo Nascimento, Rita Julia.

Figure 1
Figure 1. Figure 1: Game overview showing the platformer environment. The HUD [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Casino area and credit card interface. The player stands in front of the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Architecture of the Agentic BKT pipeline. Phase 1 captures structured game events. Phase 2 classifies each event on a four-point scale using GPT [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows the Agentic BKT scatter plots against all three test measures, illustrating the significant positive trend with learning gain and post-test alongside the expected null relationship with pre-test. For comparison, the single-LLM baseline scatter plots are shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Single-LLM BKT baseline P(mastery) vs. learning gain (left), pre-test (center), and post-test (right). No correlations reach significance, illustrating the limitation of event-level classification without domain decomposition. (SD = 0.036, with a ceiling near 1.0), likely reflecting that most players made broadly reasonable investment choices, limiting the agent’s discriminative power. These per-domain res… view at source ↗
read the original abstract

Assessing financial literacy during gameplay without disrupting the learning experience remains a key challenge in serious games for education. We present the Agentic BKT pipeline, a multi-agent large language model architecture for stealth assessment of financial competencies from open-ended gameplay events. The pipeline processes events from a 2D platformer serious game aligned with the OECD/INFE financial literacy framework through four phases: (1) the game captures every player decision as a structured event log; (2) an LLM event classifier labels each action on a four-point rubric validated against three domain experts (Fleiss kappa = 0.624, substantial agreement); (3) four domain-specific agents specializing in risk mitigation, investing, spending, and credit management perform session-level reasoning over behavioral trajectories, feeding per-competency Bayesian Knowledge Tracing that estimates mastery within each domain; and (4) an expert judge agent synthesizes the domain-level estimates into an overall mastery score. Evaluated with 193 K-12 participants across 264 game sessions, the Agentic BKT pipeline yields mastery estimates significantly correlated with learning gain (r = 0.276, p = 0.0001) and post-test scores (r = 0.333, p < 0.0001) while showing no correlation with pre-test scores, providing both convergent and discriminant validity. The multi-agent approach approximately triples the predictive validity of a single-LLM baseline (r = 0.095, not significant) in this study, demonstrating that domain decomposition and session-level reasoning play a central role in capturing the multidimensional nature of financial literacy from gameplay

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes the Agentic BKT pipeline, a multi-agent LLM architecture for stealth assessment of financial literacy from open-ended gameplay in a 2D platformer serious game aligned with the OECD/INFE framework. The pipeline includes event capture, LLM-based classification on a four-point rubric (validated with Fleiss' kappa = 0.624 against three experts), session-level reasoning by four domain-specific agents (risk mitigation, investing, spending, credit management), per-competency Bayesian Knowledge Tracing, and synthesis by an expert judge agent. In an evaluation with 193 K-12 participants across 264 sessions, the pipeline's mastery estimates correlate with learning gain (r = 0.276, p = 0.0001) and post-test scores (r = 0.333, p < 0.0001), show no correlation with pre-test scores, and approximately triple the validity of a single-LLM baseline (r = 0.095, ns).

Significance. If the event labels are reliable, this work demonstrates that decomposing financial literacy into domain-specific agents combined with BKT can yield mastery estimates with meaningful convergent and discriminant validity in serious games. The empirical results against independent external measures (post-test and learning gain) provide a solid basis for the claim that multi-agent session-level reasoning improves upon single-LLM approaches. This has potential significance for AI-driven educational assessment, particularly in capturing multidimensional competencies without disrupting gameplay. The use of BKT on top of LLM reasoning is a notable integration of traditional educational data mining with modern LLM capabilities.

major comments (2)
  1. [Abstract] Abstract: The description of the rubric validation states only that it was 'validated against three domain experts (Fleiss kappa = 0.624, substantial agreement)' without reporting the number of events or items rated, whether real gameplay event logs were used in the validation, or agreement broken down by the four competencies. This information is critical because the downstream correlations with learning gain and post-test scores rest on the assumption that the LLM classifier accurately applies the OECD/INFE competencies to open-ended gameplay actions.
  2. [Abstract] Abstract: Details on the single-LLM baseline are limited; it is not specified whether the baseline used identical event processing, the same four-point rubric, or equivalent prompting to the multi-agent system, which affects the interpretation that domain decomposition 'approximately triples' the predictive validity (r = 0.095 vs. r = 0.333).
minor comments (1)
  1. [Abstract] The abstract mentions p = 0.0001 and p < 0.0001; consider standardizing the reporting format for consistency across results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting areas where the abstract could better support the core claims. We address each point below and will revise the manuscript to incorporate the requested details.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The description of the rubric validation states only that it was 'validated against three domain experts (Fleiss kappa = 0.624, substantial agreement)' without reporting the number of events or items rated, whether real gameplay event logs were used in the validation, or agreement broken down by the four competencies. This information is critical because the downstream correlations with learning gain and post-test scores rest on the assumption that the LLM classifier accurately applies the OECD/INFE competencies to open-ended gameplay actions.

    Authors: We agree the abstract is too concise on this point. The Methods section of the full manuscript details the validation procedure, which used real gameplay event logs sampled from the study sessions. We will revise the abstract to report the number of events rated, confirm the use of authentic logs, and include per-competency agreement statistics where they were computed. This change will directly address the concern about the reliability of the classifier underpinning the reported correlations. revision: yes

  2. Referee: [Abstract] Abstract: Details on the single-LLM baseline are limited; it is not specified whether the baseline used identical event processing, the same four-point rubric, or equivalent prompting to the multi-agent system, which affects the interpretation that domain decomposition 'approximately triples' the predictive validity (r = 0.095 vs. r = 0.333).

    Authors: We concur that the abstract should make the baseline comparison more transparent. The single-LLM baseline applied the identical event capture pipeline and four-point rubric, with prompting limited to direct classification without domain-specific agents or session-level reasoning. We will revise the abstract (and expand the corresponding Methods paragraph) to explicitly state these design equivalences, enabling readers to evaluate the contribution of the multi-agent decomposition. revision: yes

Circularity Check

0 steps flagged

No significant circularity; central results are empirical correlations to independent external measures

full rationale

The paper reports correlations (r=0.333 post-test, r=0.276 learning gain) between Agentic BKT mastery estimates and separate post-test/learning-gain instruments. These quantities are not defined by the model's own parameters or equations. The upstream LLM classifier applies a rubric validated against three external domain experts (Fleiss κ=0.624); BKT is invoked as a standard method. No self-definitional equations, fitted-input predictions, self-citation load-bearing premises, or ansatz smuggling appear in the derivation. The multi-agent improvement over single-LLM baseline is likewise an empirical comparison, not a definitional identity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 2 invented entities

The pipeline rests on standard BKT assumptions and an expert-validated rubric; no free parameters are explicitly fitted in the abstract beyond the reported correlations.

axioms (2)
  • domain assumption The four-point rubric and LLM labels accurately capture OECD/INFE financial competencies from gameplay events
    Invoked in phase 2 of the pipeline and used to justify the classifier
  • domain assumption Bayesian Knowledge Tracing models can be applied to per-competency estimates produced by LLM agents
    Core mechanism in phase 3
invented entities (2)
  • Four domain-specific LLM agents (risk mitigation, investing, spending, credit management) no independent evidence
    purpose: Perform session-level reasoning over behavioral trajectories before feeding BKT
    New architectural component introduced to decompose the multidimensional construct
  • Expert judge agent no independent evidence
    purpose: Synthesize domain-level BKT estimates into overall mastery score
    Final aggregation step in phase 4

pith-pipeline@v0.9.1-grok · 5827 in / 1592 out tokens · 23007 ms · 2026-06-25T21:21:19.154900+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 3 canonical work pages

  1. [1]

    OECD/INFE 2020 international survey of adult financial literacy,

    OECD, “OECD/INFE 2020 international survey of adult financial literacy,” OECD, Paris, Tech. Rep.,

    2020

  2. [2]

    Available: https://www.oecd.org/financial/education/ oecd-infe-2020-international-survey-of-adult-financial-literacy.pdf

    [Online]. Available: https://www.oecd.org/financial/education/ oecd-infe-2020-international-survey-of-adult-financial-literacy.pdf

    2020

  3. [3]

    ——,PISA 2022 Results (Volume IV): How Financially Smart Are Students?, ser. PISA. Paris: OECD Publishing, 2024

    2022

  4. [4]

    A review of financial-literacy education programs for children and adolescents,

    A. Amagir, W. Groot, H. M. van den Brink, and A. Wilschut, “A review of financial-literacy education programs for children and adolescents,” Citizenship, Social and Economics Education, vol. 17, no. 1, pp. 56–80, 2018

    2018

  5. [5]

    Aprea, J

    C. Aprea, J. Schultheis, and K. Stolle,Instructional Integration of Digital Learning Games in Financial Literacy Education, 11 2017

    2017

  6. [6]

    Development and pilot testing of a financial literacy game for young adults: The Happy Life Game,

    A. M. Pf ¨andler, “Development and pilot testing of a financial literacy game for young adults: The Happy Life Game,” inGame-based Learning Across the Disciplines, ser. Advances in Game-Based Learning, C. Aprea and D. Ifenthaler, Eds. Cham: Springer, 2021, pp. 59–84

    2021

  7. [7]

    Financial literacy games—increasing utility value by instructional design in upper secondary education,

    J. Platzet al., “Financial literacy games—increasing utility value by instructional design in upper secondary education,”Education Sciences, vol. 15, no. 2, p. 227, 2025

    2025

  8. [8]

    Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow,

    V . J. Shute, M. Ventura, M. I. Bauer, and D. Zapata-Rivera, “Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow,” inSerious Games: Mechanisms and Effects, U. Ritterfeld, M. Cody, and P. V orderer, Eds. Routledge, 2009, pp. 295–321

    2009

  9. [9]

    Stealth assessment in computer-based games to support learning,

    V . J. Shute, “Stealth assessment in computer-based games to support learning,” inComputer Games and Instruction, S. Tobias and J. D. Fletcher, Eds. Charlotte, NC: Information Age Publishers, 2011, pp. 503–524

    2011

  10. [10]

    V . J. Shute and M. Ventura,Measuring and Supporting Learning in Games: Stealth Assessment. Cambridge, MA: MIT Press, 2013

    2013

  11. [11]

    Knowledge tracing: Modeling the acquisition of procedural knowledge,

    A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,”User Modeling and User-Adapted Interaction, vol. 4, no. 4, pp. 253–278, 1995

    1995

  12. [12]

    Towards modeling learner performance with large language models,

    R. Schmuckeret al., “Towards modeling learner performance with large language models,”arXiv preprint arXiv:2403.14661, 2024. [Online]. Available: https://arxiv.org/abs/2403.14661

    work page arXiv 2024

  13. [13]

    Exploring knowledge tracing in tutor-student dialogues using LLMs,

    A. Scarlatos, R. S. Baker, and A. Lan, “Exploring knowledge tracing in tutor-student dialogues using LLMs,” inProceedings of the 15th International Learning Analytics and Knowledge Conference (LAK 2025). Dublin, Ireland: ACM, 2025

    2025

  14. [14]

    LLM agents for education: Advances and applications,

    X. Wanget al., “LLM agents for education: Advances and applications,” arXiv preprint arXiv:2503.11733, 2025. [Online]. Available: https: //arxiv.org/abs/2503.11733

    work page arXiv 2025

  15. [15]

    A behavioral science approach to financial literacy games,

    J. Corbin, J. Dalton, R. Bhatt, and D. Ariely, “A behavioral science approach to financial literacy games,” Center for Advanced Hindsight, Duke University / GBH NOV A, 2022. [Online]. Available: https:// advanced-hindsight.com/wp-content/uploads/2022/03/CAH-NOV A.pdf

    2022

  16. [16]

    FinCraft: Immersive personalised persuasive serious games for financial literacy among young decision-makers,

    A. Rasco, J. Chan, G. Peko, and D. Sundaram, “FinCraft: Immersive personalised persuasive serious games for financial literacy among young decision-makers,” inProceedings of the 53rd Hawaii International Conference on System Sciences (HICSS), 2020, pp. 1–10. [Online]. Available: https://scholarspace.manoa.hawaii.edu/items/ aeae2621-8ab3-4d76-b73c-ecdcfdbbe563

    2020

  17. [17]

    Deep knowledge tracing,

    C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. Guibas, and J. Sohl-Dickstein, “Deep knowledge tracing,” inAdvances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc., 2015, pp. 505–513

    2015

  18. [18]

    Context-aware attentive knowl- edge tracing,

    A. Ghosh, N. Heffernan, and A. S. Lan, “Context-aware attentive knowl- edge tracing,” inProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2020, pp. 2330–2339

    2020

  19. [19]

    arXiv preprint arXiv:2511.02599 (2025)

    M. Norris and S. Bulathwela, “Next token knowledge tracing: Exploiting pretrained LLM representations to decode student behaviour,” arXiv preprint arXiv:2511.02599, 2025. [Online]. Available: https: //arxiv.org/abs/2511.02599

    work page arXiv 2025

  20. [20]

    Llm supporting knowledge tracing leveraging global subject and student specific knowledge graphs,

    L. Li, Z. Wang, J. M. Jose, and X. Ge, “Llm supporting knowledge tracing leveraging global subject and student specific knowledge graphs,” Information Fusion, p. 103577, 2025

    2025

  21. [21]

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” inAdvances in Neural Information Processing Systems, vol. 36. Curran Associates, Inc., 2023

    2023

  22. [22]

    Identifiability: A fundamental problem of student modeling,

    J. E. Beck and K.-m. Chang, “Identifiability: A fundamental problem of student modeling,” inProc. 11th Int. Conf. on User Modeling (UM), 2007, pp. 137–146

    2007