Pith. sign in

REVIEW 2 major objections 34 references

EE-Eval assesses interactivity in AI-generated explorable explanations by extracting finite state machines and comparing them to ideal pedagogical models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 00:53 UTC pith:QWNDKZ7G

load-bearing objection EE-Eval formalizes interactivity via FSM extraction and comparison but the load-bearing steps of ideal model construction and faithful extraction lack described validation. the 2 major comments →

arxiv 2606.31012 v1 pith:QWNDKZ7G submitted 2026-06-30 cs.HC

Evaluating Interactivity: Toward Automated Assessment of AI-Generated Explorable Explanations

Xiaozao Wang , Zhewei Wang , Hongyi Wen This is my paper
classification cs.HC
keywords automated evaluationexplorable explanationsfinite state machinesinteractivity assessmentAI-generated contentpedagogical effectivenesshuman judgments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces EE-Eval to evaluate the quality of dynamic interactions in AI-generated learning materials, which existing benchmarks overlook in favor of code executability or visuals. It represents interactivity as finite state machines that capture learner-controlled states, transitions, and context-sensitive responses. Generated explanations are converted into explicit FSM graphs, then scored against an ideal FSM that encodes intended pedagogical behaviors using graph metrics and semantic embeddings. Tests on thousands of outputs from six models across 127 concepts show the method distinguishes interaction quality more effectively and matches human ratings of interactivity and teaching value better than baselines. This turns evaluation into a diagnostic process that supports targeted improvements in educational content.

Core claim

EE-Eval formalizes interactivity as a finite space of learner-controllable states and transitions represented as a Finite State Machine. It extracts FSMs from AI-generated explorable explanations and compares each to an ideal FSM encoding pedagogical intent using a combination of graph-based metrics and embedding-based comparison of states, actions, and feedback. Across thousands of generated explorable explanations spanning 127 concepts and produced by 6 AI models, EE-Eval consistently differentiates interaction quality beyond surface-level criteria such as functional correctness or visual quality, and exhibits substantially stronger alignment with human judgments of interactivity and pedag

What carries the argument

Finite State Machine extraction from generated explanations followed by structural and semantic comparison to an ideal pedagogical FSM via graph metrics and embedding similarity.

Load-bearing premise

An ideal FSM can be defined that accurately encodes pedagogical intent for each concept, and extraction from generated explanations faithfully captures the implicit interaction logic without distortion.

What would settle it

New AI-generated explanations on held-out concepts where human raters assign interactivity or pedagogical scores that show low correlation with EE-Eval's FSM similarity measures.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation moves beyond functional correctness or visual quality to behavioral interaction quality.
  • Results align more closely with human judgments than prior automated baselines.
  • Evaluation becomes a reflective diagnostic that identifies specific mismatches in states or transitions.
  • The approach supports iterative human-AI refinement of interactive educational content.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same FSM comparison approach could be tested on interactive AI outputs outside education, such as simulation tools or games.
  • Direct integration of EE-Eval scores into generation prompts might steer models toward higher-alignment interaction designs.
  • Ideal FSMs could serve as shared references for curriculum designers to specify desired learner behaviors before generation begins.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper introduces EE-Eval, an automated framework for assessing interactivity in AI-generated explorable explanations. It models interactivity via finite state machines (FSMs) extracted from generated artifacts, compares these to ideal FSMs encoding pedagogical intent using graph-based and embedding-based metrics, and reports that the approach differentiates interaction quality beyond functional or visual correctness while showing stronger correlation with human judgments than baselines. The evaluation spans thousands of examples across 127 concepts and 6 AI models.

Significance. If the FSM extraction procedure and ideal-FSM construction can be shown to be reliable and non-circular, the framework would offer a useful diagnostic for pedagogically grounded evaluation of interactive learning materials, moving beyond surface metrics. The emphasis on behavioral models rather than emergent properties is a constructive direction for human-AI collaboration in educational content creation.

major comments (2)
  1. [Abstract] Abstract: The central claim that EE-Eval differentiates interaction quality and aligns with human judgments rests on faithful extraction of implicit interaction logic into FSMs and on the existence of unambiguous ideal FSMs for each of the 127 concepts. The abstract provides no description of the extraction algorithm, no validation against human-annotated ground-truth FSMs, and no discussion of inter-annotator agreement or handling of context-sensitive feedback, leaving the reported differentiation and correlation results unverifiable.
  2. [Abstract] Abstract: The evaluation is described as comparing generated FSMs to ideal FSMs via graph and embedding metrics, yet no details are given on how pedagogical intent is formalized into a single canonical FSM per concept or on the sensitivity of the similarity scores to extraction errors. Without these, it is impossible to determine whether the superiority over baselines is driven by the intended pedagogical signal or by artifacts of the unstated extraction heuristics.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the careful reading and for identifying areas where the abstract could better support verifiability of the central claims. We address each comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that EE-Eval differentiates interaction quality and aligns with human judgments rests on faithful extraction of implicit interaction logic into FSMs and on the existence of unambiguous ideal FSMs for each of the 127 concepts. The abstract provides no description of the extraction algorithm, no validation against human-annotated ground-truth FSMs, and no discussion of inter-annotator agreement or handling of context-sensitive feedback, leaving the reported differentiation and correlation results unverifiable.

    Authors: The abstract is a high-level summary; the full manuscript details the FSM extraction procedure (including state identification, transition extraction, and context-sensitive feedback handling) in Section 3.2 and reports inter-annotator agreement for the human evaluation study in Section 5.1. We did not conduct direct validation of extracted FSMs against human-annotated ground-truth FSMs, as the evaluation centers on correlation with human ratings of interactivity and pedagogical effectiveness rather than FSM-level matching. We will revise the abstract to include a concise description of the extraction approach and the evaluation methodology. revision: partial

  2. Referee: [Abstract] Abstract: The evaluation is described as comparing generated FSMs to ideal FSMs via graph and embedding metrics, yet no details are given on how pedagogical intent is formalized into a single canonical FSM per concept or on the sensitivity of the similarity scores to extraction errors. Without these, it is impossible to determine whether the superiority over baselines is driven by the intended pedagogical signal or by artifacts of the unstated extraction heuristics.

    Authors: Section 3.3 of the manuscript describes how pedagogical intent is formalized into canonical ideal FSMs for each of the 127 concepts, drawing on expert-defined learning objectives. Sensitivity of the graph and embedding metrics to extraction errors is examined via ablation studies in Section 4.4 and the supplementary material. We will revise the abstract to briefly note the ideal-FSM construction process and the robustness checks performed. revision: yes

standing simulated objections not resolved
  • Direct validation of the FSM extraction procedure against human-annotated ground-truth FSMs is not present in the manuscript.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper defines EE-Eval via explicit FSM extraction from generated artifacts followed by graph and embedding comparison against separately specified ideal FSMs that encode pedagogical intent; the central claims rest on empirical differentiation and correlation with external human judgments rather than any self-referential construction, fitted parameter renamed as prediction, or load-bearing self-citation. No quoted step reduces the reported metrics to the inputs by definition.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Based on abstract only; no free parameters, axioms, or invented entities are specified. The approach implicitly relies on the existence of definable ideal FSMs but provides no details on how they are constructed or validated.

pith-pipeline@v0.9.1-grok · 5781 in / 1347 out tokens · 57646 ms · 2026-07-01T00:53:33.874817+00:00 · methodology

0 comments
read the original abstract

While large language models now enable rapid generation of interactive learning materials, evaluating the interaction quality of these explorable explanations remains an open challenge. Existing benchmarks largely focus on code executability or visual fidelity, providing limited insight into dynamic interaction behaviors such as learner-controlled state transitions and context-sensitive system responses, which are factors that critically shape learners' conceptual understanding. We present EE-Eval, an automated evaluation framework that formalizes interactivity as a finite space of learner-controllable states and transitions, represented as a Finite State Machine (FSM). By extracting FSMs from AI-generated explorable explanations, EE-Eval externalizes implicit interaction logic into an explicit, machine-interpretable graph. Evaluation is performed by comparing each generated FSM to an ideal FSM that encodes pedagogical intent, using a combination of graph-based metrics and embedding-based comparison of states, actions, and feedback to measure their structural and semantic similarity. Across thousands of generated explorable explanations spanning 127 concepts and produced by 6 AI models, EE-Eval consistently differentiates interaction quality beyond surface-level criteria such as functional correctness or visual quality, and exhibits substantially stronger alignment with human judgments of interactivity and pedagogical effectiveness than existing baselines. By framing interactivity as testable behavioral models rather than an emergent byproduct of LLM generation, EE-Eval transforms evaluation into a reflective diagnostic tool, enabling pedagogically grounded and actionable human-AI collaboration in creating interactive educational content.

Figures

Figures reproduced from arXiv: 2606.31012 by Hongyi Wen, Xiaozao Wang, Zhewei Wang.

Figure 3
Figure 3. Figure 3: EE-Eval’s user-facing interfaces explanations using multiple AI models, extracts FSMs from the resulting HTML/JavaScript artifacts, and computes FSM-based evaluation scores across all dimensions. To support transparency and interpretability, we provide a user-facing visualization interface that allows inspection of generated UIs, corresponding FSM structures, and detailed evaluation results ( [PITH_FULL_I… view at source ↗
Figure 4
Figure 4. Figure 4: FSM Similarity Score Distribution by Model [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages

  1. [1]

    Herbert James Banda and Joseph Nzabahimana. The impact of physics education technology (phet) interactive simulation-based learning on motivation and academic achievement among malawian physics students.Journal of Science Education and Technology, 32(1):127–141, 2023

    work page 2023

  2. [2]

    A meta-survey of generative ai in education: Trends, challenges, and research directions.Big Data and Cognitive Computing, 9(9), 2025

    Sirine Bouguettaya, Francesco Pupo, Min Chen, and Giancarlo Fortino. A meta-survey of generative ai in education: Trends, challenges, and research directions.Big Data and Cognitive Computing, 9(9), 2025

    work page 2025

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    work page 2020

  4. [4]

    Viseval: A benchmark for data visualization in the era of large language models, 2024

    Nan Chen, Yuge Zhang, Jiahang Xu, Kan Ren, and Yuqing Yang. Viseval: A benchmark for data visualization in the era of large language models, 2024

    work page 2024

  5. [5]

    Interactive sketchpad: A multimodal tutoring system for collaborative, visual problem-solving, 2025

    Steven-Shine Chen, Jimin Lee, and Paul Pu Liang. Interactive sketchpad: A multimodal tutoring system for collaborative, visual problem-solving, 2025

    work page 2025

  6. [6]

    Nl2interface: Interactive visualization interface generation from natural language queries, 2022

    Yiru Chen, Ryan Li, Austin Mac, Tianbao Xie, Tao Yu, and Eugene Wu. Nl2interface: Interactive visualization interface generation from natural language queries, 2022

    work page 2022

  7. [7]

    Augmented math: Authoring ar-based explorable explanations by augmenting static math textbooks

    Neil Chulpongsatorn, Mille Skovhus Lunding, Nishan Soni, and Ryo Suzuki. Augmented math: Authoring ar-based explorable explanations by augmenting static math textbooks. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, page 1–16. ACM, October 2023

    work page 2023

  8. [8]

    Ai-based learning content generation and learning pathway augmentation to increase learner engagement.Computers and Education: Artificial Intelligence, 4:100110, 2022

    Chaitali Diwan, Srinath Srinivasa, Gandharv Suri, Saksham Agarwal, and Prasad Ram. Ai-based learning content generation and learning pathway augmentation to increase learner engagement.Computers and Education: Artificial Intelligence, 4:100110, 2022

    work page 2022

  9. [9]

    Firat and Robert S

    Elif E. Firat and Robert S. Laramee. Towards a survey of interactive visualization for education. InComputer Graphics and Visual Computing, 2018

    work page 2018

  10. [10]

    An llm based learning framework for adaptive feedback mechanisms in gamified xr

    Anna Maria Gianni, Nikolaos Nikolakis, and Nikolaos Antoniadis. An llm based learning framework for adaptive feedback mechanisms in gamified xr. 7, 2025

    work page 2025

  11. [11]

    Augmented physics: Creating interactive and embedded physics simulations from static textbook diagrams, 2024

    Aditya Gunturu, Yi Wen, Nandi Zhang, Jarin Thundathil, Rubaiat Habib Kazi, and Ryo Suzuki. Augmented physics: Creating interactive and embedded physics simulations from static textbook diagrams, 2024

    work page 2024

  12. [12]

    Maple: A mobile assistant with persistent finite state machines for recovery reasoning, 2025

    Linqiang Guo, Wei Liu, Yi Heng, Chen Tse-Hsun, and Yang Wang. Maple: A mobile assistant with persistent finite state machines for recovery reasoning, 2025

    work page 2025

  13. [13]

    Chartllama: A multimodal llm for chart understanding and generation, 2023

    Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal llm for chart understanding and generation, 2023

    work page 2023

  14. [14]

    A meta-analysis of llm effects on students across qualification, socialisation, and subjectification, 2025

    Jiayu Huang, Ruoxin Ritter Wang, Jen-Hao Liu, Boming Xia, Yue Huang, Ruoxi Sun, Jason Minhui Xue, and Jinan Zou. A meta-analysis of llm effects on students across qualification, socialisation, and subjectification, 2025

    work page 2025

  15. [15]

    From eduvisbench to eduvisagent: A benchmark and multi-agent framework for reasoning-driven pedagogical visualization, 2025

    Haonian Ji, Shi Qiu, Siyang Xin, Siwei Han, Zhaorun Chen, Dake Zhang, Hongyi Wang, and Huaxiu Yao. From eduvisbench to eduvisagent: A benchmark and multi-agent framework for reasoning-driven pedagogical visualization, 2025

    work page 2025

  16. [16]

    Digital simulations in stem education: Insights from recent empirical studies, a systematic review.Encyclopedia, 5:10, 2025

    Chrysovalantis Kefalis, Constantine Skordoulis, and Athanasios Drigas. Digital simulations in stem education: Insights from recent empirical studies, a systematic review.Encyclopedia, 5:10, 2025

    work page 2025

  17. [17]

    Ai tutoring outperforms in-class active learning: an rct introducing a novel research-based design in an authentic educational setting.Scientific Reports, 15, 2025

    Greg Kestin, Kelly Miller, Anna Klales, Timothy Milbourne, and Gregorio Ponti. Ai tutoring outperforms in-class active learning: an rct introducing a novel research-based design in an authentic educational setting.Scientific Reports, 15, 2025

    work page 2025

  18. [18]

    Theoremexplainagent: Towards video-based multimodal explanations for llm theorem understanding, 2025

    Max Ku, Thomas Chong, Jonathan Leung, Krish Shah, Alvin Yu, and Wenhu Chen. Theoremexplainagent: Towards video-based multimodal explanations for llm theorem understanding, 2025

    work page 2025

  19. [19]

    nvbench: A large-scale synthesized dataset for cross-domain natural language to visualization task, 2021

    Yuyu Luo, Jiawei Tang, and Guoliang Li. nvbench: A large-scale synthesized dataset for cross-domain natural language to visualization task, 2021

    work page 2021

  20. [20]

    Chat2vis: Generating data visualisations via natural language using chatgpt, codex and gpt-3 large language models, 2023

    Paula Maddigan and Teo Susnjak. Chat2vis: Generating data visualisations via natural language using chatgpt, codex and gpt-3 large language models, 2023

    work page 2023

  21. [21]

    Crawling ajax-based web applications through dynamic analysis of user interface state changes.ACM Trans

    Ali Mesbah, Arie van Deursen, and Stefan Lenselink. Crawling ajax-based web applications through dynamic analysis of user interface state changes.ACM Trans. Web, 6(1), March 2012. 10 Toward Automated Assessment of AI-Generated Explorable Explanations

    work page 2012

  22. [22]

    nvagent: Automated data visualization from natural language via collaborative agent workflow, 2025

    Geliang Ouyang, Jingyao Chen, Zhihe Nie, Yi Gui, Yao Wan, Hongyu Zhang, and Dongping Chen. nvagent: Automated data visualization from natural language via collaborative agent workflow, 2025

    work page 2025

  23. [23]

    Vi(e)va llm! a conceptual stack for evaluating and interpreting generative ai-based visualizations, 2024

    Luca Podo, Muhammad Ishmal, and Marco Angelini. Vi(e)va llm! a conceptual stack for evaluating and interpreting generative ai-based visualizations, 2024

    work page 2024

  24. [24]

    Seskir, Piotr Migda, Carrie Weidner, Aditya Anupam, Nicky Case, Noah Davis, Chiara Decaroli, lke Ercan, Caterina Foti, Paweł Gora, Klementyna Jankiewicz, Brian R

    Zeki C. Seskir, Piotr Migda, Carrie Weidner, Aditya Anupam, Nicky Case, Noah Davis, Chiara Decaroli, lke Ercan, Caterina Foti, Paweł Gora, Klementyna Jankiewicz, Brian R. La Cour, Jorge Yago Malo, Sabrina Maniscalco, Azad Naeemi, Laurentiu Nita, Nassim Parvin, Fabio Scafirimuto, Jacob F. Sherson, Elif Surer, James Wootton, Lia Yeh, Olga Zabello, and Maril...

    work page 2022

  25. [25]

    Minilm: Deep self -attention distillation for task -agnostic compression of pretrained transformers

    Wei Wang and et al. Minilm: Deep self -attention distillation for task -agnostic compression of pretrained transformers. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    work page 2020

  26. [26]

    Visualization generation with large language models: An evaluation, 2025

    Xinyu Wang, Chenwei Liang, Shunyuan Zheng, Jinyuan Liang, Guozheng Li, Yu Zhang, and Chi Harold Liu. Visualization generation with large language models: An evaluation, 2025

    work page 2025

  27. [27]

    Wang, Robert Turko, Omar Shaikh, Haekyu Park, Nilaksh Das, Fred Hohman, Minsuk Kahng, and Duen Horng Polo Chau

    Zijie J. Wang, Robert Turko, Omar Shaikh, Haekyu Park, Nilaksh Das, Fred Hohman, Minsuk Kahng, and Duen Horng Polo Chau. Cnn explainer: Learning convolutional neural networks with interactive visualization. IEEE Transactions on Visualization and Computer Graphics, 27(2):1396–1406, February 2021

    work page 2021

  28. [28]

    Evens, Reva Freedman, Michael Glass, Leem Seop Shim, Yuemei Zhang, Yujian Zhou, and Joel Michael

    Chong Woo Woo, Martha W. Evens, Reva Freedman, Michael Glass, Leem Seop Shim, Yuemei Zhang, Yujian Zhou, and Joel Michael. An intelligent tutoring system that generates a natural language dialogue using dynamic multi-level planning.Artificial Intelligence in Medicine, 38(1):25–46, 2006. Intelligent Medical Training Systems

    work page 2006

  29. [29]

    Practical and ethical challenges of large language models in education: A systematic scoping review.British Journal of Educational Technology, 55(1):90–112, August 2023

    Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gaševi´c. Practical and ethical challenges of large language models in education: A systematic scoping review.British Journal of Educational Technology, 55(1):90–112, August 2023

    work page 2023

  30. [30]

    Landay, and Monica S

    Jackie (Junrui) Yang, Yingtian Shi, Yuhan Zhang, Karina Li, Daniel Wan Rosli, Anisha Jain, Shuning Zhang, Tianshi Li, James A. Landay, and Monica S. Lam. Reactgenie: A development framework for complex multimodal interactions using large language models. InProceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’24, page 1–23. ACM, May 2024

    work page 2024

  31. [31]

    Auto-slides: An interactive multi-agent system for creating and customizing research presentations, 2025

    Yuheng Yang, Wenjia Jiang, Yang Wang, Yiwei Wang, and Chi Zhang. Auto-slides: An interactive multi-agent system for creating and customizing research presentations, 2025

    work page 2025

  32. [32]

    Matplotagent: Method and evaluation for llm-based agentic scientific data visualization, 2024

    Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, and Maosong Sun. Matplotagent: Method and evaluation for llm-based agentic scientific data visualization, 2024

    work page 2024

  33. [33]

    Virtual reality practice and learning platform based on generative adversarial networks algorithm.Computer-Aided Design and Applications, pages 266–279, 2024

    Rui Zhang and Yueping Yang. Virtual reality practice and learning platform based on generative adversarial networks algorithm.Computer-Aided Design and Applications, pages 266–279, 2024

    work page 2024

  34. [34]

    Fisminess: A finite state machine based paradigm for emotional support conversations, 2025

    Yue Zhao, Qingqing Gu, Xiaoyu Wang, Teng Chen, Zhonglin Jiang, Yong Chen, and Luo Ji. Fisminess: A finite state machine based paradigm for emotional support conversations, 2025. 11

    work page 2025