REVIEW 5 major objections 6 minor 29 references
ClickSight: Interpreting Student Clickstreams to Reveal Insights on Learning Strategies via LLMs
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read ClickSight shows that a large language model can interpret raw student clickstreams in terms of predefined learning strategies, with zero-shot prompting yielding the highest expert-rated quality in two simulation environments.
desk verdict A credible demonstration that LLMs can interpret clickstreams, but the prompting-strategy ranking rests on an evaluation sample that makes it shakier than the paper suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The ClickSight pipeline structures clickstreams as function-call sequences (for example, discuss(mother, symptoms, t)) and gathers environment context plus a list of learning strategies; it then prompts an LLM with one of four prompting strategies (Zero-shot, Chain-of-Thought, Meta-Prompting, Chain-of-Prompts), optionally followed by up to three self-refinement rounds where the model answers binary rubric questions and revises its interpretation. Domain experts grade the outputs with a rubric of Completeness, Correctness, Justifiedness, and Comprehensibility. The load-bearing scoring identity is the multiplicative composite: for each of the nine strategy-aspect pairs the three criterion scores are multiplied, the nine values are averaged, and the result is multiplied by the Comprehensibility score, so a single ambiguous sentence or a single unjustified strategy claim drags the whole quality score down.
What would settle it
A rival explanation would be exposed if, holding clickstreams fixed, expert-rated quality did not track the clickstream content at all—for example, if the LLM's correctness score stayed high when the strategy definitions were shuffled or replaced with nonsense labels—or if the same zero-shot pipeline scored no better than chance at matching independently human-authored strategy annotations on the same clickstreams.
Extended reading notes
Core claim
The paper claims that an in-context LLM pipeline, ClickSight, can generate high-quality textual interpretations of student clickstreams in terms of predefined learning strategies, in open-ended learning environments, without training or handcrafted features. The evidence is a rubric-based expert evaluation across two environments: in PharmaSim and Beer's Law Lab, zero-shot prompting achieved the highest overall scores (0.79 and 0.76 respectively), perfect completeness in both, and perfect comprehensibility in PharmaSim; Chain-of-Prompts followed closely. Chain-of-Thought and Meta-Prompting scored lower on completeness or justifiability, often omitting strategies or lacking clickstream evidence. Self-refinement was not consistently helpful: it slightly reduced Zero-shot performance in PharmaSim due to hallucinations during self-evaluation and left Beer's Law Lab results unchanged because the model judged its initial output as already satisfactory.
Load-bearing premise
The result depends on the rubric-based binary-question scoring, and on sampling five students per behavior cluster, being a faithful measure of interpretation quality; if that score misrepresents what a good interpretation is, the prompting-strategy ranking and the high-quality conclusion do not generalize.
Editorial extensions
If this is right
- Without any per-environment feature engineering or supervised training, an LLM can produce strategy interpretations that domain experts score as high quality in two different open-ended environments.
- Zero-shot prompting is a competitive default; more elaborate scaffolds such as chain-of-thought and meta-prompting can reduce completeness or justifiability rather than improve them.
- Self-refinement is not a reliable quality boost: it can introduce hallucinated criticisms (PharmaSim) or make no changes (Beer's Law Lab), so its value depends on the prompting strategy and the environment.
- The rubric-based binary-question evaluation protocol provides a reusable, expert-grounded way to measure LLM interpretation quality in educational settings.
Reading between the lines
- The same in-context approach should transfer to other interaction-log formats (drag-and-drop, tool use, code editor events) provided the clickstreams are structured into comparable action sequences, which is a testable extension beyond the two environments.
- Because zero-shot already outperforms scaffolding, the practical bottleneck is the quality of the strategy taxonomy and rubric rather than the LLM's reasoning ability; investing in better theory-grounded strategy definitions may improve outputs more than prompt engineering.
- A downstream test would be to use ClickSight's interpretations as automated feedback to students, or as features for predicting learning outcomes; if those uses show no benefit over raw log statistics, the interpretation quality measure may not translate into practical value.
- The decline of zero-shot under self-refinement suggests that an adaptive refinement step which verifies each suggested revision against the original clickstream (rather than the model's memory) might preserve the gains without the hallucinations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ClickSight, an LLM-based pipeline that takes raw clickstreams from two open-ended learning environments (PharmaSim and Beer's Law Lab) together with a list of predefined learning strategies and generates textual interpretations of students' behavior. The pipeline is evaluated under four prompting strategies (Zero-shot, Chain-of-Thought, Meta-Prompting, Chain-of-Prompts), with an optional self-refinement step. Domain experts score the interpretations with a rubric covering Completeness, Correctness, Justifiedness, and Comprehensibility, using binary questions per strategy-aspect pair. The reported results are that Zero-shot achieves the highest overall score in both environments (PharmaSim 0.79±0.02, Beer's Law Lab 0.76±0.04), and that self-refinement helps Chain-of-Prompts but not Zero-shot. The authors conclude that LLMs can generate theory-aligned strategy interpretations from clickstream data.
Significance. If the central claim holds, ClickSight addresses a real bottleneck in educational data mining: expert annotation of clickstreams is labor-intensive, and prior approaches rely on handcrafted features or supervised models. The paper has concrete strengths: it ships reproducible code, evaluates on two distinct environments, reports high inter-annotator agreement (average Cohen's kappa mostly above 0.9), and its zero-shot absolute scores are high. The central capability claim, that LLMs can produce reasonable strategy-based interpretations, is therefore credible. However, the more specific claims about which prompting strategy is best and how self-refinement behaves are substantially less secure, because the evaluation sample is cluster-balanced rather than population-representative and no statistical tests are reported.
major comments (5)
- [Section 2.3, Table 2] The evaluation sample is built by taking five representative students per behavior cluster (30 PharmaSim students across 6 clusters, 20 Beer's Law students across 4 clusters). The overall scores in Table 2 are therefore unweighted averages over a cluster-balanced sample, not estimates of the mean over the student population. If cluster sizes are imbalanced, a prompting strategy that excels in small clusters can win the reported average while performing worse for the majority of students. The paper reports neither cluster sizes nor per-cluster scores, so the claim that 'Zero-shot achieved the highest overall score in both environments' is underdetermined. Please report cluster sizes, per-cluster scores, and population-weighted estimates, or justify explicitly why cluster-balanced sampling is the right target for RQ1.
- [Section 3.1, Table 2] The differences among prompting strategies are reported only as means and standard deviations, with no significance tests or confidence intervals. With only 30 and 20 sampled students, and with multiple interpretations per student inducing clustering, the observed ordering of Zero-shot over Chain-of-Prompts and the larger gaps for Chain-of-Thought and Meta-Prompting may not be statistically reliable. Please add appropriate inference, for example paired bootstrap at the student level, a mixed-effects model with student as a random effect, or at least exact confidence intervals for the reported differences.
- [Sections 2.2 and 2.3] There is partial circularity in the evaluation: the same rubric is included in the prompts to shape LLM outputs and is also used by human experts to score those outputs, and the self-refinement step guides the LLM with binary questions drawn from that same rubric. Human expert judgment provides independent grounding, so this is not full circularity, but the prompting-strategy ranking could partly reflect how well each method optimizes the rubric rather than genuine interpretation quality. I recommend adding an independent holistic evaluation, a held-out rubric dimension, or an evaluation where experts are blind to prompting condition, to break this loop.
- [Section 2.3, Grading Process] After computing inter-annotator agreement on a small subset (24 samples for PharmaSim, 16 for Beer's Law Lab), the remaining interpretations are annotated by a single expert. This is a reasonable cost-control measure, but the standard deviations in Table 2 are then computed over single-expert scores and may convey false precision. Please report how many outputs received double coding, the per-cell number of interpretations behind each mean, and ideally a second reliability sample drawn from the remaining outputs to confirm that single-expert scoring remains consistent.
- [Section 2.3 and Section 3] The evaluation contains no non-LLM baseline, such as a rule-based strategy detector, a handcrafted-feature classifier, or a simple n-gram/sequence-mining method. Without any baseline, the paper can support the claim that LLMs produce plausible interpretations, but it cannot support the framing that ClickSight offers advantages over prior approaches in generalizability or scalability. I consider this load-bearing for the introduction's motivation, even though it does not invalidate the central capability result.
minor comments (6)
- [Section 3 heading] The heading contains a typo: 'stratey' should be 'strategy'.
- [References] References [14] and [15] are identical (same authors, title, venue, and DOI); one should be removed or replaced with the intended distinct citation.
- [Figure 2] Figure 2 reports self-refinement effects but does not show error bars, confidence intervals, or per-condition sample sizes; please add these so the reader can judge the size of the reported improvements.
- [Section 2.2, Self-Refinement] The authors state that three refinement rounds 'we found sufficient for meaningful refinement' without reporting an ablation or a stopping criterion; please provide evidence for this choice or soften the claim.
- [Section 2.3, Grading Process] The clustering step is described only as 'k-means clustering of clickstreams' with the elbow method; the feature representation and preprocessing used for clustering are not specified, which hampers reproducibility. Please include these details or point to the exact scripts in the repository.
- [Table 2] The formatting of Table 2 is difficult to parse because criterion columns run together; please reformat the table so each environment and criterion is clearly separated.
Circularity Check
No significant circularity: ClickSight's evaluation rests on independent expert judgments against an external rubric, and no reported score is constructed from the prompt inputs themselves.
full rationale
The central claim is empirical: LLM-generated interpretations are scored by human experts using a rubric grounded in human-centered explanation theory. The derivation chain is clickstream + strategy list + prompt → LLM output → expert rubric score. No parameter is fitted to a subset of the evaluation data and then reported as a prediction, and no equation equates an input with an output. The rubric being included in the prompt is a design choice: it tells the LLM what to aim for, but the rubric does not specify the strategy execution levels that must be inferred from the clickstream, so expert scoring still requires independent comparison of claims against data. The self-refinement step uses binary questions derived from the same rubric, which could bias revisions toward rubric alignment, but the paper reports mixed results (Zero-shot performance decreased in PharmaSim), so improvement is not forced by construction. Self-citations such as [17] and [21] supply data formats and cluster setups from prior work; they are inputs rather than conclusions, and the human-expert evaluation is independent of them. The cluster-balanced sampling design is a statistical generalizability concern about whether the reported averages represent the student population, but it is not a circularity between inputs and outputs. Overall, the paper's evaluation is self-contained with respect to its claims, and no load-bearing circular step is present.
Assumptions & free parameters
free parameters (5)
- self_refinement_rounds =
3
- students_per_cluster =
5
- merge_threshold_seconds =
3
- num_clusters_pharmasim =
6
- num_clusters_beers_law =
4
assumptions (5)
- domain assumption The predefined learning strategy taxonomies (9 PharmaSim strategies, 3 Beer's Law Lab strategies) are valid and sufficient descriptors of student behavior.
- domain assumption Binary yes/no questions, one per strategy-aspect pair, can measure completeness, correctness, and justifiedness.
- domain assumption Human expert judgments using the rubric are reliable and valid ground truth.
- domain assumption The clickstream representation, including functional call formatting and the exclusion of analysis actions in Beer's Law Lab, preserves behavior relevant to strategies.
- domain assumption GPT-4o is a stable and representative LLM for this task.
Cite this review
Pith. "Pith review of ClickSight: Interpreting Student Clickstreams to Reveal Insights on Learning Strategies via LLMs." pith.science (2026). https://pith.science/paper/TTYKBA6P
@misc{pith2026250515410,
author = {Pith},
title = {Pith review of: ClickSight: Interpreting Student Clickstreams to Reveal Insights on Learning Strategies via LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/TTYKBA6P}},
note = {Machine review of arXiv:2505.15410}
}
read the original abstract
Clickstream data from digital learning environments offer valuable insights into students' learning behaviors, but are challenging to interpret due to their high dimensionality and granularity. Prior approaches have relied mainly on handcrafted features, expert labeling, clustering, or supervised models, therefore often lacking generalizability and scalability. In this work, we introduce ClickSight, an in-context Large Language Model (LLM)-based pipeline that interprets student clickstreams to reveal their learning strategies. ClickSight takes raw clickstreams and a list of learning strategies as input and generates textual interpretations of students' behaviors during interaction. We evaluate four different prompting strategies and investigate the impact of self-refinement on interpretation quality. Our evaluation spans two open-ended learning environments and uses a rubric-based domain-expert evaluation. Results show that while LLMs can reasonably interpret learning strategies from clickstreams, interpretation quality varies by prompting strategy, and self-refinement offers limited improvement. ClickSight demonstrates the potential of LLMs to generate theory-driven insights from educational interaction data.
Figures
Reference graph
Works this paper leans on
-
[1]
In: Rafferty, A.N., Whitehill, J., Romero, C., Cavalli-Sforza, V
Akpinar, N., Ramdas, A., Acar, U.A.: Analyzing student strategies in blended courses using clickstream data. In: Rafferty, A.N., Whitehill, J., Romero, C., Cavalli-Sforza, V. (eds.) Proceedings of the 13th International Conference on Educational Data Mining, EDM 2020, Fully virtual conference, July 10-13,
work page 2020
-
[2]
In: Proceedings of the 8th International Conference on Intelligent Tutoring Sys- tems
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Wagner, A.Z.: Gaming the system: Learning from mixed-response data. In: Proceedings of the 8th International Conference on Intelligent Tutoring Sys- tems. pp. 272–281. Springer (2006)
work page 2006
-
[3]
Baker, R.S., Inventado, P.S.: Educational Data Mining and Learning Analytics, pp. 61–75. Springer New York, New York, NY (2014), https://doi.org/10.1007/978-1-4614-3305-7_4
-
[4]
Denny, P., Gulwani, S., Heffernan, N.T., K¨ aser, T., Moore, S., Rafferty, A.N., Singla, A.: Generative AI for education (GAIED): advances, opportunities, and challenges. CoRR abs/2402.01580 (2024). https://doi.org/10.48550/ARXIV.2402.01580, https://doi.org/10.48550/arXiv.2402.01580
-
[5]
European Journal of Population 33(2), 163–184 (2017)
Erola, J., Kilpi-Jakonen, E.: Family dynamics and child outcomes: An overview of research and open questions. European Journal of Population 33(2), 163–184 (2017)
work page 2017
-
[6]
StatPearls Publishing (2020), available from:https://www.ncbi.nlm.nih
Gillis, A.: Family Dynamics. StatPearls Publishing (2020), available from:https://www.ncbi.nlm.nih. gov/books/NBK560487/
work page 2020
-
[7]
Hanamanthrao, R., Thejaswini, S.: Real-time clickstream data analytics and visualization. In: 2017 2nd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT). pp. 2139–2144 (2017). https://doi.org/10.1109/RTEICT.2017.8256978
-
[8]
International Journal of Artificial Intelligence in Education 30(3), 504–535 (2020)
K¨ aser, T., Schwartz, D.L.: Modeling and Analyzing Inquiry Strategies in Open-Ended Learning Environments. International Journal of Artificial Intelligence in Education 30(3), 504–535 (2020). https://doi.org/10.1007/S40593-020-00199-Y, https://doi.org/10.1007/s40593-020-00199-y
Show all 29 references
- [9]
-
[10]
Biochemia Medica (Zagreb) 22(3), 276–282 (2012), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900052/
McHugh, M.L.: Interrater reliability: the kappa statistic. Biochemia Medica (Zagreb) 22(3), 276–282 (2012), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900052/
2012
-
[11]
In: Conference on Robot Learning (2023), https://api.semanticscholar.org/CorpusID:259501163
Mirchandani, S., Xia, F., Florence, P.R., Ichter, B., Driess, D., Arenas, M.G., Rao, K., Sadigh, D., Zeng, A.: Large language models as general pattern machines. In: Conference on Robot Learning (2023), https://api.semanticscholar.org/CorpusID:259501163
2023
-
[12]
Chemistry Education Research and Practice 14(3), 257–268 (2013)
Moore, E.B., Herzog, T.A., Perkins, K.K.: Interactive simulations as implicit support for guided-inquiry. Chemistry Education Research and Practice 14(3), 257–268 (2013)
2013
-
[13]
OpenAI: Gpt-4o system card (Aug 2024), https://openai.com/index/gpt-4o-system-card/ , ac- cessed: 2024-11-14
2024
-
[15]
In: Proceedings of the 14th Learning Analytics and Knowledge Conference, LAK 2024, Kyoto, Japan, March 18-22, 2024
Phung, T., Padurean, V., Singh, A., Brooks, C., Cambronero, J., Gulwani, S., Singla, A., Soares, G.: Automating human tutor-style programming feedback: Leveraging GPT-4 tutor model for hint gener- ation and GPT-3.5 student model for hint validation. In: Proceedings of the 14th...
2024
-
[16]
In: Olney, A.M., Chounta, I.A., Liu, Z., Santos, O.C., Bittencourt, I.I
Prihar, E., Radmehr, B., K¨ aser, T.: An interpretable approach to identify performance indicators within unstructured learning environments. In: Olney, A.M., Chounta, I.A., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds.) Artificial Intelligence in Education. Posters and Late ...
2024
-
[17]
Journal of Educational Data Mining 17(1), 1–40 (Jan 2025)
Radmehr, B., Singla, A., K¨ aser, T.: Pharmasimtext: A text-based educational playground filled with rl-llm agents that work together even in disagreement. Journal of Educational Data Mining 17(1), 1–40 (Jan 2025). https://doi.org/10.5281/zenodo.14681290, https://jedm.educatio...
2025 doi
-
[18]
Journal of Learning Analytics 2(1), 7–12 (2015)
Roll, I., Winne, P.H.: Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics 2(1), 7–12 (2015)
2015
-
[19]
In: Olney, A.M., Chounta, I.A., Liu, Z., Santos, O.C., Bittencourt, I.I
Scaria, N., Dharani Chenna, S., Subramani, D.: Automated educational question generation at different bloom’s skill levels using large language models: Strategies and evaluation. In: Olney, A.M., Chounta, I.A., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds.) Artificial Intelli...
2024
-
[20]
https://doi.org/https://doi.org/10.1016/j.caeo.2025.100247, https://www.sciencedirect.com/ science/article/pii/S2666557325000060
Schulz, A., Voermanek, J.: How do help-seeking and help-abuse affect learning achievement in an interactive learning environment? Computers and Education Open 8, 100247 (2025). https://doi.org/https://doi.org/10.1016/j.caeo.2025.100247, https://www.sciencedirect.com/ science/a...
2025
-
[21]
In: Artificial Intelligence in Education: 25th International Conference, AIED 2024, Recife, Brazil, July 8–12, 2024, Proceedings, Part I
Shved, E., Bumbacher, E., Mejia-Domenzain, P., Kapur, M., K¨ aser, T.: Teaching and measuring multidimensional inquiry skills using interactive simulations. In: Artificial Intelligence in Education: 25th International Conference, AIED 2024, Recife, Brazil, July 8–12, 2024, Pro...
2024 doi
-
[22]
In: Pro- ceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
Srinivasan, R., Chander, A.: Explanation perspectives from the cognitive sciences—a survey. In: Pro- ceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. IJCAI’20 (2021)
2021
-
[23]
In: Ku, L.W., Martins, A., Srikumar, V
Sun, S., Yuan, R., Cao, Z., Li, W., Liu, P.: Prompt chaining or stepwise prompt? refinement in text summarization. In: Ku, L.W., Martins, A., Srikumar, V. (eds.) Findings of the Association for Compu- tational Linguistics: ACL 2024. pp. 7551–7558. Association for Computational...
2024 doi
-
[24]
In: David, J.M., Kriv- ine, J.P., Simmons, R
Swartout, W.R., Moore, J.D.: Explanation in second generation expert systems. In: David, J.M., Kriv- ine, J.P., Simmons, R. (eds.) Second Generation Expert Systems. pp. 543–585. Springer Berlin Heidel- berg, Berlin, Heidelberg (1993)
1993
-
[25]
In: Olney, A.M., Chounta, I.A., Liu, Z., Santos, O.C., Bittencourt, I.I
Villalobos, E., P´ erez-Sanagust ´ ın, M., Broisin, J.: From learning actions to dynamics: Characterizing students’ individual temporal behavior with sequence analysis. In: Olney, A.M., Chounta, I.A., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds.) Artificial Intelligence in E...
2024
-
[26]
In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Informatio...
2022
-
[27]
International Journal of Artificial Intelligence in Education 32(4), 931–970 (2022)
Zhang, N., Biswas, G., Hutchins, N.: Measuring and Analyzing Students’ Strategic Learning Behaviors in Open-Ended Learning Environments. International Journal of Artificial Intelligence in Education 32(4), 931–970 (2022). https://doi.org/10.1007/S40593-021-00275-X
2022 doi
-
[28]
Zhang, Y., Yuan, Y., Yao, A.C.C.: Meta prompting for ai systems (2025), https://arxiv.org/abs/ 2311.11482
2025 arXiv
-
[29]
Theory Into Practice 41(2), 64–70 (2002)
Zimmerman, B.J.: Becoming a self-regulated learner: An overview. Theory Into Practice 41(2), 64–70 (2002). https://doi.org/10.1207/s15430421tip4102 2
2002 doi
-
[2020]
International Educational Data Mining Society (2020), https://educationaldatamining.org/ files/conferences/EDM2020/papers/paper_172.pdf
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.