REVIEW 5 major objections 6 minor 33 references
From Formulas to Figures: How Visual Elements Impact User Interactions in Educational Videos
T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Text-heavy video moments drive pauses, rewinds, dropouts
desk verdict The taxonomy and real-world dataset are the real contributions, but the permutation test's failure to preserve autocorrelation undermines the headline p-values as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a two-part pipeline. First is the paper's fine-grained taxonomy of visual objects in STEM videos, which classifies annotated elements into textual categories (equations, terms, lines of text) and information-visualization categories (diagrams, tables, charts, technical drawings), making visual complexity a concrete count of distinguishable meaningful objects per frame. Second is a modified Dynamic Time Warping (DTW) procedure used as the test statistic in a permutation test: it aligns the visual-complexity time series with each user-interaction time series while enforcing a causality constraint that interactions can only map to preceding visual changes and a reaction-window constraint that limits mappings to about ten seconds (two five-second bins) after a stimulus. Per-video p-values are aggregated with Fisher's method, and a permutation effect size (PES) is computed as the standardized distance between the observed DTW cost and the null distribution.
What would settle it
Independently re-annotate all 25 videos, or a large random sample of them, with a second annotator or an automated object detector using the same taxonomy, recompute the visual-complexity time series, and rerun the permutation tests; if the reported associations ($p = 0.0$, PES $= 0.89$ for pausing; $p = 0.017$, PES $= 0.47$ for dropouts; $p = 0.001$, PES $= 0.79$ for rewinds) do not replicate with the independent series, the original result reflects the single annotator's labeling habits rather than a property of the videos.
Extended reading notes
Core claim
The paper's central claim is that visual complexity, measured as the number of newly appearing annotated objects per five-second bin, is statistically associated with user behavior in authentic viewing logs, and the association is driven by textual elements. It reports that users paused more frequently in sections with high textual change counts ($p = 0.0$, PES $= 0.89$), paused more under overall visual complexity ($p = 0.009$, PES $= 0.64$), rewound away from text-heavy sections more often ($p = 0.001$, PES $= 0.79$), and dropped out of videos more often at textually complex points ($p = 0.017$, PES $= 0.47$). The binary version of the complexity signal—whether any change occurred—was weaker or non-significant for dropouts and rewinds, which the authors take to mean the magnitude of textual change, not the mere occurrence, drives behavior. No significant effects were found for graphical or visualization elements, leading to the conclusion that on-screen text imposes higher cognitive load than graphics in real learning settings.
Load-bearing premise
The entire visual-complexity time series for all 25 videos rests on annotations by a single primary annotator, and agreement with a second annotator was measured on only five frames; if that annotator's judgments drift over the full corpus or the taxonomy misses relevant object classes, the predictor series is biased and every reported association could be an artifact of the labeling rather than the video content.
Editorial extensions
If this is right
- Video creators can use the number of newly appearing text objects per time window as a risk score: text spikes mark where viewers will pause, rewind, or leave, so those segments can be redesigned or broken up before release.
- Because graphical elements such as diagrams and charts showed no significant negative associations, shifting information from dense text to visual representations should reduce disruptive interactions without losing content.
- The weaker binary-change results imply that the magnitude of a text burst matters more than the mere fact of change, so incremental text reveals are less disruptive than large simultaneous text additions.
- The pattern supports adding system-determined pauses or interactive prompts at text-heavy points in the video, building on the segmenting-effect findings the paper cites.
Reading between the lines
- A testable extension the paper does not run: replace equation-heavy text segments with narrated visualizations of the same content and compare pause, rewind, and dropout rates; the paper's text-vs-graphics asymmetry predicts fewer disruptions in the visualization condition.
- Varying the 10-second reaction window in the DTW analysis could produce a temporal response curve showing how quickly viewers pause or rewind after a text spike, distinguishing immediate overload from delayed comprehension attempts.
- Automating the taxonomy with an object detector would let the same analysis scale from 25 videos to thousands and turn the correlational finding into a practical pre-publication screening tool for video producers.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether visual complexity in STEM educational videos is associated with user interactions in real-world viewing logs. The authors develop a fine-grained taxonomy of visual objects, annotate 25 videos from the TIB AV-Portal (SciFox channel), and construct per-video time series of the number of newly appearing visual objects (textual vs. information-visualization categories) in 5-second bins. They relate these to binned user interactions (pauses, rewinds, skips, dropouts) using a modified dynamic time warping (DTW) test statistic with a causality constraint and a 10-second reaction window, and assess significance with a permutation test aggregated across videos via Fisher's method. The headline findings are that textual visual complexity is associated with more pauses (p=0.0, PES=0.89), rewinds (p=0.001, PES=0.79), and dropouts (p=0.017, PES=0.47), while information-visualization elements show no such effects.
Significance. The study addresses a genuinely understudied question with authentic interaction data, which is a strength relative to laboratory studies. The taxonomy is a useful contribution, and the public code and careful preprocessing (e.g., normalizing by active sessions, unique-session counting) support reproducibility. If the statistical analysis is validated, the findings would have practical implications for educational video design, suggesting that text-heavy segments impose processing costs that manifest in pausing, rewinding, and dropout. However, the current statistical machinery has serious validity concerns (permutation null, reaction-window selection, multiple comparisons, annotation reliability), so the empirical contribution is not yet established at the level claimed.
major comments (5)
- [§4.3, Table 1] The permutation test permutes the visual-complexity series VisCom_v without preserving its temporal autocorrelation, while the modified DTW test statistic with a two-bin warping window is sensitive to the autocorrelation structure of the series. Because both VisCom_v and the interaction series are bursty (visual changes cluster at slide transitions and interactions cluster in bursts), the original series can achieve a lower DTW cost than a randomly shuffled series that has been whitened, so the null distribution is shifted toward higher costs and the p-values in Table 1 are biased toward zero. This affects every row of Table 1, including the headline pause/T VisCom result (p=0.0, PES=0.89) and rewind-from/T (p=0.001, PES=0.79). Please re-run the analysis under a null that preserves the autocorrelation of VisCom_v (e.g., block permutation, circular shifting, or phase-randomized surrogates) and report the resulting p-values and PES.
- [§4.3, Algorithm 1] The causality constraint is described inconsistently with its stated intent. The text says that setting all costs below the main diagonal to infinity 'prevents future user interactions from being mapped to past visual complexity changes,' but in the matrix orientation implied by Algorithm 1 (s1 = visual complexity, s2 = user interactions, D[i,j] = (s1[j]-s2[i])^2), cells below the diagonal have j < i and correspond to mapping a current interaction to an earlier visual stimulus, which is precisely the desired causal direction. Either the orientation or the direction of the constraint is wrong. Please specify the exact indexing and verify the implementation in the public code, because a reversed constraint would change the test statistic into one that aligns interactions with future stimuli and invalidates the interpretation.
- [§4.3, Table 1] The paper reports 36 significance tests (6 actions × 3 modality groupings × 2 conditions) in Table 1 without any multiple-comparison correction. With a Bonferroni threshold of approximately 0.0014, only the pause/T VisCom (p=0.0) and rewind-from/T VisCom (p=0.001) results remain significant; the dropout/T result (p=0.017) would not survive. Please apply a suitable correction (e.g., Benjamini-Hochberg) or explicitly label the analysis as exploratory, and revise the conclusions to match the corrected outcome.
- [§3.2/§4.1] The predictor series is built from annotations by a single primary annotator (A1), with inter-annotator agreement measured on only five frames against A2 before A1 completed the full annotation. Because the visual-complexity time series is defined by the appearance of new objects across frames, the static IoU check does not validate the temporal tracking that determines 'new' objects, and any annotation drift or tracking error is propagated into every permutation test. Please report per-video or per-change-event annotation quality (e.g., a sample of change events re-annotated by A2) or otherwise demonstrate that the predictor series is reliable.
- [§4.3] The two-bin (10 s) reaction window and the 5 s bin size are chosen based on exploratory visual analysis of the same data that is later used for hypothesis testing. This post-hoc tuning of the test statistic's parameters can inflate the apparent significance because the window is selected to make the alignment cost small. Please report a sensitivity analysis over the window size (e.g., 1, 2, 3 bins) and, ideally, confirm the findings on a hold-out set or with pre-registered parameter choices.
minor comments (6)
- [§3.2] The phrase 'Reviewer guidelines were established and continuously adapted' appears to be a typo; it should presumably read 'Annotation guidelines were established and continuously adapted.'
- [§4.3, Eq. (2)] The sentence 'The negative sign ensures a positive PESv indicates a positive.' is incomplete; it should specify that a positive PESv indicates a positive relationship between visual complexity and the respective user interaction.
- [Table 1] The meaning of 'time' under the 'action' column is ambiguous; the labels 'from' and 'to' for rewind and skip should be defined in the caption (e.g., 'from' = source position of the seek, 'to' = target position).
- [Algorithm 1] The pseudocode declares a window parameter but does not specify its exact indexing; please state precisely which cells are set to infinity in terms of i and j, both for the causality constraint and the reaction window.
- [§4.3] The statement 'Based on exploratory visual analysis, we observed that user responses often occur up to 10 seconds after a visual stimulus' should be accompanied by a concrete description of the exploration (e.g., number of videos inspected and the procedure used to arrive at the 10 s threshold).
- [§5] The abbreviation 'CTLM' should be 'CTML' (Cognitive Theory of Multimedia Learning) to match the usage in the introduction.
Circularity Check
No significant circularity: the paper's claims are empirical associations evaluated with a permutation test, not constructed from their inputs.
full rationale
The central results (pauses, rewinds, and dropouts following textual visual changes) are statistical findings from annotated videos and user logs, not quantities derived by definition from the input series. The visual-complexity series is constructed from bounding-box annotations via a stated taxonomy, and the user-interaction series comes from logged events; the DTW cost is a test statistic, and the p-values arise from permuting the visual-complexity series against fixed interaction series. No parameter is fitted to the outcome and then reported as a prediction: the 10-second reaction window is an exploratory modeling choice, and the binary-condition reanalysis is a robustness check rather than a fitted quantity. The self-citations (e.g., Merkt et al. 2022, which includes two coauthors of the present paper) are used for related work and to motivate bin size, not as a load-bearing proof or uniqueness theorem. The paper also states its own limitation in Section 6: 'our results are based on 25 videos, and the findings' generalizability requires further research.' The skeptic's concern that random permutation destroys autocorrelation and biases the null distribution is a statistical validity critique, not circularity; it does not make any equation of the paper equivalent to its own inputs. Therefore no pattern of self-definition, fitted-input-as-prediction, or load-bearing self-citation is present.
Assumptions & free parameters
free parameters (4)
- Reaction window size =
2 bins (10 s)
- Time bin size =
5 s
- Trimmed edge bins =
2 bins at start and 2 at end
- Annotation agreement threshold =
IoU 0.88
assumptions (5)
- domain assumption User interactions (pause, rewind, skip, dropout) are caused by preceding visual content rather than by audio, difficulty, or other confounds.
- domain assumption Visual complexity is adequately measured by the number of newly appearing annotated elements between frames, not by static element counts.
- domain assumption The taxonomy is complete and valid for physics and chemistry videos from the SciFox channel.
- standard math Permutation of the visual complexity series yields a valid null distribution for the DTW cost statistic.
- domain assumption TIB AV-Portal logs with at least one play event reflect genuine viewing sessions.
Cite this review
Pith. "Pith review of From Formulas to Figures: How Visual Elements Impact User Interactions in Educational Videos." pith.science (2026). https://pith.science/paper/2SQJTNGN
@misc{pith2026250501753,
author = {Pith},
title = {Pith review of: From Formulas to Figures: How Visual Elements Impact User Interactions in Educational Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/2SQJTNGN}},
note = {Machine review of arXiv:2505.01753}
}
read the original abstract
Educational videos have become increasingly relevant in today's learning environments. While prior research in laboratory studies has provided valuable insights, analyzing real-world interaction data can enhance our understanding of authentic user behavior. Previous studies have investigated technical aspects, such as the influence of cuts on pausing behavior, but the impact of visual complexity remains understudied. In this paper, we address this gap and propose a novel approach centered on visual complexity, defined as the number of visually distinguishable and meaningful elements in a video frame, such as mathematical equations, chemical formulas, or graphical representations. Our study introduces a fine-grained taxonomy of visual objects in educational videos, expanding on previous classifications. Applying this taxonomy to 25 videos from physics and chemistry, we examine the relationship between visual complexity and user behavior, including pauses, in-video navigation, and session dropouts. The results indicate that increased visual complexity, especially of textual elements, correlates with more frequent pauses, rewinds, and dropouts. The results offer a deeper understanding of how video design affects user behavior in real-world scenarios. Our work has implications for optimizing educational videos, particularly in STEM fields. We make our code publicly available (https://github.com/TIBHannover/from_formulas_to_figures).
Figures
Reference graph
Works this paper leans on
-
[1]
International Review of Research in Open and Distributed Learning 25(2), 94–113 (2024)
Akçapınar, G., Er, E., Bayazıt, A.: Decoding video logs: Unveiling student engage- ment patterns in lecture capture videos. International Review of Research in Open and Distributed Learning 25(2), 94–113 (2024). https://doi.org/10.19173/irrodl. v25i2.7621
doi:10.19173/irrodl 2024
-
[2]
Computers & Education212, 104996 (2024)
Anders, G., Buder, J., Merkt, M., Egger, E., Huff, M.: Associations between mind wandering, viewer interactions, and the meaningful structure of educational videos. Computers & Education212, 104996 (2024). https://doi.org/10.1016/j.compedu. 2024.104996
arXiv 2024
-
[3]
Computers in Human Behavior89, 411–417 (2018)
Biard, N., Cojean, S., Jamet, E.: Effects of segmentation and pacing on procedural learning by video. Computers in Human Behavior89, 411–417 (2018). https://doi. org/10.1016/J.CHB.2017.12.002
-
[4]
In- ternational Journal of Education in Mathematics, Science, and Technology9(4), 585–613 (2021)
Castro, P., Gomez, P.: Taxonomy of key terms for mathematics education. In- ternational Journal of Education in Mathematics, Science, and Technology9(4), 585–613 (2021). https://doi.org/10.46328/ijemst.1289
-
[5]
Psychological Bulletin132(1), 73–97 (2006)
Donderi, D.C.: Visual complexity: A review. Psychological Bulletin132(1), 73–97 (2006). https://doi.org/10.1037/0033-2909.132.1.73
-
[6]
Faber, M., Radvansky, G.A., D’Mello, S.K.: Driven to distraction: A lack of change gives rise to mind wandering. Cognition173, 133–137 (2018). https://doi.org/10. 1016/j.cognition.2018.01.007
work page 2018
-
[7]
Fisher, R.A.: Statistical Methods for Research Workers, pp. 66–70. Springer New York, New York, NY (1992). https://doi.org/10.1007/978-1-4612-4380-9_6
-
[8]
(ed.) Engineering Psychology and Cognitive Ergonomics
Forsythe, A.: Visual complexity: Is that all there is? In: Harris, D. (ed.) Engineering Psychology and Cognitive Ergonomics. pp. 158–166. Springer Berlin Heidelberg, Berlin, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02728-4_17
Show all 33 references
-
[9]
In: Conference on Learning @ Scale, L@S 2014, Atlanta, GA, USA, March 4-5, 2014
Guo, P.J., Kim, J., Rubin, R.: How video production affects student engagement: an empirical study of MOOC videos. In: Conference on Learning @ Scale, L@S 2014, Atlanta, GA, USA, March 4-5, 2014. pp. 41–50. ACM (2014). https://doi. org/10.1145/2556325.2566239 14 W. Gritz et al
2014
-
[10]
Applied Cognitive Psychology 21(6), 713–729 (2007)
Hasler, B.S., Kersten, B., Sweller, J.: Learner control, cognitive load and instruc- tional animation. Applied Cognitive Psychology 21(6), 713–729 (2007). https: //doi.org/10.1002/acp.1345
2007 doi
-
[11]
In: Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, January 7-11, 2019
Haurilet, M., Al-Halah, Z., Stiefelhagen, R.: Spase - multi-label page segmentation for presentation slides. In: Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, January 7-11, 2019. pp. 726–734 (2019). https://doi.org/10.1109/WACV.2019.00082
2019
-
[12]
In: International Conference on Document Analysis and Recogni- tion, ICDAR 2019, Sydney, Australia, 20-25 September, 2019
Haurilet, M., Roitberg, A., Martinez, M., Stiefelhagen, R.: Wise — slide segmenta- tion in the wild. In: International Conference on Document Analysis and Recogni- tion, ICDAR 2019, Sydney, Australia, 20-25 September, 2019. pp. 343–348 (2019). https://doi.org/10.1109/ICDAR.2019.00062
2019
-
[13]
Journal of Experimental Psychology: Applied22(3), 305–318 (2016)
Jing, H.G., Szpunar, K.K., Schacter, D.L.: Interpolated testing influences focused attention and improves integration of information during a video-recorded lecture. Journal of Experimental Psychology: Applied22(3), 305–318 (2016). https://doi. org/10.1037/xap0000087
2016 doi
-
[14]
In: Conference on Human Factors in Computing Systems, CHI 2022, New Orleans, LA, USA 29 April - 5 May, 2022
Kim, J., Choi, Y., Kahng, M., Kim, J.: Fitvid: Responsive and flexible video con- tent adaptation. In: Conference on Human Factors in Computing Systems, CHI 2022, New Orleans, LA, USA 29 April - 5 May, 2022. Association for Computing Machinery (2022). https://doi.org/10.1145/3...
2022
-
[15]
Australasian Journal of Educational Technology 38(1), 102––114 (2021)
Liu, T.C., Lin, Y.C., Kalyuga, S.: Effects of complexity-determined system pausing on learning from multimedia presentations. Australasian Journal of Educational Technology 38(1), 102––114 (2021). https://doi.org/10.14742/ajet.7267
2021 doi
-
[16]
https://doi.org/10.1111/J.1467-8535.2008.00848.X
Lusk, D.L., Evans, A.D., Jeffrey, T.R., Palmer, K.R., Wikstrom, C.S., Doolittle, P.E.: Multimedia learning and individual differences: Mediating the effects of work- ingmemorycapacitywithsegmentation.BritishJournalofEducationalTechnology 40(4), 636–651 (2009). https://doi.org/...
2009
-
[17]
(ed.): The Cambridge Handbook of Multimedia Learning
Mayer, R.E. (ed.): The Cambridge Handbook of Multimedia Learning. Cambridge University Press, 2nd edn. (2014). https://doi.org/10.1017/CBO9781139547369
2014 doi
-
[18]
Educational Psychology Review36(1), 8 (2024)
Mayer, R.E.: The past, present, and future of the cognitive theory of multime- dia learning. Educational Psychology Review36(1), 8 (2024). https://doi.org/10. 1007/s10648-023-09842-1
2024
-
[19]
Meier, J., de Jong, B., van Montfort, D.P., Verdonschot, A., van Wermeskerken, M., van Gog, T.: Do social cues in instructional videos affect attention allocation, perceived cognitive load, and learning outcomes under different visual complexity conditions?JournalofComputerAss...
2023 doi
-
[20]
Computers in Human Behavior89, 399–410 (2018)
Merkt, M., Ballmann, A., Felfeli, J., Schwan, S.: Pauses in educational videos: Testing the transience explanation against the structuring explanation. Computers in Human Behavior89, 399–410 (2018). https://doi.org/10.1016/J.CHB.2018.01. 013
2018 doi
-
[21]
https://doi.org/10.1016/j.compedu.2021.104355
Merkt, M., Hoppe, A., Bruns, G., Ewerth, R., Huff, M.: Pushing the button: Why do learners pause online videos? Computers & Education 176, 104355 (2022). https://doi.org/10.1016/j.compedu.2021.104355
2022
-
[22]
In: Work- shops co-located with the International Conference on Information and Knowledge Management, CIKM, Gold Coast, Queensland, Australia, November 1-5, 2021
Navarrete, E., Hoppe, A., Ewerth, R.: A review on recent advances in video-based learning research: Video features, interaction, tools, and technologies. In: Work- shops co-located with the International Conference on Information and Knowledge Management, CIKM, Gold Coast, Que...
2021
-
[23]
arXiv preprint abs/2301.13617 (2023)
Navarrete, E., Nehring, A., Schanze, S., Ewerth, R., Hoppe, A.: A closer look into recent video-based learning research: A comprehensive review of video From Formulas to Figures 15 characteristics, tools, technologies, and learning effectiveness. arXiv preprint abs/2301.13617 ...
-
[24]
Educational Psychologist 38(1), 1–4 (2003)
Paas, F., Renkl, A., Sweller, J.: Cognitive load theory and instructional design: Recent developments. Educational Psychologist 38(1), 1–4 (2003). https://doi. org/10.1207/S15326985EP3801_1
2003 doi
-
[25]
Journal of Machine Learning Research12, 2825–2830 (2011)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Mach...
2011
-
[26]
Educational Psychology Review31(2), 389–419 (2019)
Rey, G.D., Beege, M., Nebel, S., Wirzberger, M., Schmitt, T.H., Schneider, S.: A meta-analysis of the segmenting effect. Educational Psychology Review31(2), 389–419 (2019). https://doi.org/10.1007/s10648-018-9456-4
2019 doi
-
[27]
Technology, Knowledge and Learning 26(4), 1061–1077 (Dec 2021)
Sablić, M., Mirosavljević, A., Škugor, A.: Video-based learning (vbl)—past, present and future: an overview of the research published from 2008 to 2019. Technology, Knowledge and Learning 26(4), 1061–1077 (Dec 2021). https://doi.org/10.1007/ s10758-020-09455-5
2008
-
[28]
Learning and Instruction14(3), 293–305 (2004)
Schwan, S., Riempp, R.: The cognitive benefits of interactive videos: learning to tie nautical knots. Learning and Instruction14(3), 293–305 (2004). https://doi. org/10.1016/j.learninstruc.2004.06.005
2004 doi
-
[29]
In: International Conference on Multimedia, MM 2022, Lisboa, Portugal, October 10-14, 2022
Seng, T.: Enriching existing educational video datasets to improve slide classifica- tion and analysis. In: International Conference on Multimedia, MM 2022, Lisboa, Portugal, October 10-14, 2022. pp. 6930—-6934. Association for Computing Ma- chinery (2022). https://doi.org/10....
2022
-
[30]
Computers & Education59(2), 274–280 (2012)
Spanjers, I.A., van Gog, T., Wouters, P., van Merriënboer, J.J.: Explaining the segmentation effect in learning from animations: The role of pausing and temporal cueing. Computers & Education59(2), 274–280 (2012). https://doi.org/10.1016/ j.compedu.2011.12.024
2012
-
[31]
https://doi.org/10.1016/j.chb.2010
Spanjers, I.A., Wouters, P., van Gog, T., van Merriënboer, J.J.: An expertise rever- saleffectofsegmentationinlearningfromanimatedworked-outexamples.Comput- ers in Human Behavior 27(1), 46–52 (2011). https://doi.org/10.1016/j.chb.2010. 05.011
2011 doi
-
[32]
British Journal of Edu- cational Technology55(3), 933–956 (2024)
Yu, Q., Gou, J., Li, Y., Pi, Z., Yang, J.: Introducing support for learner control: Temporal and organizational cues in instructional videos. British Journal of Edu- cational Technology55(3), 933–956 (2024). https://doi.org/10.1111/bjet.13408
2024 doi
-
[33]
Journal of Media Psychology29(1), 18–30 (2017)
van der Zee, T., Admiraal, W., Paas, F., Saab, N., Giesbers, B.: Effects of sub- titles, complexity, and language proficiency on learning from online education videos. Journal of Media Psychology29(1), 18–30 (2017). https://doi.org/10.1027/ 1864-1105/a000208
2017
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.