{"id":"2b883b3a-ce59-478a-93f0-e87573ca34a2","arxiv_id":"2505.01753","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In 25 real-world educational videos, higher visual complexity from textual elements is associated with more viewer pauses, rewinds, and dropouts.","lead":"The authors annotated 25 real physics and chemistry educational videos to count visual elements per frame, then matched those counts against 8,180 anonymous viewing sessions. They found that frames with many textual elements, such as equations, are associated with more pauses, rewinds, and dropouts among viewers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Permutation test in §4.3 shuffles the visual-complexity series unconstrained, destroying its autocorrelation; DTW's two-bin window then favors the original series, biasing every p-value in Table 1 toward zero.","rationale":"I read the paper as an empirical claim that visual complexity, especially textual elements, is associated with more pauses, rewinds, and dropouts in authentic educational-video viewing. For that claim to be supported, the permutation test in Section 4.3 must produce valid p-values. The reader identifies annotation quality as the weakest assumption, but I find a more fundamental problem: the null distribution is generated by unrestricted random permutation of the visual-complexity series, which destroys its temporal autocorrelation. The modified DTW statistic with a two-bin warping window is sensitive to local temporal structure, so the original smooth or bursty series can be aligned to the interaction series at artificially low cost, while shuffled series yield systematically higher costs. This biases every p-value toward zero and affects all rows of Table 1, including the binary condition. The issue is well known in time-series surrogate testing: the null should preserve the autocorrelation of the predictor (e.g., circular shifts or block permutations) to test for genuine cross-series alignment. The reaction window being estimated from the same data further inflates the apparent effect. My proposed circular-shift surrogate test would settle whether the headline associations survive a null that preserves autocorrelation. If they do not, the central claim is unsupported and the paper would need substantial revision. For this reason, I would adjust the verdict from CONDITIONAL to UNVERDICTED until the reanalysis is performed.","tokens_in":11040,"tokens_out":9810,"duration_ms":108152,"concrete_test":"Re-run the Section 4.3 analysis using surrogate visual-complexity series that preserve autocorrelation: for each video, generate 5000 circular shifts of VisCom_v (or block-permute with block length at least the warping window, e.g., 5 bins) and recompute the DTW cost and p_v; then re-aggregate p and PES for Table 1. If the reported p-values (especially pause T p=0.0, rewind T p=0.001, dropout T p=0.017) become non-significant or PES shrink below the bootstrap CI, the published associations are artifacts of the permutation null. This requires only the code already released.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim rests on the permutation test in Section 4.3. For each video, the null distribution is generated by randomly permuting the visual-complexity time series VisCom_v and recomputing the modified DTW cost. Random permutation destroys the temporal autocorrelation of VisCom_v, but the DTW statistic is not invariant to autocorrelation: with only a two-bin warping window, the algorithm can align the original smooth or bursty series to the interaction series at low cost, whereas shuffled series have many isolated spikes that cannot be absorbed by the window and yield systematically higher costs. Because both the visual-complexity series and the interaction series are autocorrelated (changes cluster at slide transitions; pauses and rewinds cluster in bursts), the observed cost is biased low relative to this null, inflating significance for every row of Table 1. The problem affects the binary condition as well. This is independent of annotation quality: even with perfect annotations, the test can reject independence when the two series merely share similar autocorrelation structure. The 10-second reaction window, selected from the same data, compounds the bias. Consequently the headline results (pause/T VisCom p=0.0, PES=0.89; rewind/T p=0.001; dropout/T p=0.017) may not survive a null that preserves autocorrelation.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether visual complexity in STEM educational videos is associated with user interactions in real-world viewing logs. The authors develop a fine-grained taxonomy of visual objects, annotate 25 videos from the TIB AV-Portal (SciFox channel), and construct per-video time series of the number of newly appearing visual objects (textual vs. information-visualization categories) in 5-second bins. They relate these to binned user interactions (pauses, rewinds, skips, dropouts) using a modified dynamic time warping (DTW) test statistic with a causality constraint and a 10-second reaction window, and assess significance with a permutation test aggregated across videos via Fisher's method. The headline findings are that textual visual complexity is associated with more pauses (p=0.0, PES=0.89), rewinds (p=0.001, PES=0.79), and dropouts (p=0.017, PES=0.47), while information-visualization elements show no such effects.","tokens_in":11287,"tokens_out":9850,"duration_ms":92995,"significance":"The study addresses a genuinely understudied question with authentic interaction data, which is a strength relative to laboratory studies. The taxonomy is a useful contribution, and the public code and careful preprocessing (e.g., normalizing by active sessions, unique-session counting) support reproducibility. If the statistical analysis is validated, the findings would have practical implications for educational video design, suggesting that text-heavy segments impose processing costs that manifest in pausing, rewinding, and dropout. However, the current statistical machinery has serious validity concerns (permutation null, reaction-window selection, multiple comparisons, annotation reliability), so the empirical contribution is not yet established at the level claimed.","major_comments":[{"comment":"The permutation test permutes the visual-complexity series VisCom_v without preserving its temporal autocorrelation, while the modified DTW test statistic with a two-bin warping window is sensitive to the autocorrelation structure of the series. Because both VisCom_v and the interaction series are bursty (visual changes cluster at slide transitions and interactions cluster in bursts), the original series can achieve a lower DTW cost than a randomly shuffled series that has been whitened, so the null distribution is shifted toward higher costs and the p-values in Table 1 are biased toward zero. This affects every row of Table 1, including the headline pause/T VisCom result (p=0.0, PES=0.89) and rewind-from/T (p=0.001, PES=0.79). Please re-run the analysis under a null that preserves the autocorrelation of VisCom_v (e.g., block permutation, circular shifting, or phase-randomized surrogates) and report the resulting p-values and PES.","section":"§4.3, Table 1"},{"comment":"The causality constraint is described inconsistently with its stated intent. The text says that setting all costs below the main diagonal to infinity 'prevents future user interactions from being mapped to past visual complexity changes,' but in the matrix orientation implied by Algorithm 1 (s1 = visual complexity, s2 = user interactions, D[i,j] = (s1[j]-s2[i])^2), cells below the diagonal have j < i and correspond to mapping a current interaction to an earlier visual stimulus, which is precisely the desired causal direction. Either the orientation or the direction of the constraint is wrong. Please specify the exact indexing and verify the implementation in the public code, because a reversed constraint would change the test statistic into one that aligns interactions with future stimuli and invalidates the interpretation.","section":"§4.3, Algorithm 1"},{"comment":"The paper reports 36 significance tests (6 actions × 3 modality groupings × 2 conditions) in Table 1 without any multiple-comparison correction. With a Bonferroni threshold of approximately 0.0014, only the pause/T VisCom (p=0.0) and rewind-from/T VisCom (p=0.001) results remain significant; the dropout/T result (p=0.017) would not survive. Please apply a suitable correction (e.g., Benjamini-Hochberg) or explicitly label the analysis as exploratory, and revise the conclusions to match the corrected outcome.","section":"§4.3, Table 1"},{"comment":"The predictor series is built from annotations by a single primary annotator (A1), with inter-annotator agreement measured on only five frames against A2 before A1 completed the full annotation. Because the visual-complexity time series is defined by the appearance of new objects across frames, the static IoU check does not validate the temporal tracking that determines 'new' objects, and any annotation drift or tracking error is propagated into every permutation test. Please report per-video or per-change-event annotation quality (e.g., a sample of change events re-annotated by A2) or otherwise demonstrate that the predictor series is reliable.","section":"§3.2/§4.1"},{"comment":"The two-bin (10 s) reaction window and the 5 s bin size are chosen based on exploratory visual analysis of the same data that is later used for hypothesis testing. This post-hoc tuning of the test statistic's parameters can inflate the apparent significance because the window is selected to make the alignment cost small. Please report a sensitivity analysis over the window size (e.g., 1, 2, 3 bins) and, ideally, confirm the findings on a hold-out set or with pre-registered parameter choices.","section":"§4.3"}],"minor_comments":[{"comment":"The phrase 'Reviewer guidelines were established and continuously adapted' appears to be a typo; it should presumably read 'Annotation guidelines were established and continuously adapted.'","section":"§3.2"},{"comment":"The sentence 'The negative sign ensures a positive PESv indicates a positive.' is incomplete; it should specify that a positive PESv indicates a positive relationship between visual complexity and the respective user interaction.","section":"§4.3, Eq. (2)"},{"comment":"The meaning of 'time' under the 'action' column is ambiguous; the labels 'from' and 'to' for rewind and skip should be defined in the caption (e.g., 'from' = source position of the seek, 'to' = target position).","section":"Table 1"},{"comment":"The pseudocode declares a window parameter but does not specify its exact indexing; please state precisely which cells are set to infinity in terms of i and j, both for the causality constraint and the reaction window.","section":"Algorithm 1"},{"comment":"The statement 'Based on exploratory visual analysis, we observed that user responses often occur up to 10 seconds after a visual stimulus' should be accompanied by a concrete description of the exploration (e.g., number of videos inspected and the procedure used to arrive at the 10 s threshold).","section":"§4.3"},{"comment":"The abbreviation 'CTLM' should be 'CTML' (Cognitive Theory of Multimedia Learning) to match the usage in the introduction.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially interesting for an HCI/educational-technology audience, but the statistical analysis needs substantive rework. The public code is valuable; I recommend asking the authors to verify the DTW orientation against the code and to re-run the analysis with an autocorrelation-preserving null. If the reversed causality constraint is confirmed in the code, the paper would need rejection; as written, the ambiguity is fixable. The multiple-comparison issue and the annotation-reliability limitation should also be addressed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the note. The stress-test concern is right: the permutation test in §4.3 shuffles the visual-complexity series without preserving its temporal autocorrelation, and that biases the p-values in Table 1 downward. It's not a subtle point. Both the visual-change series and the interaction series burst at slide transitions; a random shuffle destroys that shared structure, so the DTW cost on the original alignment looks artificially good relative to the shuffled null. A block-permutation or other autocorrelation-preserving null is the obvious fix, and without it every significance claim in the paper is suspect.\n\nThat said, the paper has real value. The taxonomy for visual objects in STEM videos is carefully designed and clearly presented, and applying it to 8,180 real sessions from the TIB AV-Portal is a genuinely useful step beyond lab studies. The code is public, and the writing is honest. I also credit the authors for using a causality-constrained DTW; the idea is good even if the inference procedure is flawed.\n\nThe other issues are secondary but real: the 10-second reaction window was chosen from the same data, there are 36 tests without multiple-comparison correction, only five frames were double-annotated (IoU 0.88, but drift is unassessed), and the logs and annotations aren't released, only code. The direction of the main effect—text-heavy sections inducing more pauses, rewinds, and dropouts—is plausible and consistent with prior lab findings, but the current evidence doesn't support the specific p-values as reported.\n\nWho is this for: people working on video-based learning, learning analytics, and multimedia content analysis. The taxonomy and dataset could be citable even if the statistical conclusions need revisiting. I'd send this to peer review with major revisions; a serious referee should ask for the reanalysis and, ideally, the annotation data. If that comes back clean, this becomes a solid empirical paper. For now, take the headline numbers with a grain of salt.","headline":"The taxonomy and real-world dataset are the real contributions, but the permutation test's failure to preserve autocorrelation undermines the headline p-values as reported.","tokens_in":11826,"tokens_out":4128,"would_cite":false,"duration_ms":44631,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-heavy video moments drive pauses, rewinds, dropouts","keywords":["user interactions","user behavior","visual complexity","educational videos","STEM education","dynamic time warping","permutation testing","video analytics"],"falsifier":"Independently re-annotate all 25 videos, or a large random sample of them, with a second annotator or an automated object detector using the same taxonomy, recompute the visual-complexity time series, and rerun the permutation tests; if the reported associations ($p = 0.0$, PES $= 0.89$ for pausing; $p = 0.017$, PES $= 0.47$ for dropouts; $p = 0.001$, PES $= 0.79$ for rewinds) do not replicate with the independent series, the original result reflects the single annotator's labeling habits rather than a property of the videos.","tokens_in":10832,"feed_emoji":"🎓","tokens_out":13212,"duration_ms":118893,"temperature":0.7,"pith_summary":"This paper tries to establish that the visual complexity of educational videos—defined as the number of visually distinguishable, meaningful elements on screen, especially textual ones like equations and formulas—changes how real viewers watch. Using anonymized viewing logs from 8,180 sessions across 25 physics and chemistry videos, it reports that sections where many textual elements appear trigger significantly more pauses, rewinds, and dropouts, while graphical elements do not. The authors interpret this through cognitive-load theory: text-heavy bursts demand extra processing time, and when the demand is too high, learners stop the video or leave it. If true, the result gives video creators a measurable design lever—reduce text spikes or break them into smaller steps—rather than relying on intuition about what makes a lesson hard to follow.","feed_headline":"Text-heavy video moments drive pauses, rewinds, dropouts","feed_subtitle":"Analysis of 8,180 real sessions links textual visual complexity, not graphics, to pauses, rewinds, and dropouts.","key_machinery":"The load-bearing machinery is a two-part pipeline. First is the paper's fine-grained taxonomy of visual objects in STEM videos, which classifies annotated elements into textual categories (equations, terms, lines of text) and information-visualization categories (diagrams, tables, charts, technical drawings), making visual complexity a concrete count of distinguishable meaningful objects per frame. Second is a modified Dynamic Time Warping (DTW) procedure used as the test statistic in a permutation test: it aligns the visual-complexity time series with each user-interaction time series while enforcing a causality constraint that interactions can only map to preceding visual changes and a reaction-window constraint that limits mappings to about ten seconds (two five-second bins) after a stimulus. Per-video p-values are aggregated with Fisher's method, and a permutation effect size (PES) is computed as the standardized distance between the observed DTW cost and the null distribution.","core_discovery":"The paper's central claim is that visual complexity, measured as the number of newly appearing annotated objects per five-second bin, is statistically associated with user behavior in authentic viewing logs, and the association is driven by textual elements. It reports that users paused more frequently in sections with high textual change counts ($p = 0.0$, PES $= 0.89$), paused more under overall visual complexity ($p = 0.009$, PES $= 0.64$), rewound away from text-heavy sections more often ($p = 0.001$, PES $= 0.79$), and dropped out of videos more often at textually complex points ($p = 0.017$, PES $= 0.47$). The binary version of the complexity signal—whether any change occurred—was weaker or non-significant for dropouts and rewinds, which the authors take to mean the magnitude of textual change, not the mere occurrence, drives behavior. No significant effects were found for graphical or visualization elements, leading to the conclusion that on-screen text imposes higher cognitive load than graphics in real learning settings.","pith_inferences":["A testable extension the paper does not run: replace equation-heavy text segments with narrated visualizations of the same content and compare pause, rewind, and dropout rates; the paper's text-vs-graphics asymmetry predicts fewer disruptions in the visualization condition.","Varying the 10-second reaction window in the DTW analysis could produce a temporal response curve showing how quickly viewers pause or rewind after a text spike, distinguishing immediate overload from delayed comprehension attempts.","Automating the taxonomy with an object detector would let the same analysis scale from 25 videos to thousands and turn the correlational finding into a practical pre-publication screening tool for video producers."],"forward_implications":["Video creators can use the number of newly appearing text objects per time window as a risk score: text spikes mark where viewers will pause, rewind, or leave, so those segments can be redesigned or broken up before release.","Because graphical elements such as diagrams and charts showed no significant negative associations, shifting information from dense text to visual representations should reduce disruptive interactions without losing content.","The weaker binary-change results imply that the magnitude of a text burst matters more than the mere fact of change, so incremental text reveals are less disruptive than large simultaneous text additions.","The pattern supports adding system-determined pauses or interactive prompts at text-heavy points in the video, building on the segmenting-effect findings the paper cites."],"supporting_citations":[{"why":"Prior real-world finding that pausing associates with meaningful video structure; the baseline that motivates studying visual complexity as a driver.","marker":"[2]"},{"why":"Supplies the mathematics-education term taxonomy that the paper adapts into its fine-grained visual-object taxonomy.","marker":"[4]"},{"why":"Provides the general definition of visual complexity as level of detail, which the paper operationalizes as a count of meaningful objects.","marker":"[5]"},{"why":"Motivates the five-second binning of event logs and annotations into comparable time series.","marker":"[6]"},{"why":"Supplies the p-value aggregation procedure used to combine per-video permutation results into an overall significance statement.","marker":"[7]"},{"why":"Supplies the Cognitive Theory of Multimedia Learning, which frames why visual display properties should affect learner processing.","marker":"[17]"},{"why":"Provides authentic log-based evidence on why learners pause online videos and defines the direct time-series comparison that this paper's DTW-based method extends.","marker":"[21]"},{"why":"Supplies Cognitive Load Theory, the explanation for why textual complexity should increase pausing, rewinding, and dropout.","marker":"[24]"},{"why":"Establishes text quantity as a key dimension of visual-textual complexity, supporting the paper's separate analysis of textual elements.","marker":"[33]"}],"fun_headline_variants":["Text-heavy moments in STEM videos spur pauses and dropouts","Visual complexity: text, not graphics, drives video rewinds","On-screen formulas and text boost pauses, rewinds, dropouts","Real-world study: textual video complexity predicts user exits","Text-dense frames in educational videos trigger more pauses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire visual-complexity time series for all 25 videos rests on annotations by a single primary annotator, and agreement with a second annotator was measured on only five frames; if that annotator's judgments drift over the full corpus or the taxonomy misses relevant object classes, the predictor series is biased and every reported association could be an artifact of the labeling rather than the video content.","fun_headline_variants_meta":{"raw":{"variants":["Text-heavy moments in STEM videos spur pauses and dropouts","Visual complexity: text, not graphics, drives video rewinds","On-screen formulas and text boost pauses, rewinds, dropouts","Real-world study: textual video complexity predicts user exits","Text-dense frames in educational videos trigger more pauses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1607,"prompt_tokens":998,"completion_tokens":609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":527}},"tokens_in":614,"tokens_out":609,"duration_ms":6124,"temperature":1.0,"reasoning_tokens":527,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:11:07.272950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate all 25 videos, or a large random sample of them, with a second annotator or an automated object detector using the same taxonomy, recompute the visual-complexity time series, and rerun the permutation tests; if the reported associations ($p = 0.0$, PES $= 0.89$ for pausing; $p = 0.017$, PES $= 0.47$ for dropouts; $p = 0.001$, PES $= 0.79$ for rewinds) do not replicate with the independent series, the original result reflects the single annotator's labeling habits rather than a property of the videos.","supporting_citations":[{"cited_title":"In- ternational Journal of Education in Mathematics, Science, and Technology9(4), 585–613 (2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the mathematics-education term taxonomy that the paper adapts into its fine-grained visual-object taxonomy."},{"cited_title":"Psychological Bulletin132(1), 73–97 (2006)","cited_arxiv_id":null,"evidence_quote":"Provides the general definition of visual complexity as level of detail, which the paper operationalizes as a count of meaningful objects."},{"cited_title":"Cognition173, 133–137 (2018)","cited_arxiv_id":null,"evidence_quote":"Motivates the five-second binning of event logs and annotations into comparable time series."},{"cited_title":"Journal of Media Psychology29(1), 18–30 (2017)","cited_arxiv_id":null,"evidence_quote":"Establishes text quantity as a key dimension of visual-textual complexity, supporting the paper's separate analysis of textual elements."}],"review_version":1}