REVIEW 4 major objections 4 minor 17 references
Evaluating Alignment Approaches in Superimposed Time-Series and Temporal Event-Sequence Visualizations
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Alignment in time-event visualizations should be chosen by task: dual alignment helps read intermediate events but hurts duration judgment.
desk verdict First controlled evaluation of dual-event alignment, but the headline numbers all flow through a post-hoc participant filter that is not pre-specified and is tied to the outcome measures. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is sentinel-event alignment, a family of transforms that shifts or rescales each row of a temporal visualization so that one or two chosen events line up vertically across rows. NoAlign leaves true time intact; SingleAlign aligns rows by a single sentinel event (e.g., lunch); DualLeft aligns rows by two sentinel events at their left positions; DualStretch aligns rows by two sentinel events while stretching or compressing the intervening time scale. The experiment's task machinery consists of six low-level questions built from three task abstractions—precursor events, aftereffect events, and intermediate events—each run at two scales (3 rows and 14 rows) on de-identified Type 1 diabetes data. The load-bearing comparison is Task 4 (intermediate duration) versus Task 5 (intermediate co-occurrence) at the 14-day scale, where the same family of transforms produces opposite winners.
What would settle it
Re-running the same 12-question protocol with the exclusion rule fixed before data collection—or simply re-analyzing the 108 accepted participants without the k-means "speeding" filter—would settle the central claim. The claim fails if Task 4 at the 14-day scale no longer favors NoAlign over DualStretch on correctness, time, and error, or if Task 5 no longer favors dual alignment over NoAlign and SingleAlign.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is an interaction between alignment technique and task type in composite temporal visualizations. Using 14 days of superimposed blood-glucose time series and event markers, participants asked to identify intermediate events between two sentinel meals answered correctly 71% of the time with dual-left or dual-stretch alignment, versus 18% with no alignment or single-event alignment. Participants asked to read the duration between the two sentinel meals did the opposite: no alignment gave 88% correctness versus 36% for dual-stretch, completion time of 55 seconds versus 101 seconds for dual-left, and error of 1.5% versus 8.4% for dual-stretch. At the 3-day scale most differences disappeared. The paper therefore concludes that dual-event alignment supports intermediate-event reading, especially with more rows of data, but actively misleads for duration estimation, and that single-event alignment contributes no reliable advantage over no alignment in this superimposed setting.
Load-bearing premise
The headline comparisons rest on the post-hoc decision, made after viewing the data, to remove 46 of 108 accepted participants as "speeding" using a clustering-derived bound; if those participants were actually paying attention, or if the bound is arbitrary, every reported effect can change.
Editorial extensions
If this is right
- Designers of composite temporal visualizations should use dual-event alignment, particularly the stretch variant, when users need to see what happens between two sentinel events; it was the clear correctness winner for intermediate-event tasks.
- Dual-event alignment should be avoided for questions about the duration or size of intervals between sentinel events; no alignment beat both dual variants on correctness, time, and error for the duration task.
- Single-event alignment did not justify itself in this setting: it was no more correct than no alignment for precursor/aftereffect tasks and could be slower at the smaller scale.
- Differences among alignment approaches grow with the number of rows; at 3 days the conditions were largely equivalent, so alignment decisions matter most for dense, multi-row displays.
Reading between the lines
- This extends the paper's result: because stretch alignment rescales the time axis between sentinel events, any task that depends on comparing interval lengths, rates, or durations in an aligned composite view should be expected to suffer, in domains beyond diabetes.
- A testable extension the paper leaves open: repeat the study with simpler point-event data and lower visual clutter; the absence of a single-alignment benefit may be specific to superimposed, multi-encoding displays, and categorical encoding of event type might restore it.
- The untested symmetry assumption—that dual alignment with right justification behaves like left justification—could be checked directly, since the two variants are not visually equivalent when reading order matters.
- The reported effect sizes are large, but the post-hoc speeding filter means a pre-registered replication with a fixed exclusion rule is the natural next check; the supplemental unfiltered data would support such an analysis.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a controlled between-subjects experiment on Amazon Mechanical Turk comparing four alignment approaches for superimposed time-series and temporal event-sequence visualizations: no alignment, single-event alignment, and two dual-event alignment variants (left and stretch justification). Participants answered twelve questions (six tasks at two data scales) probing precursor/aftereffect events, intermediate events, and durations. The headline findings are that dual-event alignment improves correctness for identifying intermediate events (Task 5, 14-day scale: 71% vs 25% for NoAlign and 11% for SingleAlign), while no alignment is best for judging durations (Task 4, 14-day scale: 88% vs 36% for DualStretch, with faster times and lower error). The authors provide stimuli, data, and source code on OSF.
Significance. If the results are valid, the paper is a valuable empirical contribution: it is one of the few comparative evaluations of dual-event alignment in superimposed visualizations, and it offers a domain-independent task abstraction for temporal event-sequence and timing analysis. The open-data and open-materials practices are commendable and make independent verification possible. However, the central empirical claims are currently undermined by a post-hoc participant-exclusion step that operates on the same outcome variables used in the analysis, and by small per-condition samples with numerous unadjusted significance tests. These issues are load-bearing because every headline percentage in the abstract is computed from the filtered 62 participants.
major comments (4)
- [Section 4.3] The "speeding" filter is post-hoc and data-dependent, and it is applied to the very measures that define the study's dependent variables. After pre-specified rejection criteria had already been applied, the authors ran a k-means clustering on the collected time-spent and correctness data, discovered a "clear clustering effect," and removed 46 of the 108 accepted participants. The stated bound ("four answers out of 12 questions and at least 8 minutes") was derived from the observed data, not pre-registered. Because the filter uses the outcome variables, it can create or exaggerate between-condition differences if the removed participants are not evenly distributed across the four conditions. This is load-bearing: the Task 4 and Task 5 14-day results in the abstract (88% vs 36%; 71% vs 25% and 11%) all come from the filtered n=62, and the paper does not show that these conclusions survive without the filter.
- [Section 5, opening paragraph] The manuscript explicitly states that "results with the unfiltered data ... are relegated to supplemental material." For a paper whose central claims depend entirely on a controversial post-hoc exclusion, relegating the unfiltered analysis to the supplement is insufficient. The authors should report the unfiltered results in the main text or, at minimum, provide a full robustness table showing that the significant differences in Tasks 4 and 5 persist with all 108 accepted participants, or with any sensible alternative filter (e.g., the pre-specified criteria alone). Without this, the reader cannot verify whether the headline percentages are an artifact of the data-dependent exclusion step.
- [Sections 4.4 and 5.2] The multiple-comparison problem is not adequately handled. The analysis runs separate chi-square or Kruskal-Wallis tests for each of 12 task-scale combinations and for three measures (correctness, time, error), yet the Bonferroni adjustment mentioned in Section 4.4 appears to apply only to post-hoc pairwise comparisons after a significant omnibus test. With about 36 tests at alpha = .05, several nominally significant results are expected by chance. For example, the Task 5 14-day correctness comparison between DualLeft/SingleAlign (p = .03) and the Task 4 14-day error comparison (p = .01) would not survive a family-wise correction over all 36 tests. The paper should either apply an appropriate multiple-comparison control across the question/measure families or present a sensitivity analysis showing that the headline conclusions are robust to the correction.
- [Section 4.3 and Results] The per-condition sample sizes after filtering are 14, 14, 16, and 18, which makes the binary correctness analyses fragile. With these counts, chi-square tests on 2x2 or 2x4 tables often have expected cell counts below 5, so the reported p-values (e.g., Task 5 14-day: NoAlign 25% of 16, SingleAlign 11% of 18, DualLeft 71% of 14, DualStretch 71% of 14) are not reliable. The authors should use Fisher's exact tests or report effect sizes with confidence intervals, and they should explicitly note the small cell counts when interpreting the 71% vs 25% and 88% vs 36% contrasts. This is directly relevant to the strength of the abstract's "clear winner" language.
minor comments (4)
- [Section 4.3] In the participant counts, "DualRight" appears to be a typo; the paper's four conditions are NoAlign, SingleAlign, DualLeft, and DualStretch. Please correct the labels for consistency.
- [Section 4.3] The phrase "the bound as four answers out of 12 questions and at least 8 minutes were spent on the questions" is ambiguous. Please specify whether the criterion is "at most four correct answers" and whether the time bound is a minimum or maximum, and state the bound as a clear conjunction of conditions.
- [Figure 4] The legend in Figure 4 spells "DualStrech"; this should be "DualStretch."
- [Section 5.2] In the Task 5 3-day result, "DualStretch (100%)" with n=14 implies all 14 participants answered correctly; reporting the raw counts (e.g., 14/14) alongside percentages would help readers assess the reliability of that contrast.
Circularity Check
No significant circularity: the paper reports a controlled empirical comparison, and its conclusions are not forced by construction or by self-citation.
full rationale
This is an empirical experiment rather than a derivation, so the circularity patterns do not apply. The hypotheses are grounded in prior work, including the authors' own IDMVis paper, but the study's outcomes are not embedded in the inputs: H1, predicting a benefit for single-event alignment, was contradicted by the results, and the headline findings (dual-event alignment helping for intermediate events, NoAlign helping for duration) emerge from measured participant performance rather than from any fitted parameter or definitional equivalence. The authors do cite their own IDMVis work for the dual-event alignment technique and stimuli, but that citation provides the visualization designs under test, not the experimental conclusions; the conclusions are independently generated by the controlled comparison. The post-hoc k-means filtering of 46 of 108 participants in Section 4.3 is a genuine methodological vulnerability because the filter is derived from the same outcome variables used in the analysis and the unfiltered results are relegated to supplemental material, but this is a data-quality and statistical-inference concern, not circular reasoning: the reported percentages and p-values are computed from data rather than derived from the filter by construction. Under the review rules, 'this is not standard consensus' and methodological fragility are correctness risks, not circularity, and no quoted step shows an equation or claim reducing to its own inputs. Therefore the honest finding is no significant circularity, score 0.
Assumptions & free parameters
free parameters (1)
- Post-hoc speeding filter threshold =
4 or fewer correct answers out of 12 and at least 8 minutes on questions
assumptions (3)
- domain assumption The IDMVis stimuli and Type 1 diabetes data are representative of superimposed time-series and event-sequence visualizations.
- domain assumption The task set (precursor, aftereffect, intermediate, duration) covers the relevant user tasks for these visualizations.
- ad hoc to paper K-means clustering with the chosen bound correctly identifies low-effort participants.
Cite this review
Pith. "Pith review of Evaluating Alignment Approaches in Superimposed Time-Series and Temporal Event-Sequence Visualizations." pith.science (2026). https://pith.science/paper/3WCS2RVF
@misc{pith2026190807316,
author = {Pith},
title = {Pith review of: Evaluating Alignment Approaches in Superimposed Time-Series and Temporal Event-Sequence Visualizations},
year = {2026},
howpublished = {\url{https://pith.science/paper/3WCS2RVF}},
note = {Machine review of arXiv:1908.07316}
}
read the original abstract
Composite temporal event sequence visualizations have included sentinel event alignment techniques to cope with data volume and variety. Prior work has demonstrated the utility of using single-event alignment for understanding the precursor, co-occurring, and aftereffect events surrounding a sentinel event. However, the usefulness of single-event alignment has not been sufficiently evaluated in composite visualizations. Furthermore, recently proposed dual-event alignment techniques have not been empirically evaluated. In this work, we designed tasks around temporal event sequence and timing analysis and conducted a controlled experiment on Amazon Mechanical Turk to examine four sentinel event alignment approaches: no sentinel event alignment (NoAlign), single-event alignment (SingleAlign), dual-event alignment with left justification (DualLeft), and dual-event alignment with stretch justification (DualStretch). Differences between approaches were most pronounced with more rows of data. For understanding intermediate events between two sentinel events, dual-event alignment was the clear winner for correctness---71% vs. 18% for NoAlign and SingleAlign. For understanding the duration between two sentinel events, NoAlign was the clear winner: correctness---88% vs. 36% for DualStretch---completion time---55 seconds vs. 101 seconds for DualLeft---and error---1.5% vs. 8.4% for DualStretch. For understanding precursor and aftereffect events, there was no significant difference among approaches. A free copy of this paper, the evaluation stimuli and data, and source code are available at https://osf.io/78fs5
Figures
Reference graph
Works this paper leans on
-
[1]
R. M. Bergenstal, A. J. Ahmann, T. Bailey, R. W. Beck, J. Bissen, B. Buckingham, L. Deeb, R. H. Dolin, S. K. Garg, R. Goland, I. B. Hirsch, D. C. Klonoff, D. F. Kruger, G. Matfin, R. S. Mazze, B. A. Ol- son, C. Parkin, A. Peters, M. A. Powers, H. Rodriguez, P. Southerland, E. S. Strock, W. Tamborlane, and D. M. Wesley. Recommendations for standardizing glu...
-
[2]
J. J. Chen, N. J. Menezes, A. D. Bradley, and T. North. Opportunities for crowdsourcing research on Amazon Mechanical Turk. Interfaces, 5(3):1, 2011
work page 2011
-
[3]
Y . Chen, P. Xu, and L. Ren. Sequence synopsis: Optimize visual summary of temporal event data. TVCG, 24(1):45–55, 2018. doi: 10. 1109/TVCG.2017.2745083
arXiv 2018
-
[4]
F. Du, B. Shneiderman, C. Plaisant, S. Malik, and A. Perer. Coping with volume and variety in temporal event sequences: Strategies for sharpening analytic focus. TVCG, 23(6):1636–1649, 2017. doi: 10. 1109/TVCG.2016.2539960
arXiv 2017
-
[5]
D. Gotz and H. Stavropoulos. DecisionFlow: Visual analytics for high- dimensional temporal event sequence data. TVCG, 20(12):1783–1792,
-
[6]
T. Gschwandtner, W. Aigner, K. Kaiser, S. Miksch, and A. Seyfang. CareCruiser: exploring and visualizing plans, events, and effects interactively. In Proc. PacificVis, pp. 43–50, 2011. doi: 10.1109/ PACIFICVIS.2011.5742371
arXiv 2011
-
[7]
C.-J. Ho, A. Slivkins, S. Suri, and J. W. Vaughan. Incentivizing high quality crowdwork. In Proc. 24th International Conference on World Wide Web, WWW ’15, pp. 419–429, 2015. doi: 10.1145/2736277. 2741102
-
[8]
W. Javed and N. Elmqvist. Exploring the design space of composite vi- sualization. In Proc. PacificVis, pp. 1–8, 2012. doi: 10.1109/PacificVis. 2012.6183556
arXiv 2012
Show all 17 references
-
[9]
Kosara and C
R. Kosara and C. Ziemkiewicz. Do mechanical turks dream of square pie charts? In Proc. BELIV Workshop: BEyond Time and Errors: Novel evaLuation Methods for Information Visualization, pp. 63–70, 2010. doi: 10.1145/2110192.2110202
2010
-
[10]
Mamykina, E
L. Mamykina, E. M. Heitkemper, A. M. Smaldone, R. Kukafka, H. J. Cole-Lewis, P. G. Davidson, E. D. Mynatt, A. Cassells, J. N. Tobin, and G. Hripcsak. Personal discovery in diabetes self-management: discovering cause and effect using self-monitoring data. Journal of biomedical ...
2017 doi
-
[11]
Monroe, R
M. Monroe, R. Lan, H. Lee, C. Plaisant, and B. Shneiderman. Temporal event sequence simplification. TVCG, 19(12):2227–2236, 2013. doi: 10.1109/TVCG.2013.200
2013 doi
-
[12]
Plaisant and B
C. Plaisant and B. Shneiderman. The diversity of data and tasks in event analytics. In Proc. IEEE VIS 2016 Workshop on Temporal & Sequential Event Analysis, 2016
2016
-
[13]
T. D. Wang, C. Plaisant, A. J. Quinn, R. Stanchak, S. Murphy, and B. Shneiderman. Aligning temporal data by sentinel events: discover- ing patterns in electronic health records. In Proc. CHI, pp. 457–466,
-
[14]
Wongsuphasawat and J
K. Wongsuphasawat and J. Lin. Using visualizations to monitor changes and harvest insights from a global-scale logging infrastructure at Twitter. In Proc. VAST, pp. 113–122, 2014. doi: 10.1109/V AST. 2014.7042487
2014
-
[15]
Zhang, K
Y . Zhang, K. Chanana, and C. Dunne. IDMVis: Temporal event sequence visualization for type 1 diabetes treatment decision support. TVCG, 25(1):512–522, 2019. doi: 10.1109/TVCG.2018.2865076
2019
-
[2008]
doi: 10.1145/1357054.1357129
-
[2014]
doi: 10.1109/TVCG.2014.2346682
2014
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.