Pith. sign in

REVIEW 4 major objections 4 minor 17 references

Evaluating Alignment Approaches in Superimposed Time-Series and Temporal Event-Sequence Visualizations

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Alignment in time-event visualizations should be chosen by task: dual alignment helps read intermediate events but hurts duration judgment.

desk verdict First controlled evaluation of dual-event alignment, but the headline numbers all flow through a post-hoc participant filter that is not pre-specified and is tied to the outcome measures. read the letter →

arxiv 1908.07316 v1 pith:3WCS2RVF submitted 2019-08-20 cs.HC

classification cs.HC
keywords sentineleventalignmenttemporalsequencevisualizationtime-seriescompositecrowdsourceduserstudydual-eventType1diabetesdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the usefulness of sentinel-event alignment in superimposed time-series and event-sequence visualizations depends on the analytic question, and that the newest form—dual-event alignment—is a real improvement for one class of questions and a real drawback for another. In a crowdsourced controlled experiment using Type 1 diabetes timelines with 3 and 14 days of data, the authors compared no alignment, single-event alignment, and two dual-event alignments. They report that for understanding intermediate events between two sentinel events, dual-event alignment was far more correct (71% vs. 18% for no alignment and single alignment at the 14-day scale), whereas for judging the duration between the two sentinel events, no alignment was far better (88% correctness vs. 36% for stretch dual alignment, with lower error and faster completion). Single-event alignment showed no dependable gain over no alignment for precursor and aftereffect events. The paper concludes that designers should match alignment choice to task type rather than applying one alignment strategy everywhere.

What carries the argument

The central object is sentinel-event alignment, a family of transforms that shifts or rescales each row of a temporal visualization so that one or two chosen events line up vertically across rows. NoAlign leaves true time intact; SingleAlign aligns rows by a single sentinel event (e.g., lunch); DualLeft aligns rows by two sentinel events at their left positions; DualStretch aligns rows by two sentinel events while stretching or compressing the intervening time scale. The experiment's task machinery consists of six low-level questions built from three task abstractions—precursor events, aftereffect events, and intermediate events—each run at two scales (3 rows and 14 rows) on de-identified Type 1 diabetes data. The load-bearing comparison is Task 4 (intermediate duration) versus Task 5 (intermediate co-occurrence) at the 14-day scale, where the same family of transforms produces opposite winners.

What would settle it

Re-running the same 12-question protocol with the exclusion rule fixed before data collection—or simply re-analyzing the 108 accepted participants without the k-means "speeding" filter—would settle the central claim. The claim fails if Task 4 at the 14-day scale no longer favors NoAlign over DualStretch on correctness, time, and error, or if Task 5 no longer favors dual alignment over NoAlign and SingleAlign.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is an interaction between alignment technique and task type in composite temporal visualizations. Using 14 days of superimposed blood-glucose time series and event markers, participants asked to identify intermediate events between two sentinel meals answered correctly 71% of the time with dual-left or dual-stretch alignment, versus 18% with no alignment or single-event alignment. Participants asked to read the duration between the two sentinel meals did the opposite: no alignment gave 88% correctness versus 36% for dual-stretch, completion time of 55 seconds versus 101 seconds for dual-left, and error of 1.5% versus 8.4% for dual-stretch. At the 3-day scale most differences disappeared. The paper therefore concludes that dual-event alignment supports intermediate-event reading, especially with more rows of data, but actively misleads for duration estimation, and that single-event alignment contributes no reliable advantage over no alignment in this superimposed setting.

Load-bearing premise

The headline comparisons rest on the post-hoc decision, made after viewing the data, to remove 46 of 108 accepted participants as "speeding" using a clustering-derived bound; if those participants were actually paying attention, or if the bound is arbitrary, every reported effect can change.

Editorial extensions

If this is right

  • Designers of composite temporal visualizations should use dual-event alignment, particularly the stretch variant, when users need to see what happens between two sentinel events; it was the clear correctness winner for intermediate-event tasks.
  • Dual-event alignment should be avoided for questions about the duration or size of intervals between sentinel events; no alignment beat both dual variants on correctness, time, and error for the duration task.
  • Single-event alignment did not justify itself in this setting: it was no more correct than no alignment for precursor/aftereffect tasks and could be slower at the smaller scale.
  • Differences among alignment approaches grow with the number of rows; at 3 days the conditions were largely equivalent, so alignment decisions matter most for dense, multi-row displays.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This extends the paper's result: because stretch alignment rescales the time axis between sentinel events, any task that depends on comparing interval lengths, rates, or durations in an aligned composite view should be expected to suffer, in domains beyond diabetes.
  • A testable extension the paper leaves open: repeat the study with simpler point-event data and lower visual clutter; the absence of a single-alignment benefit may be specific to superimposed, multi-encoding displays, and categorical encoding of event type might restore it.
  • The untested symmetry assumption—that dual alignment with right justification behaves like left justification—could be checked directly, since the two variants are not visually equivalent when reading order matters.
  • The reported effect sizes are large, but the post-hoc speeding filter means a pre-registered replication with a fixed exclusion rule is the natural next check; the supplemental unfiltered data would support such an analysis.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports a controlled between-subjects experiment on Amazon Mechanical Turk comparing four alignment approaches for superimposed time-series and temporal event-sequence visualizations: no alignment, single-event alignment, and two dual-event alignment variants (left and stretch justification). Participants answered twelve questions (six tasks at two data scales) probing precursor/aftereffect events, intermediate events, and durations. The headline findings are that dual-event alignment improves correctness for identifying intermediate events (Task 5, 14-day scale: 71% vs 25% for NoAlign and 11% for SingleAlign), while no alignment is best for judging durations (Task 4, 14-day scale: 88% vs 36% for DualStretch, with faster times and lower error). The authors provide stimuli, data, and source code on OSF.

Significance. If the results are valid, the paper is a valuable empirical contribution: it is one of the few comparative evaluations of dual-event alignment in superimposed visualizations, and it offers a domain-independent task abstraction for temporal event-sequence and timing analysis. The open-data and open-materials practices are commendable and make independent verification possible. However, the central empirical claims are currently undermined by a post-hoc participant-exclusion step that operates on the same outcome variables used in the analysis, and by small per-condition samples with numerous unadjusted significance tests. These issues are load-bearing because every headline percentage in the abstract is computed from the filtered 62 participants.

major comments (4)
  1. [Section 4.3] The "speeding" filter is post-hoc and data-dependent, and it is applied to the very measures that define the study's dependent variables. After pre-specified rejection criteria had already been applied, the authors ran a k-means clustering on the collected time-spent and correctness data, discovered a "clear clustering effect," and removed 46 of the 108 accepted participants. The stated bound ("four answers out of 12 questions and at least 8 minutes") was derived from the observed data, not pre-registered. Because the filter uses the outcome variables, it can create or exaggerate between-condition differences if the removed participants are not evenly distributed across the four conditions. This is load-bearing: the Task 4 and Task 5 14-day results in the abstract (88% vs 36%; 71% vs 25% and 11%) all come from the filtered n=62, and the paper does not show that these conclusions survive without the filter.
  2. [Section 5, opening paragraph] The manuscript explicitly states that "results with the unfiltered data ... are relegated to supplemental material." For a paper whose central claims depend entirely on a controversial post-hoc exclusion, relegating the unfiltered analysis to the supplement is insufficient. The authors should report the unfiltered results in the main text or, at minimum, provide a full robustness table showing that the significant differences in Tasks 4 and 5 persist with all 108 accepted participants, or with any sensible alternative filter (e.g., the pre-specified criteria alone). Without this, the reader cannot verify whether the headline percentages are an artifact of the data-dependent exclusion step.
  3. [Sections 4.4 and 5.2] The multiple-comparison problem is not adequately handled. The analysis runs separate chi-square or Kruskal-Wallis tests for each of 12 task-scale combinations and for three measures (correctness, time, error), yet the Bonferroni adjustment mentioned in Section 4.4 appears to apply only to post-hoc pairwise comparisons after a significant omnibus test. With about 36 tests at alpha = .05, several nominally significant results are expected by chance. For example, the Task 5 14-day correctness comparison between DualLeft/SingleAlign (p = .03) and the Task 4 14-day error comparison (p = .01) would not survive a family-wise correction over all 36 tests. The paper should either apply an appropriate multiple-comparison control across the question/measure families or present a sensitivity analysis showing that the headline conclusions are robust to the correction.
  4. [Section 4.3 and Results] The per-condition sample sizes after filtering are 14, 14, 16, and 18, which makes the binary correctness analyses fragile. With these counts, chi-square tests on 2x2 or 2x4 tables often have expected cell counts below 5, so the reported p-values (e.g., Task 5 14-day: NoAlign 25% of 16, SingleAlign 11% of 18, DualLeft 71% of 14, DualStretch 71% of 14) are not reliable. The authors should use Fisher's exact tests or report effect sizes with confidence intervals, and they should explicitly note the small cell counts when interpreting the 71% vs 25% and 88% vs 36% contrasts. This is directly relevant to the strength of the abstract's "clear winner" language.
minor comments (4)
  1. [Section 4.3] In the participant counts, "DualRight" appears to be a typo; the paper's four conditions are NoAlign, SingleAlign, DualLeft, and DualStretch. Please correct the labels for consistency.
  2. [Section 4.3] The phrase "the bound as four answers out of 12 questions and at least 8 minutes were spent on the questions" is ambiguous. Please specify whether the criterion is "at most four correct answers" and whether the time bound is a minimum or maximum, and state the bound as a clear conjunction of conditions.
  3. [Figure 4] The legend in Figure 4 spells "DualStrech"; this should be "DualStretch."
  4. [Section 5.2] In the Task 5 3-day result, "DualStretch (100%)" with n=14 implies all 14 participants answered correctly; reporting the raw counts (e.g., 14/14) alongside percentages would help readers assess the reliability of that contrast.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a controlled empirical comparison, and its conclusions are not forced by construction or by self-citation.

full rationale

This is an empirical experiment rather than a derivation, so the circularity patterns do not apply. The hypotheses are grounded in prior work, including the authors' own IDMVis paper, but the study's outcomes are not embedded in the inputs: H1, predicting a benefit for single-event alignment, was contradicted by the results, and the headline findings (dual-event alignment helping for intermediate events, NoAlign helping for duration) emerge from measured participant performance rather than from any fitted parameter or definitional equivalence. The authors do cite their own IDMVis work for the dual-event alignment technique and stimuli, but that citation provides the visualization designs under test, not the experimental conclusions; the conclusions are independently generated by the controlled comparison. The post-hoc k-means filtering of 46 of 108 participants in Section 4.3 is a genuine methodological vulnerability because the filter is derived from the same outcome variables used in the analysis and the unfiltered results are relegated to supplemental material, but this is a data-quality and statistical-inference concern, not circular reasoning: the reported percentages and p-values are computed from data rather than derived from the filter by construction. Under the review rules, 'this is not standard consensus' and methodological fragility are correctness risks, not circularity, and no quoted step shows an equation or claim reducing to its own inputs. Therefore the honest finding is no significant circularity, score 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper contributes an empirical comparison; it contains no mathematical derivation. The only fitted number is the post-hoc participant-exclusion threshold. The main assumptions are about whether the chosen stimuli, tasks, and participant pool represent real-world composite visualization use; these are domain assumptions rather than axioms in the formal sense.

free parameters (1)
  • Post-hoc speeding filter threshold = 4 or fewer correct answers out of 12 and at least 8 minutes on questions
    Derived from k-means clustering on the collected data in Section 4.3; determines which 46 of 108 participants are excluded, and thus which data support the central claims.
assumptions (3)
  • domain assumption The IDMVis stimuli and Type 1 diabetes data are representative of superimposed time-series and event-sequence visualizations.
    Generalization from this specific stimuli set to the class of composite visualizations is assumed throughout, especially in the design implications in Section 6.
  • domain assumption The task set (precursor, aftereffect, intermediate, duration) covers the relevant user tasks for these visualizations.
    Task selection is justified by synthesis of prior work in Section 2.2, but the completeness of the abstraction is an assumption.
  • ad hoc to paper K-means clustering with the chosen bound correctly identifies low-effort participants.
    The method is introduced post hoc in Section 4.3 and is not validated against an independent ground truth for attentiveness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Alignment Approaches in Superimposed Time-Series and Temporal Event-Sequence Visualizations." pith.science (2026). https://pith.science/paper/3WCS2RVF

@misc{pith2026190807316,
  author       = {Pith},
  title        = {Pith review of: Evaluating Alignment Approaches in Superimposed Time-Series and Temporal Event-Sequence Visualizations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3WCS2RVF}},
  note         = {Machine review of arXiv:1908.07316}
}
read the original abstract

Composite temporal event sequence visualizations have included sentinel event alignment techniques to cope with data volume and variety. Prior work has demonstrated the utility of using single-event alignment for understanding the precursor, co-occurring, and aftereffect events surrounding a sentinel event. However, the usefulness of single-event alignment has not been sufficiently evaluated in composite visualizations. Furthermore, recently proposed dual-event alignment techniques have not been empirically evaluated. In this work, we designed tasks around temporal event sequence and timing analysis and conducted a controlled experiment on Amazon Mechanical Turk to examine four sentinel event alignment approaches: no sentinel event alignment (NoAlign), single-event alignment (SingleAlign), dual-event alignment with left justification (DualLeft), and dual-event alignment with stretch justification (DualStretch). Differences between approaches were most pronounced with more rows of data. For understanding intermediate events between two sentinel events, dual-event alignment was the clear winner for correctness---71% vs. 18% for NoAlign and SingleAlign. For understanding the duration between two sentinel events, NoAlign was the clear winner: correctness---88% vs. 36% for DualStretch---completion time---55 seconds vs. 101 seconds for DualLeft---and error---1.5% vs. 8.4% for DualStretch. For understanding precursor and aftereffect events, there was no significant difference among approaches. A free copy of this paper, the evaluation stimuli and data, and source code are available at https://osf.io/78fs5

Figures

Figures reproduced from arXiv: 1908.07316 by the authors.

Figure 1
Figure 1. User task performance was evaluated on four temporal event sequence alignment visualization approaches: no sentinel [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Legend of the main visual encodings used in the study. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Tasks for evaluating precursor, aftereffect, intermediate, and co-occurrence of events in superimposed visualizations. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Correctness and time by scale, task, and approach with [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages

  1. [1]

    R. M. Bergenstal, A. J. Ahmann, T. Bailey, R. W. Beck, J. Bissen, B. Buckingham, L. Deeb, R. H. Dolin, S. K. Garg, R. Goland, I. B. Hirsch, D. C. Klonoff, D. F. Kruger, G. Matfin, R. S. Mazze, B. A. Ol- son, C. Parkin, A. Peters, M. A. Powers, H. Rodriguez, P. Southerland, E. S. Strock, W. Tamborlane, and D. M. Wesley. Recommendations for standardizing glu...

  2. [2]

    J. J. Chen, N. J. Menezes, A. D. Bradley, and T. North. Opportunities for crowdsourcing research on Amazon Mechanical Turk. Interfaces, 5(3):1, 2011

  3. [3]

    Y . Chen, P. Xu, and L. Ren. Sequence synopsis: Optimize visual summary of temporal event data. TVCG, 24(1):45–55, 2018. doi: 10. 1109/TVCG.2017.2745083

  4. [4]

    F. Du, B. Shneiderman, C. Plaisant, S. Malik, and A. Perer. Coping with volume and variety in temporal event sequences: Strategies for sharpening analytic focus. TVCG, 23(6):1636–1649, 2017. doi: 10. 1109/TVCG.2016.2539960

  5. [5]

    Gotz and H

    D. Gotz and H. Stavropoulos. DecisionFlow: Visual analytics for high- dimensional temporal event sequence data. TVCG, 20(12):1783–1792,

  6. [6]

    Gschwandtner, W

    T. Gschwandtner, W. Aigner, K. Kaiser, S. Miksch, and A. Seyfang. CareCruiser: exploring and visualizing plans, events, and effects interactively. In Proc. PacificVis, pp. 43–50, 2011. doi: 10.1109/ PACIFICVIS.2011.5742371

  7. [7]

    C.-J. Ho, A. Slivkins, S. Suri, and J. W. Vaughan. Incentivizing high quality crowdwork. In Proc. 24th International Conference on World Wide Web, WWW ’15, pp. 419–429, 2015. doi: 10.1145/2736277. 2741102

  8. [8]

    Javed and N

    W. Javed and N. Elmqvist. Exploring the design space of composite vi- sualization. In Proc. PacificVis, pp. 1–8, 2012. doi: 10.1109/PacificVis. 2012.6183556

Show all 17 references
  1. [9]

    Kosara and C

    R. Kosara and C. Ziemkiewicz. Do mechanical turks dream of square pie charts? In Proc. BELIV Workshop: BEyond Time and Errors: Novel evaLuation Methods for Information Visualization, pp. 63–70, 2010. doi: 10.1145/2110192.2110202

  2. [10]

    Mamykina, E

    L. Mamykina, E. M. Heitkemper, A. M. Smaldone, R. Kukafka, H. J. Cole-Lewis, P. G. Davidson, E. D. Mynatt, A. Cassells, J. N. Tobin, and G. Hripcsak. Personal discovery in diabetes self-management: discovering cause and effect using self-monitoring data. Journal of biomedical ...

  3. [11]

    Monroe, R

    M. Monroe, R. Lan, H. Lee, C. Plaisant, and B. Shneiderman. Temporal event sequence simplification. TVCG, 19(12):2227–2236, 2013. doi: 10.1109/TVCG.2013.200

  4. [12]

    Plaisant and B

    C. Plaisant and B. Shneiderman. The diversity of data and tasks in event analytics. In Proc. IEEE VIS 2016 Workshop on Temporal & Sequential Event Analysis, 2016

  5. [13]

    T. D. Wang, C. Plaisant, A. J. Quinn, R. Stanchak, S. Murphy, and B. Shneiderman. Aligning temporal data by sentinel events: discover- ing patterns in electronic health records. In Proc. CHI, pp. 457–466,

  6. [14]

    Wongsuphasawat and J

    K. Wongsuphasawat and J. Lin. Using visualizations to monitor changes and harvest insights from a global-scale logging infrastructure at Twitter. In Proc. VAST, pp. 113–122, 2014. doi: 10.1109/V AST. 2014.7042487

  7. [15]

    Zhang, K

    Y . Zhang, K. Chanana, and C. Dunne. IDMVis: Temporal event sequence visualization for type 1 diabetes treatment decision support. TVCG, 25(1):512–522, 2019. doi: 10.1109/TVCG.2018.2865076

  8. [2008]

    doi: 10.1145/1357054.1357129

  9. [2014]

    doi: 10.1109/TVCG.2014.2346682

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.