{"id":"4d58557a-b4fa-4681-8c77-35ff11d7b6fa","arxiv_id":"1908.00576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Dissimilarity-based axis ordering improves cluster accuracy in star glyph tasks, with the strongest benefit on cluttered data.","lead":"Star glyphs are small star-shaped charts for comparing multi-dimensional data points. This paper's user study found that ordering their axes to create spiky shapes helps people group them correctly, especially when the data has clutter.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central DIS-over-SIM cluster-quality advantage in §3.4 rests on significance tests whose experimental units are not established; the reported N=270 chi-squares and trial-level t-tests suggest non-independence, so the p-values may not survive a participant- and dataset-level re-analysis.","rationale":"The reader's CONDITIONAL verdict is reasonable. In good faith, the study is clearly reported and the reported effect is directionally consistent with prior work. The most load-bearing condition for the central claim is that the statistical comparison of SIM vs DIS under clutter is computed at the correct experimental unit and is not confounded by dataset difficulty. The paper's own reporting does not establish this: it lists Wilcoxon and paired t-tests without identifying pairs, and uses N=270 in chi-square statistics, which is the number of trials, not participants or independent blocks. That is an internal statistical concern, not a disagreement with the field consensus. The OSF materials are promised, so the concern is directly settleable. If a re-analysis with participant and dataset random effects reproduces the 2C DIS advantage, I would accept the claim; if not, the paper should be revised to report a more cautious conclusion. I therefore do not change the reader's verdict: it should remain conditional on that re-analysis. My concern differs from the reader's weakest assumption, which focused on the validity of the manually created ground truth; I view the DBSCAN verification as less central because the manual labels are the actual benchmark, and the randomization and dataset-difficulty confound is the more serious threat.","tokens_in":10281,"tokens_out":10211,"duration_ms":102026,"concrete_test":"Obtain the OSF artifacts (https://osf.io/bje89) and re-analyze the 2C cluster-quality data with (a) a participant-level paired test on each participant's mean cluster quality under DIS minus SIM (n=15), and (b) a linear mixed model: cluster_quality ~ ordering + (1|participant) + (1|dataset). If the DIS advantage remains significant under both, the central 'especially when clutter is present' claim survives; if either analysis shifts the effect to a wide interval containing zero, the reported p<.001 is an artifact of non-independence or dataset imbalance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 reports the decisive H2b result — DIS cluster-quality median 0.69 vs SIM 0.67 overall, and 0.58 vs 0.52 in the 2C condition (p<.001) — from Wilcoxon signed-rank and paired t-tests, but the manuscript never states the pairing unit. In the design, each of the 15 participants sees all 18 datasets once, with 3 SIM and 3 DIS trials per clutter level; the natural units are participant (or participant × cluster-count blocks), not individual glyph-grouping trials. The reported χ²(2, N=270) values for H1 treat 270 trials as independent observations, which inflates the effective sample size. If the SIM-vs-DIS comparisons are similarly trial-level, the central p-values are invalid because repeated measures are non-independent. Even if the tests are correctly paired within participant, using only six datasets per clutter level and ignoring dataset identity as a random factor leaves the DIS advantage vulnerable to one or two unusually easy or hard datasets. The DBSCAN ground-truth check in §3.2 is less load-bearing here: the manual PCDC labels are the operative ground truth, and a DBSCAN mismatch would not by itself explain an ordering effect. The concern is that the claimed DIS advantage under clutter has not yet been shown at the correct statistical unit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled user study (N = 15) comparing two star-glyph axis ordering strategies, similarity-based (SIM) and dissimilarity-based (DIS), for a visual grouping/noise-identification task. Eighteen synthetic 50-point datasets with 0, 1, or 2 clutter dimensions serve as ground truth, and each participant performs 18 trials (all datasets, with 3 SIM and 3 DIS trials per clutter level). The main claims are that (H1) clutter degrades cluster quality, noise identification, and confidence, and (H2) DIS ordering yields better cluster quality than SIM, especially under clutter, and better noise identification in the 2-clutter-dimension condition. The results section reports significant clutter effects on cluster quality and noise identification, a small but significant overall DIS advantage in cluster quality (median 0.69 vs 0.67, p < .05), and a larger DIS advantage in the 2C condition (0.58 vs 0.52, p < .001). Time and confidence effects are largely non-significant. The authors provide public study materials and analysis scripts.","tokens_in":10528,"tokens_out":3447,"duration_ms":37064,"significance":"If the central claim is statistically sound, this is a useful empirical contribution to a sparse literature: previous work (Klippel et al.) suggested that salient/spikey glyph shapes help grouping, but direct comparisons of data-driven ordering strategies with controlled ground truth were missing. The paper also provides practical design considerations and makes the study material, data, and scripts publicly available, which supports reproducibility. The strength of the contribution, however, rests almost entirely on the statistical analysis of the cluster-quality and noise-identification measures; the manuscript currently does not establish that the reported significance survives the repeated-measures structure of the experiment or the small number of datasets per condition.","major_comments":[{"comment":"The statistical tests appear to treat the 270 individual trials as independent observations, as indicated by the reported chi-square statistics with N = 270 and by the Wilcoxon/t-tests comparing SIM and DIS. In this design, each of the 15 participants contributes 18 trials, and each of the 18 datasets is seen by every participant, so trial-level observations are not independent. This inflates the effective sample size and may invalidate the reported p-values for the central DIS-over-SIM cluster-quality claim. The authors should state the pairing unit explicitly and re-analyze the data at the participant level (or with a mixed model including random intercepts for participant and dataset), reporting the effect sizes and confidence intervals for the overall comparison and for the 2C condition.","section":"Section 3.4 (H1b, H2b)"},{"comment":"The post-processing rule that converts groups containing one or two glyphs into noise is a hand-set threshold, and the manuscript provides no analysis of how this rule affects the SIM versus DIS comparison. Because the two-step Jaccard quality measure is sensitive to exact cluster membership, this rule can change which trials are scored as near-correct, and its effect may differ between orderings. The authors should justify the threshold a priori or include a sensitivity analysis showing that the reported DIS advantage is robust to alternative thresholds (e.g., only singletons, or groups of size up to three).","section":"Section 3.3 (Post-Processing and Analysis)"},{"comment":"The DBSCAN verification of the manually created PCDC clusters is reported without any numerical agreement measure. Since the DBSCAN parameters (minPts = 3, epsilon = 0.5) are hand-set, the reader cannot assess whether the ground-truth labels for all 18 datasets are consistent with the intended cluster structure. The authors should report an agreement statistic (e.g., adjusted Rand index or cluster purity) for each dataset, and state whether any dataset failed the verification and how such a failure would be handled.","section":"Section 3.2 (Benchmark Datasets)"}],"minor_comments":[{"comment":"Multiple SIM-versus-DIS comparisons are reported (overall and within 0C, 1C, 2C) without an adjustment for multiple testing; the authors should state whether the reported p-values are corrected or interpret the uncorrected comparisons as exploratory.","section":"Section 3.4 (H2b)"},{"comment":"Figure 3 shows the cluster-quality distributions but does not include effect sizes or confidence intervals, and the axes/legend are not described in the caption; adding these would help readers judge the practical size of the DIS advantage.","section":"Figure 3"},{"comment":"The trials are blocked by increasing difficulty (0C, then 1C, then 2C) for all participants, which may introduce order or fatigue effects that are confounded with clutter level; this design choice should be discussed as a limitation or mitigated in the analysis.","section":"Section 3.3 (Randomization)"},{"comment":"The manuscript contains minor typographical issues (e.g., 'execute a more coherent analysis', 'influenced', and inconsistent use of 'ε' vs 'eps' for the DBSCAN parameter); these do not affect the content but should be cleaned up.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is the first direct user-study comparison of data-driven similarity-based (SIM) and dissimilarity-based (DIS) axis ordering for star glyphs, using a controlled grouping task with synthetic datasets and three clutter levels. That alone makes it a useful contribution to the visualization literature. The materials and analysis scripts are on OSF, and the authors are explicit about their hypotheses and assumptions.\n\nThe main finding—that DIS ordering improves cluster quality, especially with two clutter dimensions (median 0.58 vs 0.52, p < .001)—is directionally consistent with Klippel et al.'s earlier work on salience. The effect is small but plausible. I also credit the authors for measuring time, confidence, and noise identification, not just cluster accuracy.\n\nThe soft spots are mostly statistical. The report repeatedly uses chi-square tests with N = 270 and t-tests with df = 44, treating each of the 270 trials as an independent observation. They are not: each of the 15 participants contributes 18 trials, and each of the 18 datasets is re-used across participants. The proper analysis would be a mixed-effects model with participant and dataset as random factors, or at least participant-level aggregation. The stress-test note is right that the central p-values may not survive that re-analysis. The effect might still be real, but the paper has not yet shown it at the correct statistical unit.\n\nA few smaller issues: the DBSCAN ground-truth verification is asserted without reporting any quantitative agreement, and the rule for converting one- and two-point groups into noise is hand-set. Neither is fatal, but the DBSCAN result should be reported. The authors themselves flag the small number of trials per condition as a limitation.\n\nOverall, this is a solid first step that needs a statistical overhaul before the headline claim is credible. Worth sending to peer review—the question is relevant and the study is reproducible—but a serious referee should require a reanalysis at the participant/dataset level before acceptance.\n\nBest.","headline":"Useful first comparison of star-glyph axis orderings, but the central DIS advantage rests on non-independent trial-level statistics.","tokens_in":10997,"tokens_out":3063,"would_cite":true,"duration_ms":29777,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports a user study showing that dissimilarity-based star-glyph ordering improves clustering accuracy over similarity-based ordering, with the largest advantage when the data includes clutter dimensions.","keywords":["star glyphs","axes ordering","dissimilarity ordering","similarity ordering","clustering tasks","clutter dimensions","empirical user study"],"falsifier":"A permutation test that reassigns the six two-clutter datasets between SIM and DIS and recomputes the median cluster-quality gap would settle whether the 0.58-versus-0.52 advantage is a property of the ordering strategy or of particular datasets; if the gap does not survive random reassignments, the claimed DIS advantage under clutter would not hold.","tokens_in":10054,"feed_emoji":"⭐","tokens_out":7356,"duration_ms":65318,"temperature":0.7,"pith_summary":"This paper asks whether the order of the axes around a star glyph changes how well people can group multidimensional data points into clusters. It compares similarity-based ordering, which places similar dimensions next to each other and produces homogeneous glyph shapes, with dissimilarity-based ordering, which places dissimilar dimensions next to each other and produces spiky, salient shapes. The study found that dissimilarity-based ordering improves clustering accuracy, and that the improvement is largest when the data includes dimensions that carry no cluster information. It also found that clutter substantially harms performance regardless of ordering. The paper thus offers an empirical basis for a design choice that had previously been argued mainly from perceptual principles.","feed_headline":"Spiky star glyphs beat smooth ones for spotting clusters","feed_subtitle":"On cluttered data, dissimilarity-ordered axes improve grouping accuracy, especially with two noise dimensions.","key_machinery":"The central object is the eight-axis star glyph, whose axes can be permuted. The paper defines two orderings: similarity-based (SIM), computed by exhaustive search for the permutation maximizing pairwise Euclidean similarity between adjacent dimensions, yielding homogeneous convex shapes; and dissimilarity-based (DIS), the same search minimizing adjacency similarity, yielding spiky salient shapes. The benchmark uses synthetic eight-dimensional datasets with two to four known clusters plus manually planted noise and clutter dimensions, validated by a density-based clustering check. User grouping quality is scored by a two-step Jaccard-index procedure that accounts for both user-selected groups and ground-truth clusters.","core_discovery":"The paper's central claim is that dissimilarity-based star-glyph layouts help users in visual clustering tasks more than similarity-based layouts, and that this advantage grows when the data contains clutter dimensions. Participants were significantly more accurate under the dissimilarity ordering than the similarity ordering overall, with median cluster quality 0.69 versus 0.67 (p < .05), and the gap was largest with two clutter dimensions: 0.58 versus 0.52 (p < .001). Noise identification also favored the dissimilarity ordering at that clutter level, while completion time and confidence showed no significant overall difference. These results are presented as direct support for the hypothesis that salient, spiky shapes aid grouping.","pith_inferences":["If the advantage comes from spike saliency, a natural next test is whether a few dominant spikes suffice or whether the full dissimilarity ordering is needed; the paper leaves the required number of spikes open.","The results suggest an adaptive rule: detect clutter dimensions automatically and switch from similarity to dissimilarity ordering only when clutter is present; this rule is testable against either fixed strategy.","The same ordering logic could be examined in other axes-based radial visualizations, such as RadViz or parallel coordinates, since their axis-ordering effects are task-dependent; the paper does not make this claim."],"forward_implications":["For datasets with clutter, glyph designers should prefer dissimilarity-based axis ordering to support clustering and noise identification.","For clutter-free datasets, similarity-based ordering is at least as accurate and trends faster and more confident, so it remains a reasonable default there.","Adding clutter dimensions substantially degrades cluster quality, noise identification, and user confidence, so suppressing or removing clutter dimensions before glyph display is advisable.","Users reported seeing clusters more clearly with spiky dissimilarity-ordered glyphs, matching the measured accuracy advantage even though confidence ratings did not differ significantly."],"supporting_citations":[{"why":"Supplies the prior result that salient spiky shapes support grouping and the study design that this experiment adapts.","marker":"[24, 25]"},{"why":"Supplies the similarity-ordering algorithm whose objective defines the SIM condition; the paper inverts it for the DIS condition.","marker":"[3]"},{"why":"The PCDC tool generates the synthetic benchmark datasets with known clusters and noise.","marker":"[8]"},{"why":"DBSCAN validates the manually created clusters, establishing the ground truth for scoring user groupings.","marker":"[13]"},{"why":"Taxonomizes star-glyph ordering strategies and motivates the SIM/DIS contrast and the need for empirical evaluation.","marker":"[35]"}],"fun_headline_variants":["Dissimilar axis order sharpens star glyph clustering","Star glyphs: spiky layouts improve cluster spotting","Clutter favors spiky star glyphs for grouping tasks","Dissimilarity ordering beats similarity in star glyphs","Spiky star glyphs aid clustering, especially with noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the handful of datasets within each clutter level are interchangeable between the SIM and DIS conditions, so the measured DIS advantage is not an artifact of easier datasets having been assigned to DIS.","fun_headline_variants_meta":{"raw":{"variants":["Dissimilar axis order sharpens star glyph clustering","Star glyphs: spiky layouts improve cluster spotting","Clutter favors spiky star glyphs for grouping tasks","Dissimilarity ordering beats similarity in star glyphs","Spiky star glyphs aid clustering, especially with noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1150,"prompt_tokens":820,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":253}},"tokens_in":436,"tokens_out":330,"duration_ms":3442,"temperature":1.0,"reasoning_tokens":253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:46:25.787884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A permutation test that reassigns the six two-clutter datasets between SIM and DIS and recomputes the median cluster-quality gap would settle whether the 0.58-versus-0.52 advantage is a property of the ordering strategy or of particular datasets; if the gap does not survive random reassignments, the claimed DIS advantage under clutter would not hold.","supporting_citations":[{"cited_title":"Ankerst, S","cited_arxiv_id":null,"evidence_quote":"Supplies the similarity-ordering algorithm whose objective defines the SIM condition; the paper inverts it for the DIS condition."},{"cited_title":"Bremm, M","cited_arxiv_id":null,"evidence_quote":"The PCDC tool generates the synthetic benchmark datasets with known clusters and noise."},{"cited_title":"Ester, H","cited_arxiv_id":null,"evidence_quote":"DBSCAN validates the manually created clusters, establishing the ground truth for scoring user groupings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Taxonomizes star-glyph ordering strategies and motivates the SIM/DIS contrast and the need for empirical evaluation."}],"review_version":1}