{"id":"e6587ae0-b342-4e8f-9778-a6b8e2431a9e","arxiv_id":"1908.00630","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pie charts were estimated more accurately than plain horizontal bar charts, and adding visual cues such as scales or decile lines improved bar chart accuracy.","lead":"This paper reports crowdsourced experiments comparing how accurately people estimate percentages from pie charts versus horizontal bar charts, with and without additional visual cues. The results suggest pie charts are at least as accurate as plain bars, and adding scales or cue lines to bars can improve accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central pie-superiority claim rests on unadjusted 95% CI non-overlap for non-independent repeated-measures data, with no raw data or formal test to verify it.","rationale":"The reader correctly identified the confidence-interval overlap procedure as the weakest assumption, and I agree that this is the key vulnerability. However, the more precise and severe issue is not merely that CI overlap is an informal significance test, but that the observations are non-independent: repeated responses from the same workers are treated as independent, which can artificially shrink confidence intervals. This directly threatens the central pie-versus-bar comparison, not only the secondary cue comparisons. In addition, the paper treats overlap as proof of no difference in Section 4.3, which is invalid, and Table 1 contains a numerical inconsistency (sum 1425 vs stated 1415) that raises questions about the reported data handling. Given the absence of raw data, code, or any formal statistical test, the central claim cannot be verified from the preprint. I therefore move the verdict from CONDITIONAL to UNVERDICTED: the direction of the finding may be correct, but the evidence as presented is insufficient to assess it, and the requested reanalysis would settle the concern.","tokens_in":5622,"tokens_out":4483,"duration_ms":50280,"concrete_test":"Release per-impression data (or per-cell means, standard deviations, and per-worker counts) and re-estimate the baseline pie versus baseline bar comparison with a mixed-effects model that includes a random intercept for participant and fixed effects for chart type and target value; compute the 95% confidence interval for the pie-minus-bar difference using cluster-robust or by-participant bootstrap resampling. If the interval includes zero, the Section 4.1 'significant' claim and the Section 5 recommendation lose their statistical basis.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's central recommendation (Section 5) is that pie charts are at least as accurate as bar charts and that the data show 'better accuracy' for pies. The sole statistical basis for this is Section 4.1: baseline pie mean absolute error 1.7665 vs baseline bar 2.3458, with the statement that 'the lack of overlap indicates a significant difference' per Cumming [18]. Three problems make this inference load-bearing and unverified. First, the paper does not report standard errors, confidence interval widths, or the method used to compute the intervals; only the means and the overlap claim are given. Second, the responses are not independent observations: 316 workers produced 1,415 impressions, with one worker seeing up to 25 charts. Per-impression confidence intervals ignore this clustering, so they are likely too narrow; a mixed-effects analysis that accounts for participant-level variation could easily widen the intervals enough to overlap. Third, the paper uses overlap of 95% CIs both as evidence of significance and as evidence of non-significance. Non-overlap is a sufficient but not necessary condition for a significant difference at the 0.05 level; overlap does not imply the absence of a difference. Section 4.3 explicitly concludes that the quartile-cue hypothesis 'can be rejected' solely because intervals overlap, which is not a valid equivalence test. There is also an internal inconsistency: the row sums in Table 1 total 1,425 rather than the stated 1,415 valid responses. Without raw data, formal tests, and cluster-aware intervals, the central statistical claim cannot be independently confirmed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a crowdsourced experiment (316 Amazon Mechanical Turk workers, 1,415 responses after Tukey-fence outlier removal) in which participants estimate the percentage of a highlighted segment in pie charts and horizontal stacked-bar charts, with variants adding quartile lines, decile lines, an external scale, and quartile ticks on pies. The author compares mean absolute error (MAE) across conditions, concludes that baseline pie charts yield lower MAE than baseline bar charts, that decile cues on bars and an external scale improve accuracy, and that the study does not substantiate the claim that bar charts are preferable to pies. The conclusion advises visualization practitioners that using a pie chart for part-to-whole comparison is not an error.","tokens_in":5801,"tokens_out":5800,"duration_ms":57716,"significance":"If validated, the result would provide a useful replication of earlier empirical findings (Eells; Spence and Lewandowsky; Kosara) using modern crowdsourced data and would inform practical recommendations for visualization design. The paper's strengths are its simple, concrete estimation task, its use of mean absolute error, and its transparent description of the outlier-removal procedure. However, the central claims are not currently supported by the reported statistical analysis: inference is based on visual overlap of unreported confidence intervals computed on non-independent observations, and the paper contains an internal count inconsistency. The conclusions therefore cannot be accepted as stated without a re-analysis of the data.","major_comments":[{"comment":"The central significance claims—that the pie chart performed better than the bar chart and that decile or scale cues significantly reduce error—rest entirely on visual comparisons of 95% confidence intervals, but the manuscript reports neither the interval widths, standard errors, nor the method used to compute them. Moreover, the observations are not independent: 316 workers produced 1,415 impressions, with some workers seeing up to 25 charts. Per-impression confidence intervals ignore this clustering and are likely too narrow, so the reported lack of overlap cannot be taken as evidence of a reliable difference. The author should report a mixed-effects model or a participant-level analysis with formal tests and effect sizes, and provide the raw data or sufficient summary statistics for verification.","section":"Section 4.1 and Figures 3, 5, 7, 9"},{"comment":"The paper uses confidence-interval overlap both as evidence of significance and as evidence of no effect. In Section 4.3, the conclusion that the quartile-cue hypothesis 'can be rejected' is based solely on overlapping intervals, and in Section 4.4 a similar interval comparison is used to claim that 'the hypothesis cannot be rejected.' Non-overlap of 95% confidence intervals is a sufficient but not necessary condition for a significant difference at the 0.05 level; overlap does not imply equivalence or absence of a difference. Rejection of the hypothesis of no improvement requires an equivalence test, a non-inferiority test, or a properly powered null result with reported confidence bounds.","section":"Sections 4.3 and 4.4"},{"comment":"The row counts in Table 1 sum to 1,425, not the 1,415 'valid responses' stated in Section 3 and in the table caption. This discrepancy must be resolved before any conclusions can be checked, because the reported means and confidence intervals depend on the exact per-cell counts. The author should correct the counts or the text and explain which number was actually used in the analysis.","section":"Table 1 and Section 3"},{"comment":"The study tests four hypotheses and makes multiple pairwise comparisons across chart types and cue conditions without any adjustment for multiple comparisons or a full account of all conducted tests. Given the number of comparisons shown in Figures 5 and 9, the probability of at least one false positive is inflated. The author should apply a suitable correction (for example, Tukey HSD, Bonferroni, or pre-specified contrasts) or explicitly state that these comparisons are exploratory.","section":"Sections 4.1-4.4"}],"minor_comments":[{"comment":"The caption refers to 'baseline bar chart and baseline pie chart,' but the text then discusses the bar with scale; the caption should be rewritten to describe the conditions actually plotted.","section":"Figure 9 caption"},{"comment":"The name 'Cummings' should be 'Cumming,' matching reference [18].","section":"Figure 3 caption"},{"comment":"There are typographical errors: 'platted' should be 'plotted,' and 'the different is not significant' should be 'the difference is not significant.'","section":"Section 4.3"},{"comment":"The sentence 'The significance of the difference in mean absolute error that the hypothesis cannot be rejected' is grammatically incomplete; please rephrase it.","section":"Section 4.4"},{"comment":"The author's name is spelled 'Spence,' not 'Spense,' in references [7] and [10].","section":"References [7] and [10]"},{"comment":"The manuscript does not include a data availability statement; sharing the de-identified response data would support the re-analysis required by the major comments.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central concern is statistical: the reported inferences are not valid for this clustered design, and the numerical discrepancy in Table 1 needs resolution. The paper's conclusion about pie charts is plausible and worth publishing if the author can reanalyze the data with appropriate mixed-effects or participant-level methods and report the missing summary statistics. If the raw data are no longer available, the manuscript would not be salvageable in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Stephen's paper is a small crowdsourced experiment that lands on the right side of a long-running debate: for simple part-to-whole estimates, pie charts are not the disaster some pundits claim. The new part is not the pie-vs-bar baseline—that replicates Eells 1926 and later work—but the systematic look at visual cues: quartile and decile ticks on bars, quartile ticks on pies, and an external quantitative scale. That is a useful contribution, and the design is straightforward: five target values, six chart types, MTurk participants, MAE with Tukey-fence outlier removal.\n\nThe qualitative pattern is believable and consistent with prior results. But the statistical backing is thinner than the text suggests. The paper leans on 95% CI non-overlap to claim significance and on overlap to declare 'no significant difference.' Non-overlap is a sufficient condition for a difference at 0.05 for independent means, but overlap is not evidence of equivalence; the paper uses it that way in Sections 4.2 and 4.3. There are also no formal tests, no multiple-comparison correction, and no account of the fact that 316 workers produced 1,415 impressions—up to 25 per worker. The CIs are drawn as if each impression were independent, which likely overstates precision.\n\nThere's also a concrete inconsistency: the row sums in Table 1 total 1,425, while the text says 1,415 valid responses. That's ten observations unaccounted for, and it undermines confidence in the reported counts. No raw data or analysis scripts are provided, so the key numbers can't be checked.\n\nThese flaws are real but not fatal to the main direction. The pie-superiority result replicates decades of empirical work, and the cue effects are plausible. What the paper cannot support is the rhetorically strong conclusion that practitioners 'should not feel that they are making an error' with pie charts. That generalization goes beyond what one MTurk study with this analysis can bear.\n\nI'd send this to peer review, but with a strong request for revision: reanalyze with mixed-effects models, run proper pairwise tests with corrections, report CI widths, publish raw data and code, and fix the table. The topic is worth refereeing; the current analysis is not ready as-is.","headline":"A modest, plausible MTurk study that extends the pie-vs-bar debate with visual cues, but the statistical support is weaker than the paper claims and a table count doesn't add up.","tokens_in":6392,"tokens_out":3311,"would_cite":false,"duration_ms":32016,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that for part-to-whole comparisons, baseline pie charts yield lower mean absolute estimation error than horizontal bar charts, and that adding decile cues to bars improves accuracy, concluding that data visualization…","keywords":["pie charts","bar charts","part-to-whole comparison","visual cues","perceptual anchors","mean absolute error","crowdsourcing","Mechanical Turk"],"falsifier":"Re-analyze the paper's mean absolute error data with a proper inferential test—a two-sample t-test or permutation test on the per-impression errors for baseline pie versus baseline bar. If the resulting p-value is not below 0.05, the central claim that pies are significantly better than bars collapses. Also, check whether the confidence intervals were computed on per-task means rather than per-impression responses; using the wrong unit of analysis would invalidate the non-overlap claim.","tokens_in":5349,"feed_emoji":"🥧","tokens_out":4290,"duration_ms":39392,"temperature":0.7,"pith_summary":"The paper tries to settle a century-old dispute: are pie charts actually worse than bars for reading part-to-whole proportions? Through crowdsourced estimation tasks, it finds that a plain pie chart produces smaller mean absolute errors than a plain horizontal bar chart, with non-overlapping 95% confidence intervals. Adding visual cues to bar charts, especially decile lines or a quantitative scale, lowers error further, while adding quarter cues to pies does not help. The author concludes that using a pie chart for part-to-whole comparisons is defensible, and that natural perceptual anchors in pies explain their performance.","feed_headline":"Pie charts beat bar charts for part-to-whole estimates","feed_subtitle":"A 1,415-response crowdsourced experiment finds lower error for pies; decile cues help bars.","key_machinery":"The central mechanism is the perceptual anchor: the angles 0°, 90°, and 180° in a pie provide natural reference points that participants use when judging segment size, giving pies an advantage over bars, which only offer start and end anchors. The experiments operationalize this by comparing baseline charts with variants that add visual cues—quartile lines, decile lines, and an external quantitative scale—and measuring mean absolute error across 1,415 valid responses after Tukey-fence outlier removal. The load-bearing identity is the claimed significant difference established by non-overlapping 95% confidence intervals, following Cumming.","core_discovery":"On its own terms, the paper's central discovery is that the baseline pie chart outperforms the baseline horizontal bar chart for estimating segment size in part-to-whole charts: mean absolute error 1.7665 for pie versus 2.3458 for bar, with the paper treating non-overlap of 95% confidence intervals as significant. A secondary result is that decile cues on bars and a quantitative scale on bars both improve estimation, while quartile cues on pies do not significantly change performance. The author states that these findings replicate Eells (1926) and support the position that pie charts have naturally occurring perceptual anchors at 0%, 25%, 50%, 75%, and 100%.","pith_inferences":["The paper's significance claim rests solely on non-overlap of confidence intervals; a reanalysis with formal hypothesis tests (permutation or t-tests) could either support or overturn the pie-over-bar conclusion, so the practical recommendation should be treated as provisional.","If natural anchors are the true explanation, then pies with cues at non-standard angles (e.g., every 30°) should improve accuracy, a testable prediction the paper does not run.","Because the participant pool was Amazon Mechanical Turk workers with no cohort selection, the results may not transfer to expert analysts or high-stakes settings; a replication with domain experts would test that boundary.","The relative advantage of pies may shrink when the target segment is far from the natural anchors—e.g., near 33% or 44%—and a per-value breakdown could reveal where bars and pies cross over."],"forward_implications":["Pie charts are not a mistake for part-to-whole comparisons: the baseline pie beat the baseline bar in this study.","Bar charts improve when given more granular internal anchors: decile cues significantly reduced error over baseline.","A quantitative scale is the best cue for bars, but when a scale is inappropriate, decile lines are a viable alternative.","Adding quartile cues to pies does not significantly help, consistent with pies already carrying quarter anchors.","Data visualization professionals can adopt pies or bars-with-cues without violating accuracy goals."],"supporting_citations":[{"why":"Supplies the original empirical finding being replicated: circles judged more accurately than bars for component parts.","marker":"[6]"},{"why":"Provides the perceptual-anchor theory the paper tests, that position and angle in pies offer superior anchors to simple length.","marker":"[9]"},{"why":"Prior study showing pies as good as or superior to stacked bars, with discussion of imaginary quarters and halves.","marker":"[7]"},{"why":"Recent evidence that pie and donut variants are as effective as alternatives, motivating the research question.","marker":"[8]"},{"why":"Describes the natural anchors of a pie at 0%, 25%, 50%, 75% and 100% compared with a bar's fewer anchors.","marker":"[10]"},{"why":"Establishes the Mechanical Turk crowdsourcing protocol for graphical perception experiments that this study employs.","marker":"[14]"},{"why":"Supplies the Tukey fences method used to remove outliers from the response data.","marker":"[15]"},{"why":"Provides the New Statistics approach of using 95% confidence interval non-overlap to claim significant differences.","marker":"[18]"}],"fun_headline_variants":["Pie charts edge out bars in part-to-whole estimation","Decile cues on bars improve part-to-whole estimates","Crowdsourced test: pie charts beat bars for segment share","Pies outperform bars in part-to-whole visual comparison","Pie chart advantage confirmed in large crowdsourced study"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that non-overlap of 95% confidence intervals is a valid significance test; if that assumption fails, the claimed differences between pie and bar charts, and the benefit of decile cues and scales, are not statistically established.","fun_headline_variants_meta":{"raw":{"variants":["Pie charts edge out bars in part-to-whole estimation","Decile cues on bars improve part-to-whole estimates","Crowdsourced test: pie charts beat bars for segment share","Pies outperform bars in part-to-whole visual comparison","Pie chart advantage confirmed in large crowdsourced study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001033,"raw_usage":{"total_tokens":4274,"prompt_tokens":792,"completion_tokens":3482,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":3399}},"tokens_in":408,"tokens_out":3482,"duration_ms":25626,"temperature":1.0,"reasoning_tokens":3399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:41:20.406357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-analyze the paper's mean absolute error data with a proper inferential test—a two-sample t-test or permutation test on the per-impression errors for baseline pie versus baseline bar. If the resulting p-value is not below 0.05, the central claim that pies are significantly better than bars collapses. Also, check whether the confidence intervals were computed on per-task means rather than per-impression responses; using the wrong unit of analysis would invalidate the non-overlap claim.","supporting_citations":[{"cited_title":"The Relative Merits of Circles and Bars for Representing Component Parts,","cited_arxiv_id":null,"evidence_quote":"Supplies the original empirical finding being replicated: circles judged more accurately than bars for component parts."},{"cited_title":"An Information -Processing Analysis of Graph Perception,","cited_arxiv_id":null,"evidence_quote":"Provides the perceptual-anchor theory the paper tests, that position and angle in pies offer superior anchors to simple length."},{"cited_title":"Displaying propo rtions and percentages,","cited_arxiv_id":null,"evidence_quote":"Prior study showing pies as good as or superior to stacked bars, with discussion of imaginary quarters and halves."},{"cited_title":"Arcs, Angles, or Areas: Individual Data Encodings in Pie and Donut Charts,","cited_arxiv_id":null,"evidence_quote":"Recent evidence that pie and donut variants are as effective as alternatives, motivating the research question."},{"cited_title":"No humble pie: The origins and usage of a statistical chart,","cited_arxiv_id":null,"evidence_quote":"Describes the natural anchors of a pie at 0%, 25%, 50%, 75% and 100% compared with a bar's fewer anchors."},{"cited_title":"Crowdsourcing graphical perception: using mechanical turk to assess visualization design,","cited_arxiv_id":null,"evidence_quote":"Establishes the Mechanical Turk crowdsourcing protocol for graphical perception experiments that this study employs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Tukey fences method used to remove outliers from the response data."},{"cited_title":"The New Statistics: Why and How,","cited_arxiv_id":null,"evidence_quote":"Provides the New Statistics approach of using 95% confidence interval non-overlap to claim significant differences."}],"review_version":1}