{"id":"a707c735-4aed-4110-ba75-5470a116ed45","arxiv_id":"1908.01697","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Chart makers rarely draw uncertainty even when they believe it matters; the paper explains this as a norm sustained by a signal mindset and a formal argument about viewer inference.","lead":"A survey of 90 visualization creators and interviews with 13 influential designers show that most authors believe uncertainty should be shown but rarely draw it. The paper gives a rhetorical model of why that happens and argues that showing uncertainty narrows how differently viewers can read a chart.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'necessarily reduces degrees of freedom' conclusion in §6.1 is not established; uncertainty can itself introduce new inferential degrees of freedom because viewers interpret interval encodings differently.","rationale":"The central empirical contribution—surveying and interviewing visualization authors and organizing their rationales into a rhetorical model—is valuable and well supported by the quoted responses. The weakest part of the paper is indeed the formal claim in §6, as the reader noted. My concern is more specific than the reader's: even if viewers do perform implicit posterior predictive model checks, the claim that uncertainty visualization 'necessarily' reduces the degrees of freedom in their inferences is not established, because uncertainty encodings are themselves ambiguous and can be mapped to different predictive models by different viewers. This is not an external objection; the paper concedes the interpretability problem and cites evidence of error-bar misinterpretation. The issue is therefore internal to the formal argument: the author moves from 'visualizing uncertainty conveys more information about potential reference distributions' to 'necessarily reduce variance in interpretations,' but conveying more information does not guarantee less ambiguity unless a shared interpretation mapping is assumed. The paper would be strengthened by defining a measure over the model space and by stating the condition under which the reduction holds, rather than asserting it universally. Because the qualitative findings and the rhetorical model do not depend on this formal necessity claim, and because the paper already frames many conclusions cautiously, the conditional verdict is appropriate. No change to the reader's verdict is needed; the formal claim should be reframed as a testable hypothesis rather than a proven theorem.","tokens_in":18532,"tokens_out":2955,"duration_ms":37078,"concrete_test":"Run a controlled study in which participants view the same line or point chart in three conditions: no uncertainty, an error bar or 95% CI, and a density/violin uncertainty mark. After each display, elicit the participant's inferred data-generating process by having them describe or select plausible models, or by fitting a Bayesian cognitive model to their parameter judgments. Compute the number or entropy of distinct model specifications compatible with responses in each condition. If the uncertainty conditions do not yield strictly lower entropy than the no-uncertainty condition, the 'necessarily reduces degrees of freedom' claim is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The formal core of §6 is that without uncertainty the viewer's implicit reference/predictive distribution is drawn from a 'seemingly infinite' space, and that uncertainty 'necessarily constrains the set of possible models.' This is presented as an adaptation of graphical inference, but no operational definition of 'degrees of freedom' or of the model space is given, and the argument is carried by illustrative examples (Fig. 4b–e). The necessity claim rests on a hidden monotonicity assumption: adding an uncertainty encoding always removes more model ambiguity than it introduces. That assumption is doubtful. The paper itself cites evidence [3, 28] that viewers systematically misinterpret confidence intervals and error bars; if a viewer reads an interval as a hard bound, a prediction range, a confidence interval, or a deliberately imprecise mark, the mapping from the display to p(yrep|y) varies at least as much as the mapping from a bare point. Section 6.1.2 concedes that an uncertainty visualization does not ensure that viewers infer the author's intended reference distribution, and that some audiences cannot reliably infer modeling assumptions from standard uncertainty displays. Thus the formal conclusion should be 'can reduce degrees of freedom under a specified interpretation model,' not 'necessarily reduces.' Without an explicit measure of the model space, the normative conclusion that omission is logically indefensible does not follow from the supplied examples.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates why visualization authors omit uncertainty from their graphics. The author reports a survey of 90 visualization authors recruited via Twitter and interviews with 13 influential visualization practitioners, documenting a tension between authors' professed belief that uncertainty matters and their frequent omission of it. The paper contributes a three-tenet rhetorical model of uncertainty omission (visualization as signal; process validates signal; uncertainty obfuscates signal), and then adapts Gelman's graphical-inference framework to argue that uncertainty communication 'necessarily reduces degrees of freedom in viewers' statistical inferences.' It closes with recommendations for future tools and evaluations.","tokens_in":18715,"tokens_out":5193,"duration_ms":55160,"significance":"The empirical contribution is valuable: it is one of the few direct, systematic elicitations of practitioner rationales for uncertainty omission, and the paper is transparent about its recruitment, coding process, and sample limitations. The rhetorical model is a plausible and useful synthesis that can orient future HCI and visualization research. The formal model is a thought-provoking adaptation of existing statistical-inference ideas rather than a new derivation, and it is presented through illustrative examples. The main weakness is that the central formal claim is stated more strongly than the supporting argument and the manuscript's own caveats allow. Rephrasing the claim as 'can reduce degrees of freedom under a specified interpretation model' would make the paper defensible; the qualitative and rhetorical contributions do not need to change.","major_comments":[{"comment":"The claim that uncertainty visualization 'necessarily reduces degrees of freedom' in viewers' inferences is not established by the formalism presented. No operational definition is given for 'degrees of freedom' or for the viewer's model space, and the argument is carried by constructed examples rather than a derivation. More importantly, the necessity claim relies on a hidden monotonicity assumption: that adding an uncertainty encoding always removes more interpretive ambiguity than it introduces. This assumption is doubtful given the paper's own citations [3,28], which show that viewers systematically misinterpret confidence intervals and error bars. Section 6.1.2 concedes that an uncertainty visualization does not ensure that viewers infer the author's intended reference distribution and that some audiences cannot reliably infer modeling assumptions from standard uncertainty displays. The logically supported conclusion is thus that uncertainty can reduce degrees of freedom under a specified interpretation model, not that it necessarily does. Because Section 6 presents this formalism as providing 'logical ground' against omission, this is a load-bearing issue.","section":"Section 6.1, Figure 4"},{"comment":"The abstract states that uncertainty communication 'necessarily reduces degrees of freedom in viewers' statistical inferences,' but the Conclusion says that uncertainty 'reduces (though does not necessarily eliminate) degrees of freedom in viewers' inferences.' These are different claims, and the manuscript should state one consistent claim. The strong 'necessarily' version is what drives the argument against omission, while the weaker version is what the analysis in Section 6.1.2 actually supports. The discrepancy should be resolved by tempering the abstract and Section 6's stronger formulations.","section":"Abstract vs. Section 8"},{"comment":"The paper acknowledges in Section 3.1.2 and Section 7.0.1 that the survey and interview samples are convenience samples that likely overrepresent authors sympathetic to uncertainty visualization. This acknowledged limitation should be more carefully reflected in the scope of the rhetorical model. The three tenets in Section 5 are inferred from this sample, and the premise in Section 5.1 that uncertainty omission is a norm is supported partly by self-reported estimates from the same respondents. The empirical characterization is still a useful contribution, but the paper should frame the rhetorical model as a hypothesis about a specific population of social-media-recruited practitioners rather than as a general account of 'authors.'","section":"Sections 3.1.2 and 7.0.1"},{"comment":"The formal argument assumes that viewers judge visualized signal strength through an implicit posterior predictive model check, with a reference distribution p(yrep|y) drawn from a 'seemingly infinite' space in the absence of uncertainty. This assumption is asserted and illustrated, not tested empirically or derived from first principles. If viewers do not perform such model comparisons, or if their implicit model space is narrow even without uncertainty, the formal conclusion does not follow. The paper could strengthen the argument by providing at least a minimal formal characterization of the relevant model space and the interpretation mapping from an uncertainty display to p(yrep|y).","section":"Section 6.1"},{"comment":"The sentence 'This reduces the amount of information that a viewer must mentally fill in, which would seem to necessarily reduce variance in interpretations across viewers' is a hedged intuition, not a proof. The hedge 'would seem to' indicates that the author is aware that the step is not fully demonstrated. The paper should either provide a concrete argument for why adding uncertainty information cannot introduce additional interpretive variance, or explicitly scope the claim to situations where viewers share the intended interpretation of the uncertainty encoding.","section":"Section 6.1.2"}],"minor_comments":[{"comment":"'In gain a deeper understanding' should be 'To gain a deeper understanding.'","section":"Section 3.2"},{"comment":"In the phrase 'T (rrep ) is drawn from a null plot distribution,' the symbol 'rrep' appears to be a typo for 'yrep.'","section":"Section 6.1"},{"comment":"'in is unlikely that all audiences could reliably infer' should be 'it is unlikely that all audiences could reliably infer.'","section":"Section 6.1.2"},{"comment":"'a formal model of of graphical statistical inference' contains a duplicated 'of.'","section":"Section 8"},{"comment":"'permissable' should be 'permissible.'","section":"Section 6.1.2"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major-revision case rather than a reject. The empirical and rhetorical contributions are sound, and the formal claim can be repaired by weakening the necessity language and clearly scoping the argument. The main risk is that the abstract overpromises a result the paper's own sections concede is not fully established. No concerns about citation integrity or novelty disclosure beyond the overstrong formal claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. This paper provides the first direct survey and interview evidence I know of on why visualization authors omit uncertainty, and that part is a real contribution. But the formal claim in the abstract—that uncertainty communication 'necessarily reduces degrees of freedom' in viewers' statistical inferences—is oversold; Section 6 demonstrates less than it asserts, and the body's own qualifiers contradict the abstract.\n\nThe empirical core is solid. A convenience sample of 90 authors plus 13 influential designers is not representative, and the author acknowledges the likely overrepresentation of uncertainty-sympathetic respondents. Even so, the results capture a clear contradiction: most authors say uncertainty should be shown more, yet nearly half admit to having left it out in the last year, mostly out of fear of confusing viewers, lack of access to uncertainty information, or not knowing how to compute it. The rhetorical model—visualization as signal, process as validator, uncertainty as obfuscating seam—is a useful synthesis that explains how the norm persists despite good intentions. The paper is honest about its limitations, and the qualitative analysis is carefully done.\n\nThe soft spot is the formal argument in Section 6. The adaptation of Gelman's graphical inference framework is suggestive, but 'necessarily reduces degrees of freedom' is under-derived. There is no operational definition of degrees of freedom or of the viewer's model space, and the argument rests on illustrative examples plus a hidden monotonicity assumption: that adding an uncertainty encoding always removes more inferential ambiguity than it introduces. That is doubtful, and the paper itself cites evidence that viewers systematically misinterpret confidence intervals and error bars. An interval can be read as a hard bound, a prediction range, a confidence interval, or deliberate imprecision; the mapping from display to reference distribution is not obviously less variable than from a bare point. The conclusion eventually hedges to 'reduces (though does not necessarily eliminate),' but the abstract and Section 6 opening still assert necessity. The claim should be reframed as a testable hypothesis.\n\nWho should read this? People working on uncertainty visualization, visualization rhetoric, or HCI studies of author practice. The empirical findings and the rhetorical model deserve engagement; the formal part is a useful starting point once cut down to size. The paper deserves a serious referee, not a desk reject—with a request to tighten the formal claims, either by proving the monotonicity assumption or by stating the weaker variance-reduction hypothesis, and to share the survey instrument and anonymized data if possible.","headline":"Empirically valuable study of why authors omit uncertainty, but the formal 'necessarily reduces degrees of freedom' claim is oversold and should be reframed as a testable hypothesis.","tokens_in":19236,"tokens_out":3935,"would_cite":true,"duration_ms":38112,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Omitting uncertainty from a chart is not neutral: viewers judge charts against imagined models, so without uncertainty the model is almost unspecified and showing uncertainty necessarily narrows interpretations.","keywords":["uncertainty visualization","visualization rhetoric","graphical statistical inference","posterior predictive model check","communicative visualization","visualization authors","data journalism","uncertainty omission"],"falsifier":"Take a single chart designed to support an inference and render two versions, one without uncertainty and one with explicit intervals or a distribution, then measure the spread of viewers' conclusions or perceived signal strength across a large sample. The paper's claim predicts that the no-uncertainty version will produce a wider spread of inferences; observing no reduction when uncertainty is added would falsify the central claim. A more targeted version would ask viewers to identify which reference distribution the chart came from and test whether uncertainty annotations improve agreement.","tokens_in":18285,"feed_emoji":"📊","tokens_out":10432,"duration_ms":101009,"temperature":0.7,"pith_summary":"The paper asks why visualization authors so often omit uncertainty from charts meant to inform the public, and it argues that the omission is not a neutral simplification. Drawing on a survey of 90 visualization authors and interviews with 13 working designers and journalists, it documents a contradiction: most authors say uncertainty matters and should appear more often, yet a large share omit it, citing fear of confusing viewers, lack of data or skill, and concern that uncertainty would make results seem questionable. The paper's central theoretical claim is that a viewer judges a chart by comparing it to an imagined reference distribution, so when uncertainty is absent that reference distribution is left almost entirely open; showing uncertainty constrains it and thereby reduces the degrees of freedom in viewers' inferences. The accompanying rhetorical model explains how the omission persists: authors treat the chart as a pure signal, trust their analytical process to validate that signal, and treat uncertainty as a seam or question that threatens the message.","feed_headline":"Omit uncertainty and a chart can support nearly any reading","feed_subtitle":"Omission lets viewers imagine the statistics differently; the paper says uncertainty narrows that freedom.","key_machinery":"The central mechanism is the implicit posterior predictive model check, a formal analogy between visual inspection and statistical model checking. A viewer of a chart is modeled as constructing an imagined distribution of replicated data $T(y_{\\mathrm{rep}})$ from a model that depends on the observed data, the viewer's priors, and the viewer's assumptions about variance and comparison cases, and then judging how discrepant the displayed data are from that distribution. The argument uses this mechanism to convert the rhetorical observation that authors treat visualization as \"signal\" into a precise statement: with no uncertainty shown, the viewer's model specification is underdetermined, so uncertainty representation narrows the space of models a viewer can plausibly entertain. Constructed examples in the paper—a two-period two-party attitude chart and a multi-country debt-to-GDP line chart—show how varying the assumed variance or the comparison data changes the conclusion a viewer would draw.","core_discovery":"The paper's claim is that uncertainty representation necessarily reduces the degrees of freedom in a viewer's statistical inferences, stated formally as \"uncertainty representation necessarily constrains the set of possible models.\" In the proposed account, looking at a chart is an implicit posterior predictive model check: a viewer mentally generates a reference distribution $T(y_{\\mathrm{rep}})$ from a model that is fit to the observed data and their prior beliefs, then compares the chart to that distribution to decide whether the signal is real. Without uncertainty, the viewer must choose variance, the data used to fit the model, the comparison set, and the model structure on their own, so different viewers can reach different conclusions from the same chart. Visualized uncertainty supplies information about the reference distribution the author had in mind, moving some of that interpretive burden from the viewer to the author; the paper grants that uncertainty does not eliminate ambiguity, because viewers can still misinterpret intervals or infer a different model than intended.","pith_inferences":["An extension the paper does not develop: if omission leaves the reference distribution underdetermined, then the persuasive effect of a chart may depend as much on the priors viewers bring as on the data shown; tests could compare conclusions across viewers with different domain priors on the same no-uncertainty chart.","Another testable extension is whether the degrees-of-freedom argument changes author behavior: researchers could show authors the spread of interpretations their own charts produce with and without uncertainty and measure whether they then choose to include uncertainty.","The paper gestures toward decision-theoretic reasoning in its discussion; a fuller formal treatment would model the worst-case decision a viewer could make under omission versus inclusion, which would make the cost of omission concrete in specific policy or consumer settings."],"forward_implications":["In a chart that omits uncertainty, viewers must fill in the missing statistical assumptions themselves; different viewers may therefore walk away with different conclusions from the same graphic.","Showing uncertainty narrows the set of interpretations but does not make interpretation unique: intervals can be misread as confidence limits or as data ranges, so authors still need to explain the reference model behind the uncertainty.","The rhetorical model implies that better uncertainty-encoding techniques alone will not change practice; authors' beliefs about signal, process, and credibility also have to shift.","Because the formal argument is logical rather than ethical, it offers a way to argue for uncertainty communication that does not depend on claims about moral duty.","Tools that help authors calculate, explain, and visualize uncertainty, and that let authors express the reference distribution behind their own analysis, are a direct practical consequence."],"supporting_citations":[{"why":"Supplies the formal account of viewing a plot as a model check against a posterior predictive reference distribution, the mechanism the paper adapts to communicative visualization.","marker":"[24, 25]"},{"why":"Earlier account of a viewing pipeline for data analysis that frames visual examination as statistical inference, giving the paper precedent for the model-check analogy.","marker":"[6]"},{"why":"Defines the null plot distribution used in visual hypothesis tests, which the paper uses to specify the reference distribution a viewer compares against.","marker":"[7]"},{"why":"Brings graphical inference to information visualization and supplies the bridge from analyst-facing visual inference to communication settings.","marker":"[54]"},{"why":"Defines the posterior predictive distribution $p(y_{\\mathrm{rep}}\\mid y)$, which the paper assumes viewers approximate when judging signal strength.","marker":"[23]"},{"why":"Defines visualization rhetoric as framing, the perspective the paper extends into its three-tenet rhetorical model of uncertainty omission.","marker":"[29]"},{"why":"Documents that researchers misinterpret confidence intervals and standard error bars, supporting the paper's caveat that visualized uncertainty narrows rather than removes ambiguity.","marker":"[3]"}],"fun_headline_variants":["Uncertainty omission lets every viewer see their own chart","Why charts skip uncertainty: it pins down the reading","Without error bars, viewers write their own statistics","Visualized uncertainty narrows the viewer's interpretive freedom","Authors omit uncertainty to keep charts flexible"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that viewers actually judge charts by doing something like an implicit model check—mentally comparing the chart to an imagined set of replications—and that without uncertainty the space of models they can imagine is wide enough that different viewers will genuinely diverge; if real viewers do not run such comparisons, or if their mental model space stays narrow even without uncertainty, the conclusion that uncertainty necessarily reduces degrees of freedom does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty omission lets every viewer see their own chart","Why charts skip uncertainty: it pins down the reading","Without error bars, viewers write their own statistics","Visualized uncertainty narrows the viewer's interpretive freedom","Authors omit uncertainty to keep charts flexible"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1798,"prompt_tokens":931,"completion_tokens":867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":794}},"tokens_in":547,"tokens_out":867,"duration_ms":7233,"temperature":1.0,"reasoning_tokens":794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:05:31.971773+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a single chart designed to support an inference and render two versions, one without uncertainty and one with explicit intervals or a distribution, then measure the spread of viewers' conclusions or perceived signal strength across a large sample. The paper's claim predicts that the no-uncertainty version will produce a wider spread of inferences; observing no reduction when uncertainty is added would falsify the central claim. A more targeted version would ask viewers to identify which reference distribution the chart came from and test whether uncertainty annotations improve agreement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier account of a viewing pipeline for data analysis that frames visual examination as statistical inference, giving the paper precedent for the model-check analogy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the null plot distribution used in visual hypothesis tests, which the paper uses to specify the reference distribution a viewer compares against."},{"cited_title":"Wickham, D","cited_arxiv_id":null,"evidence_quote":"Brings graphical inference to information visualization and supplies the bridge from analyst-facing visual inference to communication settings."},{"cited_title":"Gabry, D","cited_arxiv_id":null,"evidence_quote":"Defines the posterior predictive distribution $p(y_{\\mathrm{rep}}\\mid y)$, which the paper assumes viewers approximate when judging signal strength."},{"cited_title":"Hullman and N","cited_arxiv_id":null,"evidence_quote":"Defines visualization rhetoric as framing, the perspective the paper extends into its three-tenet rhetorical model of uncertainty omission."},{"cited_title":"Belia, F","cited_arxiv_id":null,"evidence_quote":"Documents that researchers misinterpret confidence intervals and standard error bars, supporting the paper's caveat that visualized uncertainty narrows rather than removes ambiguity."}],"review_version":1}