{"id":"1908ee13-c560-41ff-96a6-9064983459fb","arxiv_id":"2505.14816","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A structured two-step methodology for designing semantically-resonant abstract patterns, derived from expert workshops and tested with non-experts, is presented.","lead":"The authors introduce a two-step design recipe for creating black-and-white abstract patterns that visually evoke the concepts they represent, such as small dots for olives and large dots for tomatoes. They tested the recipe with non-designers, who rated it useful for generating such patterns for both concrete and abstract ideas.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation measures perceived process support, not whether the produced patterns actually evoke their concepts; Sec. 5 explicitly declines to assess design quality and Sec. 6 reports designs were often too abstract to identify, leaving the central effectiveness claim untested.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap: the evaluation relies on self-reported usefulness, ease of use, and helpfulness rather than on any objective measure of whether the produced patterns are semantically resonant. This is not a manufactured concern. The paper itself, in Sec. 5, explicitly disclaims any evaluation of output design quality, and in Sec. 6 reports that participant designs were often too abstract to identify the relevant items. If the central claim were modest, such as 'the methodology supports non-experts in generating design ideas' or 'participants find the methodology helpful,' the current evidence would be adequate. But the abstract and conclusion make a stronger claim about effectively supporting the design of semantically-resonant patterns, and that requires evidence that the output patterns actually evoke their referents for independent viewers. The concrete test proposed is a direct, feasible check using data already presented in the paper: a blind matching study on the 168 patterns from the evaluation workshops. If that test succeeds, the central claim gains real support; if it fails, the claim should be reframed to process support only. Since the reader already recommended CONDITIONAL and the proposed test would settle the issue, the reader's verdict remains appropriate and does not need adjustment.","tokens_in":24613,"tokens_out":3559,"duration_ms":34071,"concrete_test":"Run a blind matching study on the 168 patterns already collected (Figs. 70-93): recruit independent raters who were not involved in the study, show each of the 24 unlabeled pie charts (or individual pattern slices) in randomized order, and ask raters to assign each slice to one of the seven concepts in the set. Compute matching accuracy against the chance level of 1/7, with multiple raters and a permutation baseline. If accuracy is not reliably above chance, the central effectiveness claim is unsupported; if accuracy is clearly above chance, the claimed semantic resonance holds up.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, in the abstract and conclusion, is that the methodology \"effectively supports the general public in designing semantically-resonant abstract patterns.\" For this to be true, following the two-step process should yield patterns that an independent viewer can match to their intended concepts, at least at better-than-chance levels. The evaluation in Sec. 5 does not measure that outcome: participants rated usefulness, ease of use, and helpfulness (Sec. 5.5.1), and the authors state that the goal \"is to assess how our design methodology supports participants in the design process, and intentionally not to evaluate the quality of their specific designs.\" Self-reported process support can be high even when the resulting patterns do not resonate with viewers. Indeed, the Discussion (Sec. 6) concedes that participants' designs \"were often too abstract to identify the relevant items, in particular in the context of abstract concept sets.\" The methodology may plausibly support ideation and reduce perceived difficulty, but the collected data do not establish the outcome named in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a two-step design methodology for creating 'semantically-resonant' abstract patterns—first identifying the content to be visualized, then encoding it through pattern variables. The methodology is derived from qualitative coding of 13 design experts' workshop outputs, and its usefulness, ease of use, and helpfulness are evaluated in a second workshop with 12 non-expert participants who were taught the methodology and asked to create patterns for a concrete and an abstract concept set. The abstract and conclusion claim that the methodology 'effectively supports the general public in designing semantically-resonant abstract patterns for both abstract and concrete concepts.' The evaluation evidence, however, consists of self-reported Likert ratings and interview quotes about the design process, not of any independent assessment of whether the resulting patterns actually evoke their intended concepts.","tokens_in":24817,"tokens_out":3767,"duration_ms":35191,"significance":"If the central claim were supported, the paper would contribute a useful practical framework to an under-studied area, extending prior work on semantically-resonant colors to abstract patterns and providing an accessible entry point for non-experts. The paper has genuine strengths: the methodology is systematically derived from a substantial corpus of expert designs and is presented with extensive appendix material; the studies were pre-registered and the materials and data are shared in an OSF repository; and the authors are transparent about many limitations in Sec. 7 and the Discussion. The value of the methodological taxonomy itself is plausible. What is not supported by the reported evidence is the outcome claim that the process yields semantically-resonant patterns, because the evaluation measures perceived process support rather than the semantic discriminability of the generated patterns.","major_comments":[{"comment":"The central claim that the methodology 'effectively supports the general public in designing semantically-resonant abstract patterns' is not tested by the evidence in Sec. 5. The stated evaluation goal, quoted in Sec. 5, is 'to assess how our design methodology supports participants in the design process, and intentionally not to evaluate the quality of their specific designs.' The outcome measures in Sec. 5.5.1 are Likert ratings of usefulness, ease of use, and helpfulness, together with qualitative self-reports. These are process-oriented and self-referential: participants were taught the methodology and then asked whether it helped. No independent viewer was asked to match the generated patterns to their intended concepts, and no baseline condition without the methodology was included. Since the paper itself defines semantic resonance in Sec. 2.2 via Schloss et al.'s notion of semantic discriminability, which is about a viewer inferring the mapping from the visual features alone, the evaluation should have measured that inference or made clear that the effectiveness claim is only about perceived process support.","section":"Abstract and Sec. 5"},{"comment":"The paper's own discussion undermines the strong conclusion. Sec. 6 acknowledges that participants' semantically-resonant pattern designs 'were often too abstract to identify the relevant items, in particular in the context of abstract concept sets.' Sec. 7 states that the authors intentionally 'do not assess the impact of using semantically-resonant patterns in visualizations' and calls for future crowd-sourced perception experiments. These statements directly conflict with the abstract's claim that the methodology was shown to 'effectively support' the design of semantically-resonant patterns. At minimum, the claims should be scaled back to say that the methodology was perceived as useful, easy to use, and helpful for ideation, and that its effect on the resonance of the resulting patterns remains to be tested.","section":"Sec. 6 and Sec. 7"},{"comment":"The interpretation of the quantitative ratings is strained given the small sample and the lack of any comparison condition. The Likert means reported in Sec. 5.5.1 (usefulness M = 5.29, ease of use M = 5.33, helpfulness M = 5.33 on a 7-point scale) are moderate, and the concrete usefulness mean is 4.92, which is close to the scale midpoint. The claim that 'our design methodology thus appears to be meaningful in reducing the challenges users face in design' goes beyond what can be concluded from absolute Likert scores without a baseline. This is particularly important because Sec. 5.6 reports that all participants confirmed they used the methodology and attributed their designs to it; in a study where the methodology was just taught to the participants, such self-reports are partly an artifact of the instructional context rather than independent evidence of the methodology's value.","section":"Sec. 5.5.1 and Sec. 5.6"},{"comment":"The authors provide a thoughtful justification for not running a control condition, citing learning effects and between-subject variability. This is a reasonable position for a qualitative study, but it means the paper cannot support the causal or effectiveness language used in the conclusion. The conclusion should be limited to 'participants found the methodology helpful in generating and structuring their design ideas,' not 'the methodology effectively supports the design of semantically-resonant patterns.' A controlled perception study or a comparison of designs produced with and without the methodology would be needed to substantiate the stronger claim.","section":"Sec. 6, 'Methodology Evaluation' paragraph"}],"minor_comments":[{"comment":"The word 'seperately' should be corrected to 'separately.'","section":"Supplemental Material Pointers"},{"comment":"A quote attributed to P12 is followed by '(P2 for abstract),' which appears to be an incorrect participant label; please fix this inconsistency.","section":"Sec. 5.5.1"},{"comment":"The table lists 'hard working' as two words, whereas the text and concept selection in Sec. 5.1 use 'hardworking'; please standardize the spelling.","section":"Table 3"},{"comment":"The sentence reporting 'a trend that participants think our design methodology is more useful for abstract concept sets than concrete concept sets' is based on overlapping standard deviations in a sample of 12; phrasing this as a hypothesis or descriptive pattern rather than a trend would be more appropriate.","section":"Sec. 5.5.1"},{"comment":"The caption for Fig. 7 states that the plot shows ratings on the difficulty of designing patterns, but the caption does not clearly explain what the plotted percentages represent; please clarify whether they are percentages of participants rating each score or something else.","section":"Fig. 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about many of its limitations, which I read in its favor. The main issue is that the central claim is an effectiveness claim about the quality of the resulting patterns, while the evaluation is entirely about perceived process support. This can be fixed within the manuscript's scope by reframing the claims and conclusions, and perhaps by adding a small perception experiment in a revision. I do not see a reason to reject the paper outright: the methodological taxonomy is a useful contribution and the qualitative corpus is shared openly. The main risk is that the paper's title and abstract over-promise relative to what the data can show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper's real contribution is the two-step methodology for designing semantically-resonant abstract patterns, derived from 13 expert workshops with 273 patterns. That is new and genuinely useful. The evaluation, however, does not test the headline claim that the methodology 'effectively supports' the general public in producing semantically-resonant patterns. It measures participants' self-reported usefulness, ease of use, and helpfulness, with no baseline and no independent check of whether the outputs evoke their concepts. The authors are transparent about this: Sec. 5 says they intentionally did not evaluate design quality, and the Discussion concedes the patterns were 'often too abstract to identify.' So the conclusion overstates what the data support.\n\nWhat the paper does well: the methodology itself is a sensible, well-organized synthesis. Step 1 (identify content via meaning, human reaction, features, literal association) and Step 2 (encode using pattern variables, with basic vs. complex patterns) are clearly presented and grounded in embodied cognition and metaphor theory. The approach of deriving it from expert practice rather than from theory alone is a real strength, and the appendix with 273 expert-designed patterns is a useful resource. They also pre-registered both studies on OSF and share materials, which is more than many HCI papers do.\n\nSoft spots, in proportion: the main one is the claim-evidence gap. Likert means around 5.3 on a 7-point scale from 12 participants are fine as subjective impressions, but they don't establish that a non-expert following the method produces patterns that viewers can match to concepts. The methodology could be a fine ideation aid and still fail at the outcome named in the title. A second, minor issue: the evaluation used an earlier version of the methodology, which the authors acknowledge and argue is structurally equivalent; that's okay but worth noting. The self-referential aspect (experts derived the method, novices rate whether it helped) is real but not disqualifying—it's inherent to qualitative tool-building research when the claim is scoped as process support.\n\nWho should read this: visualization designers and researchers working with black-and-white patterns, accessibility, or physicalization; also anyone teaching pattern design. As a referee, I would ask the authors to either soften the central claim to 'supports the design process' or add a perception experiment with an independent audience. The methodology is worth preserving; the conclusion needs a recalibration.","headline":"A useful synthesis of expert strategies for embedding semantics into abstract patterns, but the evaluation measures perceived support rather than whether the resulting patterns actually resonate, so the headline claim runs ahead of the evidence.","tokens_in":25285,"tokens_out":2475,"would_cite":true,"duration_ms":22279,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that creating abstract patterns which intuitively evoke the concepts they represent reduces to a two-step recipe—first identify what a concept means to people, then encode it through pattern variables—and that…","keywords":["semantically-resonant patterns","data visualization","pattern design methodology","visual variables","abstract patterns","non-expert design","semantic association","design workshop"],"falsifier":"A controlled study in which naive viewers are asked to match non-expert designs made with the methodology to the intended concepts without a legend; if matching accuracy is no better than chance, the claim that the methodology effectively produces semantically-resonant patterns would be refuted. A between-group comparison of design quality with and without the methodology would serve the same purpose.","tokens_in":24432,"feed_emoji":"🧩","tokens_out":5069,"duration_ms":40753,"temperature":0.7,"pith_summary":"The paper tries to establish that designing semantically-resonant abstract patterns—black-and-white textures that intuitively evoke the data category they stand for, such as small dots for olives and large dots for tomatoes—can be reduced to a structured two-step recipe that any non-designer can follow. If true, visualization designers and the general public gain a systematic way to embed meaning into charts that must do without color, labels, or icons. The recipe was distilled from workshops in which 13 design experts produced 273 patterns for vegetable, music, and emotion concept sets, and then tested with 12 non-designers, who rated the method useful, easy to use, and helpful for both concrete and abstract concepts.","feed_headline":"Two-step method lets novices design meaningful chart patterns","feed_subtitle":"Expert-derived recipe helps people without design training encode abstract and concrete ideas into black-and-white patterns.","key_machinery":"The two-step design methodology itself is the central object. Step 1 (identifying content) offers four association routes grounded in conceptual metaphor theory and embodied cognition: meaning-based, human-reaction-based, feature-based, and literal-meaning-based. Step 2 (encoding) draws on a pattern design space that treats patterns as composite visual variables, distinguishing basic patterns (varying shape, size, fill, orientation, or forming an ordinal scale) from complex patterns (varying spatial arrangement and appearance relationships). The methodology works because it separates what to express from how to express it.","core_discovery":"The central claim is that semantically-resonant pattern design decomposes into two steps: identifying the content to be visualized (via the concept's meaning, human reactions to it, its features, or its literal name), then encoding that content into an abstract pattern using the visual variables available to patterns (shape, size, fill, orientation, and spatial or appearance relationships among primitives). The authors derive this structure from qualitative coding of expert design workshops and report that non-expert participants could apply it successfully to new concept sets (ball games and personality traits).","pith_inferences":["The paper's own limitation section concedes that it never tests whether viewers can decode the patterns; the natural completion is a perception study measuring legend-free matching accuracy, which would verify that 'resonant' holds at the reader's end, not just the designer's.","Because every pattern was created inside an equal-slice pie chart, the methodology's behavior in thin slices, small multiples, or non-circular geometries is untested; a computational pattern library parameterized by the Step-2 variables would let designers stress-test the approach across chart types.","The finding that concrete concepts were rated harder to design for than abstract ones suggests the bottleneck is abstraction skill rather than idea generation, so tools that scaffold the sketching step (pre-built primitives or generative fills) could directly relieve the difficulty participants reported.","The ordinal-scale technique (for instance, spike count mapped to emotional negativity) hints that semantically-resonant patterns could encode ordered magnitudes, not just category identity, which would extend the claimed benefit beyond categorical charts."],"forward_implications":["People with no design or sketching training can use the two-step methodology to generate semantically-resonant patterns for both concrete concepts and abstract ones.","The methodology condenses expert design strategies into a checklist of association routes and pattern variables, making the thinking directions used by professional designers available to anyone.","Visualization designers gain a systematic alternative to semantically-resonant colors for charts that must be black-and-white, including print, data embroidery, and small displays where legends and labels waste space.","The subjective ratings suggest the method helps most for abstract concepts, where there is no physical appearance to copy, which is exactly where designers previously had the least guidance.","Because prior work linked semantically-resonant abstract patterns to improved pie-chart reading speed, following this methodology may yield patterns that are both meaningful and faster to read."],"supporting_citations":[{"why":"Defines semantically-resonant design for colors, supplies the vegetable concept set, and provides the concept-selection approach the paper adopts.","marker":"[20]"},{"why":"Prior study showing abstract geometric patterns can improve pie-chart reading speed and that designers spontaneously embed semantics; its observed strategy motivates this work.","marker":"[15]"},{"why":"The pattern design space treating patterns as composite visual variables, which structures Step 2 of the methodology.","marker":"[12]"},{"why":"Concreteness ratings used to select and validate concept sets spanning concrete to abstract.","marker":"[4]"},{"why":"Conceptual metaphor theory, the theoretical basis for the content-identification approaches in Step 1.","marker":"[18]"},{"why":"Semantic discriminability framework that grounds the paper's definition of a semantically-resonant visual variable.","marker":"[32]"},{"why":"The workshop methodology the authors follow when evaluating the design methodology with non-experts.","marker":"[36]"}],"fun_headline_variants":["Two-step recipe helps anyone design pattern charts","Novices craft meaningful chart patterns with two-step method","Design method makes semantic chart patterns easy for all","Abstract pattern design: a two-step method for everyone","Make meaningful chart patterns: two-step method for all"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats participants' self-reported ratings of usefulness, ease of use, and helpfulness as evidence that the methodology effectively supports pattern design, without measuring whether an outside audience can match the resulting patterns to their concepts or comparing against a no-methodology control group.","fun_headline_variants_meta":{"raw":{"variants":["Two-step recipe helps anyone design pattern charts","Novices craft meaningful chart patterns with two-step method","Design method makes semantic chart patterns easy for all","Abstract pattern design: a two-step method for everyone","Make meaningful chart patterns: two-step method for all"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3742,"prompt_tokens":822,"completion_tokens":2920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":2847}},"tokens_in":438,"tokens_out":2920,"duration_ms":32337,"temperature":1.0,"reasoning_tokens":2847,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:28:21.632338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study in which naive viewers are asked to match non-expert designs made with the methodology to the intended concepts without a legend; if matching accuracy is no better than chance, the claim that the methodology effectively produces semantically-resonant patterns would be refuted. A between-group comparison of design quality with and without the methodology would serve the same purpose.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines semantically-resonant design for colors, supplies the vegetable concept set, and provides the concept-selection approach the paper adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior study showing abstract geometric patterns can improve pie-chart reading speed and that designers spontaneously embed semantics; its observed strategy motivates this work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The pattern design space treating patterns as composite visual variables, which structures Step 2 of the methodology."},{"cited_title":"Brysbaert, A","cited_arxiv_id":null,"evidence_quote":"Concreteness ratings used to select and validate concept sets spanning concrete to abstract."},{"cited_title":"Lakoff and M","cited_arxiv_id":null,"evidence_quote":"Conceptual metaphor theory, the theoretical basis for the content-identification approaches in Step 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Semantic discriminability framework that grounds the paper's definition of a semantically-resonant visual variable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The workshop methodology the authors follow when evaluating the design methodology with non-experts."}],"review_version":1}