Pith. sign in

REVIEW 4 major objections 3 minor 24 references

Affordances of Sketched Notations for Multimodal UI Design and Development Tools

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read People's UI sketches are ambiguous alone but clear in context, so sketch-based design tools should interpret whole designs.

desk verdict The dataset and CD comparison are genuinely useful; the claim that context disambiguates sketches is asserted rather than shown. read the letter →

arxiv 2508.09342 v1 pith:XFMKGAHW submitted 2025-08-12 cs.HC

Sam H. Ross , Yunseo Lee , Coco K. Lee , Jayne Everson , R. Benjamin Shapiro This is my paper
classification cs.HC
keywords sketch-basedUIdesignnotationcognitivedimensionsofnotationssketchrecognitionmultimodaltoolshuman-centeredAIdatasetsFlexi
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the inputs a sketch-based UI design tool can understand form a notation, and that design tools should be evaluated by how usable and expressive those notations are. To compare two extremes, the authors analyze a fixed notation taken from an existing UI sketch dataset (FixedSketch) and a flexible notation produced by 21 participants sketching interfaces with no rules (FlexiSketch). Their central finding is that participants' element-level sketches are often ambiguous when viewed alone, but become interpretable when seen in the context of the whole design. This means element-based recognition, which classifies each stroke separately, will fail for intuitive sketching; future tools need context-aware, adaptive models that can learn a user's personal and evolving notation.

What carries the argument

The Cognitive Dimensions of Notations framework (Green & Blackwell) is the analytic instrument; it provides 14 dimensions—such as viscosity, role-expressiveness, and hidden dependencies—used to compare two hypothetical systems. The central object is the 'notation' itself: the set of inputs a sketch tool can interpret, which in AI systems is implicitly defined by the training dataset. FixedSketch and FlexiSketch are contrived systems that embody a closed and an open notation, respectively, allowing a structured comparison of the two extremes.

What would settle it

Crop the 4,105 labeled elements from the participants' sketches, show each cropped element alone to independent raters, and ask them to name the UI element; if the raters can accurately identify elements without seeing the full design, the claim that element-level notations are ambiguous in isolation is falsified.

Watch

Extended reading notes

Core claim

The central claim is that people sketching user interfaces without imposed rules produce notations that are inconsistent and ambiguous at the element level, yet coherent and meaningful in the context of an entire design. In a study of 21 participants who produced 84 sketches with 4,105 labeled elements, text was drawn as literals, squiggles, lines, dots, Xs, and bounding boxes; images ranged from detailed portraits to empty placeholders; and the same mark could stand for a card, a button, or an icon depending on surrounding elements. The authors interpret these findings through the Cognitive Dimensions of Notations framework, arguing that a hypothetical FlexiSketch system built on this flexi

Load-bearing premise

The analysis assumes that the 21 participants, most of whom had recent drawing experience, sketch the way future UI designers generally would, so that a notation inferred from their sketches represents the broader design population.

Editorial extensions

If this is right

  • Existing element-based sketch recognition datasets and models will poorly support intuitive UI sketching; tools trained only on such datasets will force users to learn artificial representation rules.
  • Sketch-based design tools should model spatial and relational context among elements, not classify strokes in isolation, to interpret open-ended notations.
  • Systems need online learning to adapt to each individual's changing shorthand, rather than assuming a fixed per-user notation.
  • User corrections should be treated as permanent and used as training signal to personalize the model (via human-in-the-loop reinforcement learning).
  • Supporting secondary notations—annotations, arrows, 'x6' copy commands—would let users express behavior and layout intent directly in the sketch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ambiguity-in-context result generalizes beyond UI sketching, the same principle likely applies to other sketch-based specification domains such as diagrams and storyboards, suggesting that freeform sketch recognition should generally be holistic.
  • A testable architectural inference is that a transformer-based model encoding whole-sketch spatial relations should outperform element-level classifiers on the released FlexiSketch dataset, which could serve as a benchmark.
  • The observed mix of literal text for headers and shorthand for body text implies a 'selective fidelity' principle: design tools might preserve semantic detail where it matters and tolerate abstraction elsewhere.
  • The 'x6' and squiggle repeat notations are evidence that users naturally invent a macro language within sketching; supporting such user-defined abstractions could make sketch tools more expressive than menu-based prototyping tools.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper applies the Cognitive Dimensions of Notations framework to compare two UI sketch notations: the rule-based notation of the ImageCLEF drawnUI dataset and a 'FlexiSketch' notation derived from a study of 21 participants freely sketching UIs. The authors analyze the cognitive dimensions of two hypothetical systems (FixedSketch and FlexiSketch), argue that the open-ended notation is more expressive and less cognitively demanding, and claim that element-level representations are ambiguous in isolation but interpretable in context. They propose four recommendations for future sketch-based design tools and release an open dataset.

Significance. If the central claim were supported, the paper would make a meaningful contribution by refocusing sketch-based UI tool design on human-centered notation and by motivating context-aware, adaptive recognition. The open dataset is a concrete and citable resource, and the cognitive dimensions analysis is a structured way to compare notations. However, the key empirical claim about contextual interpretability is not yet established, which limits the current significance.

major comments (4)
  1. [V-B / VI-C / VII-B Recommendation 1] The central claim that element-level notations are ambiguous in isolation but interpretable in context is not empirically demonstrated. Phase 3 (V-B) had participants label their own sketches after receiving a UI element taxonomy; this only shows that the drawer can report their own intention. It does not show that whole-design context makes the overlapping representations in Figure 8 interpretable to an independent observer. No inter-rater reliability, independent coders, or isolated-vs-context labeling comparison is reported. Because this claim grounds Recommendation 1 and the critique of element-based recognition, it must be supported by a validation study or explicitly reframed as a hypothesis.
  2. [Abstract; VII-B] The paper asserts that the FlexiSketch notation 'cannot be supported with prevailing, element-based approaches to UI sketch recognition.' The only evidence offered is visual similarity of element representations (e.g., Fig. 8). No existing recognizer or baseline classifier is applied to the FlexiSketch dataset. Given that this is a load-bearing claim for the paper's argument, it should be tested (e.g., by running a simple element classifier on the released dataset) or softened to an untested conjecture.
  3. [VI-A; VIII] The FlexiSketch notation is defined entirely by the sketches of 21 participants, 86% of whom had recent drawing experience and 67% had interface design experience. The paper acknowledges the small sample in Section VIII but does not discuss how the experience skew might affect the observed representational patterns (e.g., shorthand, placeholders, secondary notations). Since the notation is derived from this sample, the generalizability of the cognitive dimension analysis to broader populations (including novices) is uncertain. The authors should either restrict their claims to similarly experienced populations or provide a sensitivity discussion.
  4. [III; IV; VI-C] The comparison of FixedSketch and FlexiSketch is asymmetrical: FixedSketch is analyzed heuristically from the dataset's instructions, while FlexiSketch is analyzed from empirical observations. Statements about FixedSketch's hard mental operations or error-proneness are analytic predictions, not measured outcomes. The paper should explicitly label the FixedSketch analysis as a heuristic evaluation and avoid implying that the differential affordances are empirically established for both notations.
minor comments (3)
  1. [VI-B-1-a; Fig. 3] The text states that the six Text representations were 'often combined with one another.' It is unclear whether an element drawn with both a literal and a bounding box is counted in both categories or as a separate combined category. The percentages in Figure 3 should be defined unambiguously (e.g., per-element vs. per-representation).
  2. [III-B] Minor grammar issue: 'As with premature commitment where elements can be created in any order' should read 'As with premature commitment, where elements...'
  3. [Fig. 8] Figure 8 shows examples from different participant sketches, but the caption does not state whether the similar representations occurred within a single drawing or across drawings. Since the text discusses ambiguity within whole designs, this clarification is important.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: the central 'interpretable in context' claim reduces to participants labeling their own sketches, not to independent contextual evidence.

  1. self definitional [Abstract; Section IV; Section V-B (Phase 3)]
    "We find that participants' sketches were composed of element-level notations that are ambiguous in isolation but are interpretable in context within whole designs. ... Finally, we went through each of their drawings with them and asked them to describe what each sketched UI element was. ... We treat the set of our participants sketches as a notation"

    The FlexiSketch notation is defined as the set of participant sketches (Section IV), and the element labels come from the same participants labeling their own drawings in Phase 3, with a provided taxonomy and researcher suggestions. Consequently the abstract's finding that the notation is 'interpretable in context' is true by construction: the person who produced each sketch can interpret it, so the whole-design context is not independently shown to supply the meaning. No inter-rater reliability, independent coders, or isolated-vs-in-context comparison is reported; Section VI-C only documents visually overlapping forms (Fig. 8). Recommendation 1 (context-based relational modeling) thus rests on the self-labeling protocol rather than on evidence that spatial/relational context disambiguates

full rationale

The paper's cognitive dimensions analysis is otherwise self-contained: it applies an external framework (Green's CD) to two notations, and the qualitative ratings of FixedSketch versus FlexiSketch are analytical judgments, not derived from a fitted parameter or self-citation. The open dataset is a genuine empirical contribution, and the authors acknowledge the small sample and invite generalization in Section VIII. However, the paper's strongest advertised finding—'ambiguous in isolation but interpretable in context within whole designs'—is not independently established. The interpretability half is supported only by the Phase 3 protocol in which participants labeled their own drawings; thus the notation's interpretability is co-extensive with the labeling procedure used to define it. This is a self-definitional reduction for the central claim that motivates the contextual-modeling recommendations. It does not invalidate the cognitive dimensions analysis or the dataset, but it does mean one of the paper's key predictions is partially circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on treating the Cognitive Dimensions framework as a valid evaluation lens, treating the ImageCLEF dataset instructions as the definitive FixedSketch notation, and treating participant sketches as a legitimate notation. The representativeness of the participant sample is a domain assumption the paper partially acknowledges.

assumptions (4)
  • domain assumption The Cognitive Dimensions of Notations framework is a valid and appropriate lens for evaluating sketch-based UI notations
    The paper applies Green's framework without justifying why it is suitable for visual sketches beyond analogy to programming notations (Section III, Table I).
  • domain assumption The set of ImageCLEF drawnUI dataset instructions constitutes the complete FixedSketch notation
    The authors treat the 'correct' examples in Figure 1 as the total legal notation for the hypothetical system.
  • domain assumption The 21 participants' sketches constitute a coherent notation that can be analyzed with cognitive dimensions
    The FlexiSketch notation is defined as the aggregate of participant drawings, and the paper draws conclusions about its usability without a user study of the hypothetical system.
  • domain assumption Participant experience distribution is representative of target users
    The sample skews toward experienced drawers and designers (86% drawing experience), and the paper reports no significant differences by experience but does not generalize beyond this sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Affordances of Sketched Notations for Multimodal UI Design and Development Tools." pith.science (2026). https://pith.science/paper/XFMKGAHW

@misc{pith2026250809342,
  author       = {Pith},
  title        = {Pith review of: Affordances of Sketched Notations for Multimodal UI Design and Development Tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XFMKGAHW}},
  note         = {Machine review of arXiv:2508.09342}
}
read the original abstract

Multimodal UI design and development tools that interpret sketches or natural language descriptions of UIs inherently have notations: the inputs they can understand. In AI-based systems, notations are implicitly defined by the data used to train these systems. In order to create usable and intuitive notations for interactive design systems, we must regard, design, and evaluate these training datasets as notation specifications. To better understand the design space of notational possibilities for future design tools, we use the Cognitive Dimensions of Notations framework to analyze two possible notations for UI sketching. The first notation is the sketching rules for an existing UI sketch dataset, and the second notation is the set of sketches generated by participants in this study, where individuals sketched UIs without imposed representational rules. We imagine two systems, FixedSketch and FlexiSketch, built with each notation respectively, in order to understand the differential affordances of, and potential design requirements for, systems. We find that participants' sketches were composed of element-level notations that are ambiguous in isolation but are interpretable in context within whole designs. For many cognitive dimensions, the FlexiSketch notation supports greater intuitive creative expression and affords lower cognitive effort than the FixedSketch notation, but cannot be supported with prevailing, element-based approaches to UI sketch recognition. We argue that for future multimodal design tools to be truly human-centered, they must adopt contemporary AI methods, including transformer-based and human-in-the-loop, reinforcement learning techniques to understand users' context-rich expressive notations and corrections.

Figures

Figures reproduced from arXiv: 2508.09342 by the authors.

Figure 1
Figure 1. The ImageCLEF drawnUI dataset instructions for people drawing the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Interfaces of increasing complexity that participants sketched during the study. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 6
Figure 6. The different ways people represented Icons. Most Icons were drawn [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figures from the paper (7 more)
Figure 7
Figure 7. Figure 7: (a) Example ways people represented Cards, Line Breaks, and Buttons. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 4
Figure 4. Figure 4: The different ways participants represented Images, including as [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Depictions of the circular profile photo on the copied personal [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 8
Figure 8. Figure 8: Examples of Text, Image, Icon, Card, and Button elements represented [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 3
Figure 3. Figure 3: This increase occurred both in the total amount of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 9
Figure 9. Figure 9: The navigation bar on interface 3 and a sample of sketches of the [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Participant 19’s original UI design, depicting an iceberg that should [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages

  1. [1]

    Denim: finding a tighter fit between tools and practice for web site design,

    J. Lin, M. W. Newman, J. I. Hong, and J. A. Landay, “Denim: finding a tighter fit between tools and practice for web site design,” Proceedings of the SIGCHI conference on Human Factors in Computing Systems , 2000. [Online]. Available: https://api.semanticscholar.org/CorpusID:8007626

    work page 2000

  2. [2]

    Design breakdowns: Designer-developer gaps in representing and interpreting interactive systems,

    N. Maudet, G. Leiva, M. Beaudouin-Lafon, and W. E. Mackay, “Design breakdowns: Designer-developer gaps in representing and interpreting interactive systems,” Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing , 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:14682858

    work page 2017

  3. [3]

    Sketching user experiences: Getting the design right and the right design,

    B. Buxton, “Sketching user experiences: Getting the design right and the right design,” 2007. [Online]. Available: https://api.semanticscholar. org/CorpusID:110094395

    work page 2007

  4. [4]

    Sketching user experiences: The workbook,

    S. Greenberg, S. Carpendale, N. Marquardt, and B. Buxton, “Sketching user experiences: The workbook,” 2011. [Online]. Available: https://api.semanticscholar.org/CorpusID:11484319

    work page 2011

  5. [5]

    Cognitive dimensions of information artefacts: a tutorial,

    T. R. G. Green and A. F. Blackwell, “Cognitive dimensions of information artefacts: a tutorial,” 1998. [Online]. Available: https://api.semanticscholar.org/CorpusID:59743075

    work page 1998

  6. [6]

    Sitemaps, storyboards, and specifications: a sketch of web site design practice,

    M. W. Newman and J. A. Landay, “Sitemaps, storyboards, and specifications: a sketch of web site design practice,” in Symposium on Designing Interactive Systems , 2000. [Online]. Available: https: //api.semanticscholar.org/CorpusID:1523086

    work page 2000

  7. [7]

    Rough and ready prototypes: lessons from graphic design,

    Y . Y . Wong, “Rough and ready prototypes: lessons from graphic design,” in International Conference on Human Factors in Computing Systems ,

    work page

  8. [8]

    How designers design and program interactive behaviors,

    B. A. Myers, S. Y . Park, Y . Nakano, G. Mueller, and A. J. Ko, “How designers design and program interactive behaviors,” 2008 IEEE Symposium on Visual Languages and Human-Centric Computing , pp. 177–184, 2008. [Online]. Available: https://api.semanticscholar.org/ CorpusID:1310663

    work page 2008

Show all 24 references
  1. [9]

    Silk: sketching interfaces like krazy,

    J. A. Landay, “Silk: sketching interfaces like krazy,” Conference Companion on Human Factors in Computing Systems , 1996. [Online]. Available: https://api.semanticscholar.org/CorpusID:37801494

    1996

  2. [10]

    Javasketchit: Issues in sketching the look of user interfaces,

    A. J. P. Caetano, N. Goulart, M. J. Fonseca, and J. A. Jorge, “Javasketchit: Issues in sketching the look of user interfaces,” 2002. [Online]. Available: https://api.semanticscholar.org/CorpusID:7172976

    2002

  3. [11]

    Doodle2app: native app code by freehand ui sketching,

    S. Mohian and C. Csallner, “Doodle2app: native app code by freehand ui sketching,” in Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems, ser. MOBILESoft ’20. New York, NY , USA: Association for Computing Machinery, 2020, p. 81–...

    2020

  4. [12]

    Deep learning- based prototyping of android gui from hand-drawn mockups,

    A. A. Abdelhamid, S. R. Alotaibi, and A. Mousa, “Deep learning- based prototyping of android gui from hand-drawn mockups,” IET Software, vol. 14, no. 7, pp. 816–824, 2020. [Online]. Available: https: //ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/iet-sen.2019.0378

    2020

  5. [13]

    Swire: Sketch-based user interface retrieval,

    F. Huang, J. F. Canny, and J. Nichols, “Swire: Sketch-based user interface retrieval,” Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:140226268

    2019

  6. [14]

    Uisketch: A large-scale dataset of ui element sketches,

    V . P. Sermuga Pandian, S. Suleri, and P. D. M. Jarke, “Uisketch: A large-scale dataset of ui element sketches,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , ser. CHI ’21. New York, NY , USA: Association for Computing Machinery,

    2021

  7. [15]

    Overview of ImageCLEFdrawnUI 2021: The detection and recognition of hand drawn and digital website uis task,

    R. Berari, A. Tauteanu, D. Fichou, P. Brie, M. Dogariu, L. D. S ¸tefan, M. G. Constantin, and B. Ionescu, “Overview of ImageCLEFdrawnUI 2021: The detection and recognition of hand drawn and digital website uis task,” in CLEF2021 Working Notes , ser. CEUR Workshop Pro- ceedings...

    2021

  8. [16]

    Cognitive dimensions of notations,

    T. R. G. Green, “Cognitive dimensions of notations,” 1990. [Online]. Available: https://api.semanticscholar.org/CorpusID:13447890

    1990

  9. [17]

    Epistemological pluralism and the revaluation of the concrete,

    S. Turkle and S. Papert, “Epistemological pluralism and the revaluation of the concrete,” Journal of Mathematical Behavior , vol. 11, no. 1, pp. 3–33, 1992

    1992

  10. [18]

    Natural programming languages and environments,

    B. A. Myers, J. F. Pane, and A. J. Ko, “Natural programming languages and environments,” Communications of the ACM , vol. 47, pp. 47 – 52, 2004. [Online]. Available: https://api.semanticscholar.org/CorpusID: 3200452

    2004

  11. [19]

    Webui: A dataset for enhancing visual ui understanding with web semantics,

    J. Wu, S. Wang, S. Shen, Y .-H. Peng, J. Nichols, and J. P. Bigham, “Webui: A dataset for enhancing visual ui understanding with web semantics,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , 2023, pp. 1–14

    2023

  12. [20]

    Navigating the convenience trap with chatgpt and google translate: The risks of homogenization in translation teaching for vietnamese students,

    D. H. Tu and Mai, “Navigating the convenience trap with chatgpt and google translate: The risks of homogenization in translation teaching for vietnamese students,” International Journal of Linguistics, Literature and Translation , 2024. [Online]. Available: https://api.semanti...

    2024

  13. [21]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kel- ton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructio...

    2022

  14. [22]

    Personalizing reinforcement learning from human feedback with variational preference learning,

    S. Poddar, Y . Wan, H. Ivison, A. Gupta, and N. Jaques, “Personalizing reinforcement learning from human feedback with variational preference learning,” ArXiv, vol. abs/2408.10075, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271904022

    2024 arXiv

  15. [1992]

    Available: https://api.semanticscholar.org/CorpusID: 34202293

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 34202293

  16. [2021]

    Available: https://doi.org/10.1145/3411764.3445784

    [Online]. Available: https://doi.org/10.1145/3411764.3445784

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.