REVIEW 2 major objections 3 cited by
Deep Learning in a Computational Model for Conceptual Shifts in a Co-Creative Design System
T0 review · 2 major / 0 minor · reviewed 2026-05-25 · grok-4.3
Pith's one-line read Increasing novelty in AI sketch contributions leads to higher creative outcomes in co-creative design.
desk verdict The paper implements a deep-learning embedding distance as a controllable novelty knob in a live sketching co-creative loop and reports a user-study link to higher creativity scores, but the metric is not shown to track conceptual shifts over visual ones. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
A novelty metric computed from distances in deep learning vector embeddings of sketches, used to control the degree of conceptual shift in the AI's responses.
What would settle it
Run an experiment where independent raters score the conceptual novelty of the generated sketches and check whether those scores correlate with the model's vector-based novelty values; a lack of correlation would undermine the claim.
Extended reading notes
Core claim
The paper claims that a deep learning-based novelty metric can be used to generate conceptual shifts in an AI agent's sketches within a co-creative system, and empirical results from a user study indicate that higher novelty in the AI's contributions is associated with higher creative outcomes while low novelty is associated with lower creative outcomes.
Load-bearing premise
The distance in the deep learning vector space corresponds to meaningful conceptual shifts as perceived by humans rather than mere visual similarities.
Editorial extensions
If this is right
- Design systems can intentionally vary AI novelty to support better creative partnerships.
- AI contributions that are too similar to the human's work may reduce overall creativity.
- The vector-based approach provides a computational way to implement conceptual shifts without manual rules.
- The user study results suggest that moderate to high novelty levels optimize creative output.
Reading between the lines
- This model could be tested in other visual or non-visual creative tasks to see if the novelty-creativity link holds.
- If the vector distances do not align with human judgments of conceptual difference, the model's effectiveness would need re-evaluation.
- Future systems might combine this with other metrics like usefulness or surprise to refine the AI's role.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a computational model for conceptual shifts in a co-creative design system. A deep-learning novelty metric is applied to vector representations of sketches; the AI agent responds to a human designer's sketch by generating a new sketch with intentionally varied visual and conceptual similarity. A user study is reported in which increasing novelty in the AI contribution is associated with higher creative outcomes, while low novelty is associated with lower creative outcomes.
Significance. If the novelty metric is shown to track human-perceived conceptual distance rather than low-level visual statistics, and if the user-study measures are validated, the work could inform the design of AI partners that deliberately modulate conceptual distance to support creativity. The approach of embedding DL representations directly into an interactive sketching loop is a concrete step toward computational models of co-creativity.
major comments (2)
- [Abstract] Abstract: the central empirical claim—that the DL vector-distance novelty metric produces higher creative outcomes—rests on the untested assumption that Euclidean or cosine distance in the embedding space corresponds to human conceptual shifts. No human-rating correlation, ablation against pixel-level or edge-based baselines, or control for visual complexity is described.
- [User Study] User-study description: the creativity measure and its validation are not specified (e.g., how inter-rater reliability was assessed, how confounds such as sketch complexity or style were controlled). Without these details the reported association cannot be evaluated.
Simulated Author's Rebuttal
We thank the referee for their detailed review and constructive comments on our manuscript. We address each of the major comments below.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central empirical claim—that the DL vector-distance novelty metric produces higher creative outcomes—rests on the untested assumption that Euclidean or cosine distance in the embedding space corresponds to human conceptual shifts. No human-rating correlation, ablation against pixel-level or edge-based baselines, or control for visual complexity is described.
Authors: The referee is correct that the manuscript does not include a direct human validation of the embedding space distances as measures of conceptual shifts, nor ablations or controls for visual complexity. The reported association is between the AI-generated novelty levels (computed via the DL metric) and the creative outcomes in the user study. We will revise the abstract to clarify the scope of the empirical claim and add a discussion of this limitation, including suggestions for future validation studies. revision: yes
-
Referee: [User Study] User-study description: the creativity measure and its validation are not specified (e.g., how inter-rater reliability was assessed, how confounds such as sketch complexity or style were controlled). Without these details the reported association cannot be evaluated.
Authors: We agree that the description of the user study in the manuscript lacks sufficient detail regarding the creativity measure, its validation, inter-rater reliability, and controls for potential confounds. We will revise the user study section to provide a more complete account of the methodology, including how creativity was measured and any steps taken to ensure reliability and control for confounds. revision: yes
Circularity Check
No circularity: empirical association reported from user study with no derivations or self-referential reductions
full rationale
The paper describes a deep-learning novelty metric applied to sketches, integrates it into a co-creative system, and reports a user-study correlation between AI novelty level and creative outcomes. No equations, parameter fits, or derivation chains are present. The central claim is an observed statistical association, not a quantity derived from fitted inputs or self-citations. The metric's validity (whether vector distance tracks conceptual vs. visual distance) is an external assumption, not a circularity in the reported result itself. No load-bearing self-citations or renamings of known results appear in the provided text.
Assumptions & free parameters
Cite this review
Pith. "Pith review of Deep Learning in a Computational Model for Conceptual Shifts in a Co-Creative Design System." pith.science (2026). https://pith.science/paper/WD6QNWUJ
@misc{pith2026190610188,
author = {Pith},
title = {Pith review of: Deep Learning in a Computational Model for Conceptual Shifts in a Co-Creative Design System},
year = {2026},
howpublished = {\url{https://pith.science/paper/WD6QNWUJ}},
note = {Machine review of arXiv:1906.10188}
}
read the original abstract
This paper presents a computational model for conceptual shifts, based on a novelty metric applied to a vector representation generated through deep learning. This model is integrated into a co-creative design system, which enables a partnership between an AI agent and a human designer interacting through a sketching canvas. The AI agent responds to the human designer's sketch with a new sketch that is a conceptual shift: intentionally varying the visual and conceptual similarity with increasingly more novelty. The paper presents the results of a user study showing that increasing novelty in the AI contribution is associated with higher creative outcomes, whereas low novelty leads to less creative outcomes.
Figures
Forward citations
Cited by 3 Pith papers
-
Interaction-Centered Intelligence: Toward an Interaction-Based Theory of Human-AI Co-Creation
Proposes Interaction-Centered Intelligence as a framework where intelligence emerges from interaction dynamics rather than internal agent computation.
-
A Systematic Review of Human-AI Co-Creativity
A PRISMA-style review of 62 co-creative systems identifies six design dimensions and 24 design considerations, reporting that user control and adaptive proactivity are associated with better collaboration outcomes.
-
AI Drawing Partner: Co-Creative Drawing Agent and Research Platform to Model Co-Creation
A drawing agent that logs and visualizes human-AI co-creation through the co-creative sense-making framework, tested on ten sessions.
Reference graph
Works this paper leans on
-
[1]
[Bonazza 2019] Bonazza, M
work page 2019
-
[2]
SketchTogether. https://sketchtogether.com. Accessed: 2019- 02-27. [Carbune 2017] Carbune, V
work page 2019
-
[3]
https://www.tensorflow.org/tutorials/ sequences/recurrent_quickdraw
Recur- rent neural networks for drawing classification. https://www.tensorflow.org/tutorials/ sequences/recurrent_quickdraw. [Colton et al. 2015] Colton, S.; Halskov, J.; Ventura, D.; Gouldstone, I.; Cook, M.; and Ferrer, B. P
work page 2015
-
[4]
The paint- ing fool sees! new projects with the automated painter. In ICCC, 189–196. [Das and Gamb¨ack 2014] Das, A., and Gamb ¨ack, B
work page 2014
-
[5]
Poetic machine: Computational creativity for automatic po- etry generation in bengali. In ICCC, 230–238. [Davis et al. 2015a] Davis, N.; Hsiao, C.-P.; Popova, Y .; and Magerko, B. 2015a. An enactive model of creativity for computational collaboration and co-creation. In Creativity in the Digital Age. Springer. 109–133. [Davis et al. 2015b] Davis, N. M.; H...
work page 2009
-
[6]
Imagenet: A large-scale hierarchical image database. [Gero 2000] Gero, J. S
work page 2000
-
[7]
Technological fore- casting and social change 64(2-3):183–196
Computational models of in- novative and creative design processes. Technological fore- casting and social change 64(2-3):183–196. [Grace et al. 2015] Grace, K.; Maher, M. L.; Fisher, D.; and Brady, K
work page 2015
-
[8]
In Design Computing and Cogni- tion’14
Modeling expectation for evaluating sur- prise in design creativity. In Design Computing and Cogni- tion’14. Springer. 189–206. [Guzdial et al. 2019] Guzdial, M.; Liao, N.; Chen, J.; Chen, S.-Y .; Shah, S.; Shah, V .; Reno, J.; Smith, G.; and Riedl, M
work page 2019
Show all 22 references
-
[9]
arXiv preprint arXiv:1901.06417
Friend, collaborator, student, manager: How de- sign of an ai-driven game level editor affects creators. arXiv preprint arXiv:1901.06417. [Ha and Eck 2017] Ha, D., and Eck, D
1901 arXiv
-
[10]
arXiv preprint arXiv:1704.03477
A neu- ral representation of sketch drawings. arXiv preprint arXiv:1704.03477. [Jacob et al. 2013] Jacob, M.; Coisne, G.; Gupta, A.; Sysoev, I.; Verma, G. G.; and Magerko, B
2013 arXiv
-
[11]
In Ninth Artificial Intelligence and Interactive Digital Enter- tainment Conference
Viewpoints ai. In Ninth Artificial Intelligence and Interactive Digital Enter- tainment Conference. [Johnson et al. 2009] Johnson, G.; Gross, M. D.; Hong, J.; Do, E. Y .-L.; et al
2009
-
[12]
Foundations and Trends R⃝ in Human–Computer Interaction 2(1):1–93
Computational support for sketching in design: a review. Foundations and Trends R⃝ in Human–Computer Interaction 2(1):1–93. [Jongejan et al. 2016] Jongejan, J.; Rowley, H.; Kawashima, T.; Kim, J.; and Fox-Gieg, N
2016
-
[13]
Mount View, CA, accessed Feb17:2018
The quick, draw!-ai experiment. Mount View, CA, accessed Feb17:2018. [Karimi et al. 2018a] Karimi, P.; Davis, N.; Grace, K.; and Maher, M. L. 2018a. Deep learning for identifying poten- tial conceptual shifts for co-creative drawing.arXiv preprint arXiv:1801.00723. [Karimi et ...
2018 arXiv
-
[14]
nature 521(7553):436
Deep learning. nature 521(7553):436. [Mikolov 2016] Mikolov, T
2016
-
[15]
https://code.google.com/p/word2vec
word2vec: Tool for computing continuous distributed representations of words. https://code.google.com/p/word2vec. [Oh et al. 2018] Oh, C.; Song, J.; Choi, J.; Kim, S.; Lee, S.; and Suh, B
2018
-
[16]
In Proceedings of the 2018 CHI Confer- ence on Human Factors in Computing Systems,
I lead, you help but only with enough de- tails: Understanding user experience of co-creation with ar- tificial intelligence. In Proceedings of the 2018 CHI Confer- ence on Human Factors in Computing Systems,
2018
-
[17]
[Purcell and Gero 1996] Purcell, A
ACM. [Purcell and Gero 1996] Purcell, A. T., and Gero, J. S
1996
-
[18]
Design studies 17(4):363–383
Design and other types of fixation. Design studies 17(4):363–383. [Shneiderman 2007] Shneiderman, B
2007
-
[19]
Commu- nications of the ACM 50(12):20–32
Creativity sup- port tools: Accelerating discovery and innovation. Commu- nications of the ACM 50(12):20–32. [Simonyan and Zisserman 2014] Simonyan, K., and Zisser- man, A
2014
-
[20]
arXiv preprint arXiv:1409.1556
Very deep convolutional networks for large- scale image recognition. arXiv preprint arXiv:1409.1556. [Suwa and Tversky 1997] Suwa, M., and Tversky, B
1997 arXiv
-
[21]
[V oigt, Niehaves, and Becker 2012] V oigt, M.; Niehaves, B.; and Becker, J
2012
-
[22]
In International Conference on Design Science Research in Information Systems, 152–173
Towards a unified design theory for creativity support systems. In International Conference on Design Science Research in Information Systems, 152–173. Springer. [Yannakakis, Liapis, and Alexopoulos 2014] Yannakakis, G. N.; Liapis, A.; and Alexopoulos, C
2014
Reviewed May 25, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.