{"id":"a0d3838f-9901-41c1-84c9-161d6f2a8577","arxiv_id":"2411.17673","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SketchAgent uses a multimodal LLM prompted with a numbered-grid sketching language to generate, edit, and collaboratively draw sequential vector sketches without any training.","lead":"SketchAgent lets an off-the-shelf multimodal AI model draw sketches stroke by stroke using a simple grid-based coordinate language, with no training or fine-tuning. It can generate drawings from text prompts, edit them through chat, and co-create sketches with human users on a shared canvas.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed breadth of text-to-sketch capability is not established: per-category CLIP results and the paper's own limitations show many concepts generate unrecognizable sketches.","rationale":"The reader's weakest_assumption identifies the underlying text-to-coordinate mapping as the fragile core, and the paper's Sec. 7 confirms that this mapping often fails. My concern is the consequence of that fragility for the specific claim of 'wide range': the evaluation evidence is not sufficient to establish breadth. The reader's verdict of CONDITIONAL already accounts for the limitations; my stress-test reinforces the need to scope the breadth claim but does not reveal a fatal flaw. The method has genuine positive results (e.g., human-like appearance in 2AFC, successful collaboration on selected concepts), and the paper is honest about limitations. Therefore, I recommend keeping the verdict unchanged rather than rejecting. The concrete test would quantify the breadth and could justify a more precise claim, but it is a refinement of the existing conditional acceptance rather than a reason to overturn it.","tokens_in":36063,"tokens_out":11589,"duration_ms":105644,"concrete_test":"Re-analyze the existing 500-sketch CLIP results from Sec. 5.1 by computing the per-category Top-1 accuracy across the 10 samples per category, then report the fraction of the 50 categories with accuracy clearly above chance (e.g., >20%, since 10 samples give a 2% chance baseline). If fewer than half of the categories exceed this threshold, the 'wide range' claim is not supported by the paper's own data. As a complementary check, run the released pipeline on a random sample of 100 concepts spanning abstract, scene, and multi-object prompts and have human raters judge whether each sketch is recognizable, reporting the fraction of concepts with at least one recognizable sketch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sec. 1) that SketchAgent 'can generate sketches across a wide range of textual concepts' is supported mainly by an average CLIP Top-1 of 0.23 over 50 QuickDraw categories (Sec. 5.1). The average hides that many categories are rarely or never recognized: the supplementary confusion matrix (Fig. 21) shows categories such as shark, octopus, and snake are frequently confused with visually similar classes, and Fig. 23 lists many classes with only 20–30% recognition. The paper's own Sec. 7 admits the agent 'often produces rich textual descriptions of object parts but struggles to convert these into effective sketching actions, resulting in overly abstract and unrecognizable outputs,' and Fig. 14 shows failures on unicorn, Frida Kahlo, and letters/numbers. The qualitative 'beyond categories' results (Fig. 8) are selected by the authors, and the supplementary's randomly chosen concepts include failures (Statue of Liberty, photosynthesis, pie chart). Thus the quantitative and qualitative evidence supports generation for a moderate set of simple iconic objects, not the claimed 'wide range' of textual concepts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SketchAgent, a training-free method for sequential sketch generation using a frozen multimodal large language model (Claude3.5-Sonnet). The agent is prompted with a grid-coordinate sketching language, in-context examples, and chain-of-thought instructions, and it outputs a sequence of strokes that are fitted to Bézier curves and rendered on a canvas. The paper claims that this approach supports text-conditioned sketch generation across a wide range of concepts, enables stroke-by-stroke sequential sketching with semantic stroke annotations, facilitates real-time human-agent collaborative sketching, and supports iterative chat-based editing. Evaluations include CLIP zero-shot classification on 500 sketches across 50 QuickDraw categories, a 2AFC human study comparing sketch human-likeness, a collaborative user study with 30 participants, an ablation study, and a small chat-editing study on 54 sketches. The supplementary provides additional qualitative results and detailed prompts.","tokens_in":36281,"tokens_out":4503,"duration_ms":44839,"significance":"If the central claims hold, the paper demonstrates a meaningful new capability: an off-the-shelf multimodal LLM, without any fine-tuning, can produce ordered, semantically labeled strokes that render into recognizable sketches and can participate in interactive drawing sessions. The method is simple, reproducible, and the authors commit to releasing code. The paper includes useful ablations and honest discussion of limitations, and the human studies add credibility beyond CLIP-based proxies. However, the strongest claim—\"wide range of textual concepts\"—is only partially supported by the evidence, and the editing and collaborative evaluations have important protocol gaps. The work is likely to be of interest to the sketch-generation and human-AI interaction communities, but the conclusions should be scaled to match the demonstrated scope.","major_comments":[{"comment":"The claim in Sec. 1 that SketchAgent \"can generate sketches across a wide range of textual concepts\" is not established by the quantitative evidence. The average Top-1 recognition of 0.23 in Table 1 hides substantial per-category variability: the supplementary confusion matrix and top-recognized-class figures show many categories with recognition rates of 20–30% or lower, and the paper itself (Sec. 7, Fig. 14) acknowledges failures on unicorn, Frida Kahlo, and letters/numbers. The appendix also admits that some randomly selected concepts (Statue of Liberty, photosynthesis, pie chart) were unsuccessful. The claim should be narrowed to simple iconic objects, or the evaluation should be strengthened with per-category success thresholds or human recognition judgments on the full concept set.","section":"Sec. 5.1, Table 1, Supp. Figs. 21, 23"},{"comment":"The editing evaluation reports that SketchAgent \"correctly follows instructions 92% of the time\" on 54 sketches, but the manuscript does not define what constitutes correct instruction-following, who made the judgment, how many raters were involved, or whether there was inter-rater agreement. Because interactive editing is one of the paper's three core contributions, this metric needs a transparent evaluation protocol to be load-bearing. The current description is insufficient for readers to assess the reliability of the 92% figure.","section":"Sec. 5.4, Chat-Based Sketch Editing"},{"comment":"The collaborative user study uses 8 concepts that were explicitly \"selected based on the agent's demonstrated ability to draw them independently\" (Sec. 5.3). This selection makes the study unsuitable for supporting the broader claim that SketchAgent can collaborate on arbitrary concepts. Additionally, the analysis that agent-only and user-only strokes have low CLIP recognition rates (Table 3) may conflate incompleteness with lack of contribution, since partial sketches are not necessarily expected to be recognizable. A human evaluation of partial sketches or a study with less favorably selected concepts would be needed to substantiate the collaboration claim.","section":"Sec. 5.3, Human-Agent Collaborative Sketching"},{"comment":"The paper repeatedly emphasizes that SketchAgent captures the \"dynamic, evolving process\" of sketching and \"incorporates visual feedback\" (Sec. 1). However, during a single sketch-generation turn, the model produces the entire coordinate sequence in one pass; the canvas is fed back only for later editing or collaborative turns. The sequential nature is primarily in the output representation and stroke ordering, not in the model's internal generation process. This distinction should be clearly stated, as the current wording overstates the mechanism.","section":"Sec. 4, Method Overview"}],"minor_comments":[{"comment":"There is a typo: \"We demonstrate SketchAgent's capabil to generate\" should read \"capability\".","section":"Sec. 5, opening paragraph"},{"comment":"The ablation claims that \"all components contribute to the agent's full performance,\" but the difference between the full pipeline and the w/o System Prompt condition (0.23 vs 0.20 Top-1) is within the reported error bars. The authors should either report significance tests or temper the claim for the system-prompt component.","section":"Sec. 6, Table 2"},{"comment":"The table's last row label \"Vis.\" is not explained in the text; a short caption indicating that it shows example sketches would improve clarity.","section":"Sec. 5.1, Table 1"},{"comment":"The editing prompt in the appendix says the model should \"Describe the location of the added concepts first in <thinking> tags,\" but for the animals category the instructions (e.g., \"Add a hat\") contain no location; the paper should clarify whether the model is expected to infer placement in those cases and how that inference was scored.","section":"Sec. 5.4 and Supp. B.4"},{"comment":"The discussion of direct SVG prompting in Fig. 3 is useful, but there is no direct quantitative comparison to optimization-based sketch generation methods such as CLIPasso or DiffSketch on the same QuickDraw subset; adding such a comparison would help calibrate the reported numbers against prior art.","section":"Sec. 2, Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a simple, appealing idea with honest limitations, and the supplementary material is thorough. The main gap between the broad claims (\"wide range of concepts\") and the actual evidence (moderate success on simple iconic objects, selected collaborative concepts, and a small editing study) is the primary reason I recommend major revision rather than acceptance. The authors should be encouraged to either scale down their conclusions or scale up their evaluations. Also note that the prompts were tuned on the default backbone; the paper discloses this, but it remains a limitation for cross-model generality."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read SketchAgent. The core idea is a good one: a sketching language built from grid coordinates, per-point t-values, and least-squares Bezier fitting that lets a frozen multimodal LLM draw stroke by stroke, then edit and collaborate through chat. No training, no fine-tuning. That representation is genuinely new, and the resulting sketches look more like freehand drawings than SVG prompts (Fig 3B vs 3C). The experiments are decent: CLIP recognition against a human baseline, ablations showing each component matters, a real two-alternative forced choice human study, and a collaboration study with 30 participants. The limitations section is honest—it shows failures on unicorns, human figures, letters and numbers, and admits the agent can produce rich language but weak strokes.\n\nThe main soft spot is the headline claim. The paper says 'wide range of textual concepts' (Sec 1). The evidence doesn't support that. The average CLIP Top-1 of 0.23 over 50 QuickDraw categories is fine, but the per-category numbers are all over the place: shark is confused with fish, octopus with spider, snake with squiggle; the confusion matrix and Fig 23 show many classes at 20–30%. The supplementary's random selection includes obvious misses (Statue of Liberty, photosynthesis, pie chart). So it's not 'wide range'—it's a moderate set of simple, iconic objects, with graceful degradation elsewhere. That's still a contribution, but the claim should be tamped down.\n\nSecond issue: the collaboration study picks 8 concepts based on the agent's demonstrated ability to sketch them independently. That's selection on the agent's strengths, so the collaborative results (0.75 recognition) don't generalize to arbitrary concepts. The editing eval is only 54 sketches and doesn't break down by concept; plausible but thin.\n\nNone of this is fatal. The paper is structurally sound, the method is reproducible in principle, and the authors disclose the important limitations themselves. The claim overreach is fixable with revised text and a per-category breakdown. I'd send this to peer review and insist on those changes.\n\nFor you: worth reading for the representation idea and the honest evaluation; not a breakthrough, but a solid systems paper in the sketch/HCI space. I'd bring it to a reading group.","headline":"A genuinely new sketching representation for frozen multimodal LLMs, with honest experiments—but the 'wide range' claim is overstated; the evidence supports a moderate set of simple iconic concepts.","tokens_in":36771,"tokens_out":2588,"would_cite":true,"duration_ms":23408,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen multimodal LLM can draw sketches stroke-by-stroke using only a numbered grid and a few examples.","keywords":["sketch generation","multimodal large language models","sequential drawing","in-context learning","chain-of-thought prompting","human-AI collaboration","chat-based editing","Bézier curve fitting"],"falsifier":"Re-run the 500-sketch, 50-category CLIP benchmark with the identical prompts and backbone but with the grid numbers erased from the canvas image (cells left blank), and compare Top-1 accuracy against the reported 0.23; if accuracy does not drop substantially, the numbered-grid spatial scaffolding is not the mechanism that makes the method work.","tokens_in":35912,"feed_emoji":"✏️","tokens_out":7256,"duration_ms":60431,"temperature":0.7,"pith_summary":"The paper claims that an off-the-shelf multimodal large language model, with no training or fine-tuning, can be turned into a general-purpose sketch generator that draws stroke by stroke. The key design is a string-based sketching language on a numbered grid canvas: the model outputs coordinates and timing values as text, and a parser fits smooth Bézier curves to those points and renders them. Because strokes are produced sequentially, the same agent can edit its own sketch through chat and take turns with a human on a shared canvas. If the claim holds, the bottleneck for a general-purpose sketching agent is not sketch data or generative training but a well-designed interface on top of existing model priors. On the paper's metrics, the agent approaches human-level sketch recognizability under a CLIP classifier and is judged more human-like than direct SVG prompting in a two-alternative forced-choice study.","feed_headline":"Frozen LLM draws sketches stroke by stroke on a numbered grid","feed_subtitle":"No training or fine-tuning; a chat prompt and examples turn a multimodal model into an editable sketch partner.","key_machinery":"The load-bearing object is the sketching language over a numbered grid canvas. The canvas is a 50×50 grid whose cells carry labels like x2y8, giving the text-only LLM a concrete coordinate system to reference; the paper includes a demonstration that a multimodal LLM which can name the right line to draw nevertheless fails to execute it with pixel coordinates, motivating the grid. A stroke is written as a list of grid coordinates plus a matching list of t-values, meaning the points are treated as samples along a curve rather than control points, and the system fits a cubic Bézier curve to them with a least-squares solve, splitting recursively when the fit error is large. In-context learning teaches the format and chain-of-thought prompts make the model plan stroke order in <thinking> tags; a stopping token </s{j}> allows a human to add strokes to the same canvas, which are converted back into the coordinate format by sampling the user's Bézier curve at its t-values.","core_discovery":"SketchAgent establishes that a frozen multimodal LLM, guided only by a system prompt, a user prompt with one worked example, and a numbered 50×50 grid canvas, can generate recognizable sketches of arbitrary textual concepts—including landmarks, scientific principles, and diagrams far beyond the 345 QuickDraw categories that bound trained sketch models. The model plans in <thinking> tags, emits strokes as coordinate sequences with t-values, and the system fits cubic Bézier curves to the sampled points by least squares, recursively splitting long curves. The same representation supports the whole interaction loop: the rendered canvas is fed back for chat-based edits, and a stopping token lets a human interleave their own strokes, which are re-sampled into the agent's coordinate format. Quantitative results on 500 sketches across 50 categories give CLIP zero-shot Top-1/Top-5 accuracy of 0.23/0.44 with the default backbone, approaching the 0.27/0.49 of human QuickDraw sketches under the same metric, and an ablation shows that removing the system prompt, chain-of-thought, or the complete in-context example each degrades accuracy.","pith_inferences":["Editorial inference: the failure cases the paper lists (unicorn, human figures, letters and numbers) outline the boundary of the backbone's latent spatial competence; a natural testable prediction is that these failure modes shrink as multimodal LLMs improve, without any change to the sketching pipeline.","Editorial inference: the numbered-grid protocol is a general recipe for eliciting spatial output from text-only models, so the same idea could be tested for diagram layout, floor-plan drafting, or GUI wireframes, where a grid plus in-context examples might replace fine-tuning.","Editorial inference: the 92% editing accuracy was measured on a small hand-picked set; a broader stress test that varies object combinations and demands precise relative placement would clarify whether the agent reasons spatially or falls back on stereotyped layouts.","Editorial inference: because the prompts were tuned on the default backbone, the reported gap between backbone models partly reflects prompt fit rather than raw capability, so the method's portability across models is probably understated."],"forward_implications":["A general-purpose sketching agent can be assembled without collecting human drawing data or training a generative model; the capability is inherited from the backbone LLM, so future improvements in those models should transfer directly to sketch quality.","Sketch editing becomes a natural conversational operation: because the canvas is part of the dialogue state, the agent can add, relocate, or annotate parts of an existing drawing in response to text, with 92% of tested editing prompts followed correctly.","The stroke-by-stroke output carries semantic labels assigned by the model, so sketches come with part-level annotations as a byproduct, useful for analysis and dataset construction.","Real-time collaborative sketching with a human partner is feasible: individual strokes take about 8 seconds in collaborative mode and a complete sketch about 20 seconds, matching the pace of human drawing.","The approach is largely backbone-agnostic: it works with several closed commercial models and, with lower recognition scores, with a large open-weight model, suggesting the interface is portable rather than tied to one model."],"supporting_citations":[{"why":"Supplies the default backbone multimodal LLM whose priors and sequential text generation enable drawing without training.","marker":"[3]"},{"why":"Provides the 50-category evaluation set and the human-drawn baseline for recognition and human-likeness comparisons.","marker":"[54]"},{"why":"The zero-shot CLIP classifier used to measure sketch recognizability throughout the experiments.","marker":"[88]"},{"why":"Introduced in-context learning, the mechanism that teaches the agent the sketching language from examples.","marker":"[9]"},{"why":"Introduced chain-of-thought prompting, used in the thinking tags to plan stroke order and placement.","marker":"[119]"},{"why":"SketchRNN provides the sequential sketch baseline that motivates the method and marks its category-boundary contrast.","marker":"[48]"},{"why":"A comparison backbone and the model in the spatial-reasoning failure example that motivates the numbered canvas.","marker":"[84]"},{"why":"An optimization-based text-to-sketch method whose simultaneous-stroke limitation this work targets.","marker":"[116]"}],"fun_headline_variants":["Frozen LLM sketches any concept stroke-by-stroke","No-fine-tune LLM draws with you in a chat","Grid-guided LLM draws beyond QuickDraw limits","Chat-driven sketching from an untrained multimodal LLM","Stroke-by-stroke sketching: no training, just prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an off-the-shelf multimodal LLM can convert its semantic understanding of an object into a spatially coherent, ordered sequence of grid coordinates, given only a numbered canvas and a few in-context examples; the paper itself acknowledges that the agent often writes rich textual descriptions of parts yet struggles to turn them into effective drawing actions.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLM sketches any concept stroke-by-stroke","No-fine-tune LLM draws with you in a chat","Grid-guided LLM draws beyond QuickDraw limits","Chat-driven sketching from an untrained multimodal LLM","Stroke-by-stroke sketching: no training, just prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1426,"prompt_tokens":961,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":385}},"tokens_in":577,"tokens_out":465,"duration_ms":4910,"temperature":1.0,"reasoning_tokens":385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:50:19.577378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 500-sketch, 50-category CLIP benchmark with the identical prompts and backbone but with the grid numbers erased from the canvas image (cells left blank), and compare Top-1 accuracy against the reported 0.23; if accuracy does not drop substantially, the numbered-grid spatial scaffolding is not the mechanism that makes the method work.","supporting_citations":[{"cited_title":"Emergent Graphical Conventions in a Visual Communication Game","cited_arxiv_id":"2111.14210","evidence_quote":"The zero-shot CLIP classifier used to measure sketch recognizability throughout the experiments."},{"cited_title":"ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention","cited_arxiv_id":"2311.16682","evidence_quote":"Introduced chain-of-thought prompting, used in the thinking tags to plan stroke order and placement."},{"cited_title":"I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence","cited_arxiv_id":null,"evidence_quote":"A comparison backbone and the model in the spatial-reasoning failure example that motivates the numbered canvas."},{"cited_title":"Drawing to reason and learn in science","cited_arxiv_id":null,"evidence_quote":"An optimization-based text-to-sketch method whose simultaneous-stroke limitation this work targets."}],"review_version":1}