{"id":"27aaedb1-8ae3-46fc-8003-9b741a002163","arxiv_id":"2412.16839","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A human-guided image-generation tool with contrastive multi-modal projection and sample-level prompt feedback lifted classification accuracy from 48.45% to 81.80% in a 10-class pet case study.","lead":"This paper presents DataCrafter, a visual tool that lets people guide image generation by exploring generated images and giving feedback on individual samples rather than editing text prompts. Using it to expand a 1,000-image pet dataset raised a classifier's accuracy by about 33 points, with potential for rare-object or small-data vision tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evolutionary prompt refinement's proxy objectives (Eqs. 8-9) are never validated against downstream accuracy, leaving the method's core benefit unsupported beyond one anecdotal case.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the proxy objectives in Eqs. (8)-(9) are the mechanism by which user feedback is converted into improved prompts, but the paper never shows that optimizing these CLIP-space proxies improves downstream model accuracy. This is the most load-bearing concern because every headline result—the 81.80% classification accuracy and the detection mAP gain—flows through this refinement step. If the proxy is uncorrelated with the downstream task, the method's advantage over automatic expansion could disappear entirely, even though the M2M projection may still be a fine visualization tool. The case study is inherently confounded: the expert's own judgments, the increased image count, and the specific class-name corrections could explain the gains independent of the objective. Other concerns, such as the absence of error bars or the fact that the theoretical results only show an impossibility for MFM rather than a guarantee for M2M, are real but secondary; they affect the strength of supporting evidence rather than the central causal mechanism. A conditional verdict is appropriate because the concern is addressable with a focused experiment, not a fundamental inconsistency. The proposed test isolates the contribution of the proxy objective, directly settling whether the refinement mechanism earns its keep.","tokens_in":20957,"tokens_out":6578,"duration_ms":60966,"concrete_test":"Run a controlled experiment on 3-5 classes or datasets where user feedback (the specific deleted/selected image sets from the case study, or generated by a script) is held fixed, and compare downstream test accuracy of a classifier trained on equal numbers of images generated by: (a) the full evolutionary prompt refinement with the Eq. (8)-(9) proxy objective, (b) random LLM prompt mutations with no proxy-based selection, and (c) the original unmodified prompt. If (a) does not substantially and consistently beat (b) and (c), the reported improvement cannot be credited to the proxy objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The sample-level prompt refinement (Sec. 5.3.1) is the only algorithmic bridge between user feedback and the generated training images, yet its selection criterion is a proxy defined in CLIP space: for deletion, Eq. (8) maximizes similarity of proxy images to remaining images and zero-shot confidence; for addition, Eq. (9) maximizes diversity, similarity to selected images, and confidence. No experiment verifies that this proxy correlates with the actual downstream objective, i.e., test accuracy of a ResNet50 trained on the expanded dataset. The case study confounds the proxy with human expertise: E1 inspected images, chose which to delete/select, and accepted or rejected recommended prompts, so the 81.80% accuracy cannot be attributed to the objective. Without an ablation or separate evaluation of the prompt-refinement optimizer, the method's central claim—that sample-level feedback yields better prompts than automatic expansion—rests on a single anecdote. Moreover, the proxy is evaluated on a small number of stochastic proxy images, so the selected prompt may simply overfit noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DataCrafter, an interactive visual analytics system for expanding small image datasets with a text-to-image generative model under human guidance. It introduces two technical components: M2M, a contrastive-learning-based multi-modal projection method for jointly embedding images and content labels in 2D, and a sample-level prompt refinement method that uses an evolutionary algorithm to turn user feedback (deleting undesired images or requesting more images like selected ones) into revised prompts. The evaluation consists of a quantitative comparison of M2M against single-modal and multi-modal projection baselines on Pets and COCO, a single-expert case study on a 10-class Pets subset showing classification accuracy improving from 48.45% to 81.80%, a comparison with the automatic GIF expansion method, and an extension to object detection. The paper claims that M2M preserves intra- and inter-modal relationships better than existing projection methods and that sample-level feedback makes dataset expansion more controllable and effective.","tokens_in":21100,"tokens_out":4471,"duration_ms":39329,"significance":"If the claims are substantiated, DataCrafter is a useful contribution to interactive dataset generation: it addresses the real problem of expanding small training sets and lowers the barrier to prompt refinement by letting users give image-level feedback. The M2M formulation, especially the frequency-biased negative sampling for contrastive loss, is a sensible adaptation of contrastive learning to multi-modal projection, and the authors provide source code and data, which aids reproducibility. The reported case-study gain (33.35 percentage points) is large and the qualitative finding that the class name 'Bengal' invited tiger-like generations is compelling. However, the significance is currently conditional: the theoretical analysis does not actually guarantee the superiority of the contrastive objective, the projection evaluation lacks statistical reliability, and the benefit of the prompt-refinement algorithm is not isolated from the human expert's inspection and decisions.","major_comments":[{"comment":"The sample-level prompt refinement selects mutated prompts by maximizing proxy objectives defined in CLIP space (similarity to remaining/selected images, diversity, and zero-shot confidence) evaluated on a small number of stochastic proxy images. No experiment validates that optimizing these proxies improves the actual downstream objective, namely the accuracy of a ResNet50 trained on the expanded dataset. The case study in Sec. 6.2 confounds the proxy with E1's own inspection and prompt acceptance/rejection decisions, so the 81.80% accuracy cannot be attributed to the objective in Eqs. (8)-(9). Without an ablation that isolates the prompt-refinement optimizer (e.g., comparing prompts recommended by the objective against random mutations for the same feedback, or measuring correlation between the proxy score and downstream accuracy), the central claim that sample-level feedback yields better prompts rests on a single anecdote.","section":"Sec. 5.3.1, Eqs. (8)-(9)"},{"comment":"The theoretical analysis shows only that MFM's exact multi-modal distance-order loss cannot always be satisfied in the many-to-many setting when n > 3. It does not provide any guarantee, approximation bound, or theoretical characterization for the proposed contrastive objective in Eqs. (5)-(6). The sentence in Sec. 6.1, 'This validates the correctness of our theoretical analysis and demonstrates the effectiveness of contrastive learning in multi-modal projection,' overstates the logical link, since the theorems neither predict nor explain the empirical success of M2M. The contribution statement that the projection method is 'theoretically and experimentally better than the existing methods' is therefore too strong; the authors should either add a theorem or explicit theoretical justification for contrastive loss in this setting or revise the claim to empirical superiority only.","section":"Sec. 5.2.1, Theorems 1-2 and Lemma 3"},{"comment":"All quantitative results in Table 1 are single point estimates without error bars, number of runs, or significance tests. Because M2M is trained with stochastic mini-batch sampling and frequency-biased negative sampling, the observed margins over MFM (e.g., IMS 0.7311 vs. 0.5052 on Pets) could vary across random seeds. The paper claims that M2M 'consistently outperforms' baselines, but with one run per method the consistency cannot be verified. The authors should report means and standard deviations over at least three random seeds and, where feasible, a paired significance test for the difference against MFM.","section":"Sec. 6.1, Table 1"},{"comment":"The comparison with the automatic expansion method GIF is not a controlled test of the human-guided process. The row 'Dataset expansion with human guidance' uses prompts that were modified by E1 during the interactive case study, so the 5% improvement over GIF reflects the quality of the human-curated prompts, not the ability of DataCrafter's sample-level prompt refinement to produce them automatically. Additionally, the case study involved only one expert (E1), who also participated in the requirement analysis, so the improvement may not generalize to other users. An evaluation with multiple users, or at least a separate test in which the prompt-refinement algorithm receives the same sample-level feedback without human inspection of intermediate results, is needed to support the claim that the tool itself provides an advantage over automatic expansion.","section":"Sec. 6.3, Table 2"}],"minor_comments":[{"comment":"The sentence 'Both E3 and R4 are actively involved in deploying data expansion methods' contains a typo: 'R4' should be 'E4'.","section":"Sec. 4"},{"comment":"The phrase 'we develop a conservative learning-based multi-modal projection method' appears to be a typo for 'contrastive learning-based'; please correct it.","section":"Sec. 5.2.1"},{"comment":"In Eq. (4), the distance functions h(·, ·) and l(·, ·) are described only in the surrounding prose; they should be defined formally in the equation or immediately before it for readability.","section":"Sec. 5.2.1, Eq. (4)"},{"comment":"The CMMD distance in Eq. (3) includes a Gaussian kernel scale σ, but the choice of σ is never stated; please specify the value or the heuristic used for setting it.","section":"Sec. 5.1, Eq. (3)"},{"comment":"In the object detection extension, the pseudo-labels are generated by the detector trained on the original images and then used as ground truth for retraining; this self-training protocol may yield optimistic mAP estimates. Please report the quality of the pseudo-labels (e.g., precision on a labeled subset) or use a held-out validation set to confirm the improvement.","section":"Sec. 6.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of IEEE TVCG as a visual analytics contribution, and the availability of source code is a strength. The main concern is that the empirical evaluation is not yet strong enough for the central claims: the prompt-refinement proxy is unvalidated, the projection comparison lacks error bars, and the case study is a single-expert anecdote. These issues are fixable with additional experiments, so I recommend major revision rather than rejection. I also note that the theoretical contribution is modest (an impossibility result for MFM) and should be framed accordingly; the authors should not claim a theoretical guarantee for their own method without one."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the M2M projection is a genuine contribution and the quantitative evaluation supports it; the sample-level prompt refinement is clever but under-validated, and that's where the stress-test note lands squarely.\n\nWhat's actually new: the M2M contrastive projection for many-to-many multimodal embedding spaces, plus the sample-level evolutionary prompt refinement. The theoretical analysis of MFM's distance-order loss is a nice sanity check, even though it shows an impossibility for MFM rather than a guarantee for M2M. The contrastive framework with frequency-biased negative sampling is a sensible design choice, and the improvement over MFM in inter-modal measures (Tab. 1) is large enough to be convincing even without error bars. The IMS metric is a reasonable addition.\n\nThe tool itself is thoughtfully designed, and the case study with E1 is well told. The discovery of the 'Bengal tiger' vs 'Bengal cat' issue is a compelling illustration of why human oversight matters.\n\nNow the soft spots. First, Tab. 1 has no error bars or significance tests. The margins are wide, so I'm not too worried, but a reader should note that. Second, the theory only says MFM's exact-order loss is impossible in many-to-many; it doesn't prove M2M's contrastive objective is the best choice. The empirical results carry that weight, and they mostly do. Third, and most importantly, the prompt-refinement proxy objectives (Eqs. 8-9) are never validated against the actual downstream objective. The case study confounds the proxy with E1's expertise: he inspected images, decided which to delete, and accepted/rejected recommendations. We don't know whether the evolutionary optimizer's ranking correlates with test accuracy. The stress-test note is right that a prompt could score well on the proxy yet fail to help the classifier. This is the weakest link in the paper. It's fixable with a small ablation or a correlation study, but as written the core claim about sample-level feedback rests on a single anecdote.\n\nOn the positive side, the paper is honest about its limitations (Sec. 7.2), and the code is promised. No circularity: the projection is trained on the same CLIP embeddings it is evaluated on, which is standard; downstream accuracy is measured on held-out data.\n\nBottom line: this is a solid systems paper. The M2M contribution alone justifies peer review. The prompt-refinement evidence needs strengthening. I'd send it to a serious referee with the expectation of a major revision focusing on validating the proxy, not a desk reject.","headline":"M2M projection is a real contribution; the prompt-refinement proxy is the weak link and the paper would benefit from a controlled validation.","tokens_in":21697,"tokens_out":2798,"would_cite":true,"duration_ms":23870,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human-guided dataset expansion, in which users explore generated and original images together and give feedback on individual samples, lifts classification accuracy from 48.45% to 81.80% and detection mAP from 92.4% to 94.5%, the paper…","keywords":["dataset expansion","human-guided image generation","multi-modal projection","contrastive learning","prompt refinement","visual analytics","small-scale datasets","Stable Diffusion"],"falsifier":"Generate two batches of images for the same class with equal proxy scores, one from DataCrafter's refined prompts and one from arbitrary prompts that happen to match those scores; if the high-scoring batch does not train a better downstream classifier than the low-scoring one, the refinement objective is not the cause of the reported gains. A simpler check: take the proxy images a refined prompt scores highest on and train a model on them alone; if accuracy falls below training on images the proxy scores lowest on, the load-bearing premise fails.","tokens_in":20701,"feed_emoji":"🎨","tokens_out":9215,"duration_ms":73366,"temperature":0.7,"pith_summary":"The paper's central claim is that the limiting factor in generative dataset expansion is control, not generation: automatically produced images are insufficiently diverse and contain undesired content, and users cannot easily repair them by editing prompts. To make expansion controllable, the paper builds DataCrafter, a visual analytics workflow in which a contrastive multi-modal projection places images and their content labels in a shared 2D view, and sample-level feedback--deleting undesired images or selecting images for more variety--drives an evolutionary prompt refiner that recommends better prompts. In a ten-class pet-image case study, the workflow raises a classifier's accuracy from 48.45% to 81.80%, about five percentage points above an automatic expansion baseline, and raises object-detection mAP from 92.4% to 94.5% when the expanded set is used to retrain a detector. A sympathetic reader should care because small labelled datasets are a practical bottleneck, and the paper offers a concrete way to put a human in the loop without requiring prompt-engineering skill.","feed_headline":"Human-guided image generation lifts classification to 82%","feed_subtitle":"Flagging bad synthetic images and refining prompts beats fully automatic expansion by roughly five points.","key_machinery":"Two coupled mechanisms carry the argument. The first is M2M, a six-layer feed-forward projection network trained to minimize a weighted sum of three contrastive losses--image-to-image, image-to-label, and label-to-label--with frequency-biased negative sampling that samples absent labels by their frequency to prevent rare labels from collapsing. It produces the shared scatterplot in which images and content labels are explored. The second is sample-level prompt refinement, an evolutionary algorithm that asks an LLM to mutate a prompt, generates a few proxy images with the mutated prompt, scores them against the user's feedback (away from deleted images and close to kept ones, or diverse and close to selected ones, with high confidence), and iterates until the score stops improving. The projection supplies the evidence a user needs to give feedback; the refinement supplies the mechanism that turns feedback into new prompts.","core_discovery":"On the paper's own terms, the central discovery is that many-to-many multi-modal projection--the common situation where one image carries several content labels--cannot in general preserve exact distance orders between images and labels. The paper proves the number of achievable orders is at most $n(n-1)(n^2-n+2)/8+1$, which for $n>3$ is less than $n!$; hence no single planar mapping can satisfy the distance-order loss used by the existing MFM method, and that loss fails exactly in the setting that matters for dataset exploration. The replacement is M2M, a contrastive projection that keeps similar images together, similar labels together, and images near their labels. Embedded in DataCrafter, this projection lets a user spot that the class name 'Bengal' is generating cat-tiger hybrids, remove those images with one click, and accept a refined prompt that fixes the class name; combined with a diversity-increasing refinement step, the final expanded dataset lifts classification accuracy from 48.45% to 81.80% and improves detection mAP from 92.4% to 94.5%.","pith_inferences":["The paper leaves untested whether the proxy objectives in the refinement loop (similarity, diversity, confidence on a few proxy images) are what cause the downstream accuracy gain; a controlled study that optimizes the same objectives without human feedback would isolate the mechanism.","The frequency-biased negative sampling idea is a generally useful trick for contrastive learning on long-tailed label distributions, beyond the projection setting in which it appears.","The paper's own limitation section (Sec. 7.2) notes that feedback is currently single-set, that weaker generators for other image types may hurt quality, and that user efficiency was not measured; these caveats bound, but do not undo, the reported gains.","A natural extension the paper mentions but does not develop is feedback across multiple sets of generated images, such as fusing styles of two sets; this could further increase diversity."],"forward_implications":["Small labelled datasets can be expanded with generative models in a controlled way, and the resulting classifiers beat fully automatic expansion on the same budget.","The many-to-many projection result implies that any future multi-modal projection method for multi-label image collections must handle the impossibility of exact distance-order preservation, and contrastive objectives are a viable replacement.","Users who cannot write good prompts can still steer generation by pointing at images, because the sample-level refinement loop converts deletions and selections into prompt changes.","The same expanded dataset transfers to a second downstream task (object detection), so the benefit is not specific to one classifier architecture."],"supporting_citations":[{"why":"Provides the GIF automatic expansion baseline that human-guided expansion is compared against, and supplies the informativeness metric used in the metric visualization.","marker":"[65]"},{"why":"Introduces MFM, the many-to-one multi-modal projection whose distance-order loss M2M is designed to replace in the many-to-many setting.","marker":"[63]"},{"why":"Inspires the use of multi-modal language models to extract descriptive content labels for efficient image exploration.","marker":"[32]"},{"why":"Stable Diffusion is the pre-trained text-to-image generator used for latent perturbation and image generation.","marker":"[47]"},{"why":"CLIP supplies the shared embedding space in which images and content labels are compared and the zero-shot predictions used for informativeness.","marker":"[45]"},{"why":"GPT-4 generates image captions/content labels and performs the prompt mutation step in the evolutionary refinement loop.","marker":"[43]"},{"why":"Supplies the connection between evolutionary algorithms and large language models that the sample-level prompt refinement builds on.","marker":"[17]"},{"why":"Provides the multi-task learning method used to set the weights among the three contrastive losses in M2M.","marker":"[37]"}],"fun_headline_variants":["Human-guided image generation beats auto expansion by 5 points","New projection method makes AI image expansion more controllable","Flag bad synthetic images, refine prompts, lift accuracy to 82%","Human-in-the-loop dataset expansion: 81.8% accuracy from 48.45%","Many-to-many projection flaw fixed: better dataset expansion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop assumes that a prompt scoring well on the proxy objectives--similarity to desired images, diversity, and confidence of its generated proxy images--will actually produce images that improve the downstream model; the paper does not validate that correlation.","fun_headline_variants_meta":{"raw":{"variants":["Human-guided image generation beats auto expansion by 5 points","New projection method makes AI image expansion more controllable","Flag bad synthetic images, refine prompts, lift accuracy to 82%","Human-in-the-loop dataset expansion: 81.8% accuracy from 48.45%","Many-to-many projection flaw fixed: better dataset expansion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1222,"prompt_tokens":961,"completion_tokens":261,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":185}},"tokens_in":577,"tokens_out":261,"duration_ms":2677,"temperature":1.0,"reasoning_tokens":185,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:14:56.155991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate two batches of images for the same class with equal proxy scores, one from DataCrafter's refined prompts and one from arbitrary prompts that happen to match those scores; if the high-scoring batch does not train a better downstream classifier than the low-scoring one, the refinement objective is not the cause of the reported gains. A simpler check: take the proxy images a refined prompt scores highest on and train a model on them alone; if accuracy falls below training on images the proxy scores lowest on, the load-bearing premise fails.","supporting_citations":[{"cited_title":"Zhang, D","cited_arxiv_id":null,"evidence_quote":"Provides the GIF automatic expansion baseline that human-guided expansion is compared against, and supplies the informativeness metric used in the metric visualization."},{"cited_title":"ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map","cited_arxiv_id":"2407.12315","evidence_quote":"Introduces MFM, the many-to-one multi-modal projection whose distance-order loss M2M is designed to replace in the many-to-many setting."},{"cited_title":"Radford, J","cited_arxiv_id":null,"evidence_quote":"CLIP supplies the shared embedding space in which images and content labels are compared and the zero-shot predictions used for informativeness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the connection between evolutionary algorithms and large language models that the sample-level prompt refinement builds on."}],"review_version":1}