{"id":"408a964a-e77f-4502-a648-28933d808c52","arxiv_id":"2506.20449","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Med-Art combines VLM-generated captions, LoRA fine-tuning of PixArt-alpha, and a pixel-space color-statistics loss to generate realistic 2D medical images from limited data.","lead":"Med-Art fine-tunes a large pre-trained text-to-image model on small medical image datasets by first using a vision-language model to write detailed captions for endoscopy and skin-lesion images. It adds a pixel-level color loss during training and reports better FID, KID, and downstream classification scores than Stable Diffusion and PixArt-alpha baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation prompts are generated from training images, so FID and classification gains may reflect memorization rather than generative generalization; held-out-prompt evaluation is required.","rationale":"The reader's verdict is CONDITIONAL, and I agree with the category, but for a more pointed reason than the VLM hallucination concern. The explicit statement that evaluation text prompts are generated from a random subset of training images (Section 3) makes the headline numbers in Tables 2 and 4 vulnerable to a memorization artifact. With ~8k (Kvasir) or ~6.7k (skin) training examples, 10-15 epochs, LoRA on DiT and T5, and unique detailed captions per image, the model has enough capacity to memorize caption-image associations; HLDF's Eq. (4) further ties generation to the exact training image's color statistics. Under this protocol, low FID/KID and good downstream classification can be achieved by an image-retrieval model and therefore do not by themselves establish that Med-Art generates new, clinically useful images from unseen text. I am not asserting that memorization definitely occurred; adversarial tests such as held-out-prompt FID and image-retrieval distance would settle it. The paper deserves credit for sensible ablations (Table 3) and a clear engineering contribution, but the strongest claim requires generalization evidence. I therefore keep the CONDITIONAL verdict and propose the specific re-evaluation as the acceptance condition.","tokens_in":8582,"tokens_out":12870,"duration_ms":141845,"concrete_test":"Re-run the Table 2 evaluation with VSG captions generated from the held-out test images (or from class-level prompts never paired with images during training), keeping all other hyperparameters and generation settings identical. Compare the resulting FID/KFD/KID to Table 2; additionally, for each generated image, compute its nearest-neighbor LPIPS/SSIM distance to the training set. If FID degrades substantially (e.g., >20% relative) or if generated images are near-duplicates (LPIPS < 0.1) of their caption-source training images, the reported state-of-the-art claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing concern is that the quantitative evaluation may measure memorization rather than text-to-image generalization. Section 3 Experimental Setup states: 'A random subset of training images, matching the test set size (1600 for Kvasir and 1665 for skin lesions), is used to generate text prompts for test image synthesis.' The prompts used to compute FID/KFD/KID (Table 2) and to synthesize the classification training sets (Table 4) are therefore produced by VSG from images in the training split, the very images on which Med-Art was trained. Because LoRA is applied to both DiT and T5 on only ~8k or ~6.7k images, the model can memorize the unique caption-to-image mapping for the training examples. The HLDF color loss (Eq. 4) actively pushes the generated image's channel means and standard deviations to match the original training image, reinforcing per-image copying rather than learning a conditional distribution over novel prompts. If the model reproduces training images, low FID and good downstream ViT performance are unsurprising and do not show that Med-Art can generate new, clinically useful images from unseen medical text. The baseline comparisons remain fair in a relative sense, but the central claim that the method adapts large text-to-image models to small medical datasets requires generalization beyond memorized caption-image pairs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Med-Art, a framework for medical text-to-image generation on small datasets. It uses LLaVA-Next as a Visual Symptom Generator to produce detailed captions from images, applies LoRA fine-tuning to both the PixArt-alpha Diffusion Transformer and the T5 text encoder, and introduces Hybrid-Level Diffusion Fine-tuning (HLDF) with a pixel-space color loss computed on images sampled during training. Experiments on the Kvasir endoscopy dataset and a skin lesion dataset report improved FID, KFD, and KID over several text-to-image baselines, as well as improved downstream classification performance when a ViT is trained on the generated images. The paper includes ablations showing the contribution of each module and presents qualitative comparisons.","tokens_in":8871,"tokens_out":6842,"duration_ms":70446,"significance":"The paper addresses two practical bottlenecks in medical text-to-image generation: the scarcity of paired medical text and the color oversaturation that arises when fine-tuning natural-image models. The proposed pipeline is technically coherent, and the ablations in Table 3 support the individual role of the VLM captions, LoRA on T5, and HLDF. If the generalization concern is resolved, the approach would be a practical recipe for adapting large text-to-image models to small medical datasets. The public project page, the release of code, and the inclusion of domain-specific FID variants (KFD, HFD) are strengths. The main weakness is that the evaluation protocol does not yet establish generalization to unseen prompts.","major_comments":[{"comment":"The evaluation protocol described in Section 3 states that 'A random subset of training images, matching the test set size (1600 for Kvasir and 1665 for skin lesions), is used to generate text prompts for test image synthesis.' Because these prompts are derived from the same training images on which Med-Art is fine-tuned, the FID/KFD/KID results in Table 2 and the downstream classification results in Table 4 can be explained by the model reproducing training image-caption pairs rather than by genuine text-to-image generalization. The row for Ovis in Table 3 does not remedy this, since Ovis captions are also generated from training images. The authors should add an evaluation with prompts generated from held-out images, or with class-level prompts not paired with any training image during fine-tuning, and report those metrics.","section":"§3 Experimental Setup"},{"comment":"The color loss in Eq. (4) penalizes the difference in per-channel mean and standard deviation between the generated image and the specific original training image x, while the prompt for that generated image is derived from the same x via the Visual Symptom Generator. On small training sets (about 8k and 6.7k images), this per-sample consistency loss is likely to reinforce memorization of individual training images. The authors should either add an experiment that isolates the effect of HLDF on held-out prompts, or discuss why this loss does not encourage trivial copying rather than learning a conditional distribution.","section":"§2.2 Eq. (4)"},{"comment":"All quantitative results are reported without error bars or multiple seeds. Differences such as the KFD of 1.21 versus 1.26 and the KID of 0.012 versus 0.018 between Med-Art and the variant without LoRA on T5 are small enough to fall within typical run-to-run variability for these metrics, especially without a fixed seed. Reporting mean and standard deviation over at least three seeds for the main comparisons would strengthen the claims in Table 3, including the comparison with DoRA.","section":"Tables 2–3"}],"minor_comments":[{"comment":"The claim that Med-Art achieves the best overall classification performance is too strong, since Table 4 shows SD1.4 has the highest BACC and SD1.5 has the highest AUC on the skin lesion dataset; the text should be softened to 'best on most metrics' or supported by a multi-metric aggregation.","section":"Table 2 and Table 4"},{"comment":"The word 'datastet' in the experimental setup paragraph is a typographical error and should be corrected to 'dataset'.","section":"§3 Experimental Setup"},{"comment":"The paper does not assess the clinical accuracy or diversity of the VSG captions; a small human evaluation or an automated caption-image alignment check would make the assumption about LLaVA-Next's medical proficiency explicit and would strengthen the credibility of the generated descriptions.","section":"§2.1 Visual Symptom Generator"},{"comment":"The generated images are shown at a size that makes it hard to evaluate texture and color differences; including zoomed crops or a quantitative color-statistics comparison would help the reader judge the qualitative improvement.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The memorization concern raised in the stress-test is the central issue. If the authors can show that Med-Art generalizes to held-out prompts, the contribution is significant and the paper would be a strong fit for the journal. The manuscript is otherwise well-organized, the method is clearly described, and the public code availability is a plus. I recommend major revision rather than rejection, because the evaluation gap is addressable within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what you should know: this paper is a decent empirical recipe, but its headline claim is undercut by an evaluation that only tests memorization, not generalization. The new piece is the combination of VLM-written captions, LoRA on both PixArt-alpha and T5, and a pixel-space color-statistics loss (HLDF) applied every N steps during fine-tuning. The HLDF loss is the most original part. The ablations in Table 3 are honest: removing any module hurts, and the added training cost is modest (6-9%). The baseline list is solid, including Hunyuan-DiT, DoRA, and a second VLM (Ovis), which is good practice.\n\nThe soft spot is in Section 3: they generate the evaluation prompts from a random subset of the training images. So the text-to-image model is tested on captions of images it has already seen, and the HLDF loss actively reinforces per-image color statistics. Low FID/KFD/KID and good synthetic-training classification are then exactly what you would expect from memorization. This does not break the relative comparisons against baselines, which are tested on the same prompts, but it does break the claim that Med-Art \"adapts\" large T2I models to small datasets in the sense of generating from novel clinical text. The fix is straightforward: evaluate on prompts from held-out images or from human-written clinical descriptions, and check nearest-neighbor distances to the training set.\n\nSmaller issues: no error bars or repeated runs, so the metric differences may not be significant; the skin-lesion \"best\" claim is overstated because Fast-DiT beats them on HFD and SD1.4/SD1.5 beat them on BACC/AUC; and the VLM captioning is not compared against other captioning pipelines (the Ovis check helps). No code is released yet.\n\nBottom line: this is a useful engineering paper for people fine-tuning text-to-image models on small medical datasets. The memorization problem is real but addressable, not fatal. It deserves a serious referee, with a request for held-out-prompt evaluation and error bars before publication.","headline":"Solid fine-tuning recipe undercut by an evaluation that measures memorization rather than text-to-image generalization.","tokens_in":9380,"tokens_out":3460,"would_cite":true,"duration_ms":36088,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that auto-written VLM captions plus LoRA on PixArt-α and a pixel-level color loss let small medical datasets produce state-of-the-art text-to-image generation.","keywords":["text-to-image generation","medical image synthesis","diffusion transformer (DiT)","visual symptom generation","LoRA fine-tuning","color consistency loss","Kvasir endoscopy","skin lesion dermoscopy"],"falsifier":"Replace the VSG captions with clinician-written captions for the same images and retrain under identical settings; if FID, KID, and downstream ViT classification improve or stay equal, the VLM captions are not the limiting factor, whereas if they change substantially, the caption generator is the active ingredient. Alternatively, have expert gastroenterologists and dermatologists label the VSG captions as correct, partially correct, or hallucinated, and check whether the model's per-class generation quality tracks caption correctness.","tokens_in":8407,"feed_emoji":"🩺","tokens_out":4503,"duration_ms":43384,"temperature":0.7,"pith_summary":"The paper claims that large text-to-image diffusion transformers can be adapted to small medical datasets when the missing medical text is replaced by captions auto-written by a vision-language model. On two small datasets, endoscopic images from Kvasir and dermoscopic skin lesion images, Med-Art reports the best Fréchet and kernel distances among all compared baselines, including Stable Diffusion and the base PixArt-α model. It also reports that classifiers trained on Med-Art's synthetic images reach the highest overall accuracy on real test images, suggesting the generated images preserve clinically relevant structure. The practical claim is that a pipeline of VLM captioning, LoRA on both the DiT and the text encoder, plus a pixel-level color loss is enough to make text-to-image generation clinically useful under data constraints.","feed_headline":"VLM captions let diffusion models master small medical datasets","feed_subtitle":"Fine-tuning PixArt-α on auto-written symptom descriptions improves image fidelity and downstream classification.","key_machinery":"Three components carry the argument. The Visual Symptom Generator (VSG) uses a vision-language model (LLaVA-Next) with a structured prompt to produce natural-language descriptions of each medical image, then T5 simplifies captions longer than the 120-token limit. LoRA is applied to both the DiT denoising network and the T5 text encoder, so medical semantics enters through updated text embeddings as well as cross-attention. HLDF (Hybrid-Level Diffusion Fine-tuning) generates images during training using classifier-free guidance and DPM-Solver++ (about 20 steps) and adds a color loss comparing per-channel means and standard deviations of generated and real pixels; gradient checkpointing and an every-N-steps schedule keep the added cost to roughly 6-9% training time.","core_discovery":"Med-Art's central discovery is that the language gap, not just the image gap, is what prevents pretrained text-to-image models from generating medical images. By prompting LLaVA-Next to describe the visual symptoms of each image in natural language, the authors obtain image-conditioned captions that match the kind of text the base model saw during pretraining, turning a class-label-only setup into a true text-to-image setup. They then LoRA-fine-tune PixArt-α, including its T5 text encoder, and add Hybrid-Level Diffusion Fine-tuning, which runs DPM-Solver++ during training to synthesize images and penalizes per-channel mean and standard deviation differences between generated and real images. The result is state-of-the-art FID, KFD, and KID on Kvasir and best overall FID/KID on skin lesions, with generated images that train a ViT classifier to higher performance than images from any compared baseline.","pith_inferences":["Because the caption generator is the only source of medical knowledge in the pipeline, the ceiling on clinical usefulness is set by LLaVA-Next's accuracy; a clinician-audited caption set or a medical-tuned VLM should push FID and downstream classification further, and this can be tested directly.","The same recipe likely transfers to other low-text medical modalities, such as histopathology, retinal fundus photography, or ultrasound, where class labels exist but free-text reports are scarce; a quick probe would fine-tune Med-Art on one such dataset and check KID plus a downstream classifier.","Training classifiers on these synthetic images could serve as a privacy-preserving data-sharing strategy, since synthetic images avoid direct patient-identifiable content, though whether that holds is a separate clinical and legal question.","The interval-based HLDF schedule (every 500 or 1500 steps) suggests the color loss mainly corrects a persistent bias rather than tracking every detail, so more frequent or adaptive scheduling may not improve results and could be wasteful."],"forward_implications":["If the claims hold, medical text-to-image generation no longer requires curated radiology reports or expert annotations: auto-generated visual symptom captions suffice to fine-tune a strong pretrained model.","The HLDF color loss removes the oversaturated-color artifact that plagues latent diffusion fine-tuned on small medical sets, without adding sampling-time cost.","LoRA on the text encoder is a measurable part of the gain: ablations show that removing it raises FID from 51.99 to 63.10 on Kvasir, and even without it Med-Art beats Hunyuan-DiT.","Generated images from Med-Art train a ViT to a higher F1, balanced accuracy, and AUC on Kvasir than any baseline's synthetic images, and remain the best overall on skin lesions when all three metrics are considered.","The method carries over to a different caption generator (Ovis) with similar results, so it is not tied to one specific VLM."],"supporting_citations":[{"why":"Supplies the Visual Symptom Generator that writes the natural-language captions used to fine-tune the model.","marker":"[16]"},{"why":"The base Diffusion Transformer text-to-image model that Med-Art adapts with LoRA.","marker":"[6]"},{"why":"The low-rank adaptation applied to both DiT and T5, the parameter-efficient fine-tuning backbone.","marker":"[13]"},{"why":"Fast sampler that lets the model generate images during training, enabling the pixel-level color loss.","marker":"[21]"},{"why":"The classifier-free guidance technique used when sampling images inside the HLDF training loop.","marker":"[12]"},{"why":"One of the two evaluation datasets, the Kvasir endoscopic image collection with eight classes.","marker":"[25]"},{"why":"The dermoscopy skin lesion dataset used for FID/HFD and classification tests.","marker":"[9]"},{"why":"Serves as PixArt-α's text encoder, simplifies overlength captions, and receives LoRA to improve medical text embeddings.","marker":"[27]"}],"fun_headline_variants":["VLM captions enable medical text-to-image with small data","Synthetic VLM captions help diffusion models conquer small medical datasets","Fine-tuning PixArt-α on VLM captions sets SOTA for medical imaging","Medical diffusion models get a language boost from VLM-crafted captions","Hybrid fine-tuning with VLM captions improves medical image generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes that LLaVA-Next, a general vision-language model that was not trained on endoscopy or dermoscopy, produces clinically accurate descriptions of the lesions rather than plausible but wrong ones; if the captions misdescribe what is in the image, the model learns the wrong image-text associations and the reported gains would not transfer to real clinical use.","fun_headline_variants_meta":{"raw":{"variants":["VLM captions enable medical text-to-image with small data","Synthetic VLM captions help diffusion models conquer small medical datasets","Fine-tuning PixArt-α on VLM captions sets SOTA for medical imaging","Medical diffusion models get a language boost from VLM-crafted captions","Hybrid fine-tuning with VLM captions improves medical image generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1344,"prompt_tokens":883,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":499,"tokens_out":461,"duration_ms":4806,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:47:39.178035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the VSG captions with clinician-written captions for the same images and retrain under identical settings; if FID, KID, and downstream ViT classification improve or stay equal, the VLM captions are not the limiting factor, whereas if they change substantially, the caption generator is the active ingredient. Alternatively, have expert gastroenterologists and dermatologists label the VSG captions as correct, partially correct, or hallucinated, and check whether the model's per-class generation quality tracks caption correctness.","supporting_citations":[{"cited_title":"URL https://llava-vl","cited_arxiv_id":null,"evidence_quote":"Supplies the Visual Symptom Generator that writes the natural-language captions used to fine-tune the model."},{"cited_title":"In: International Conference on Learning Representations (ICLR) (2024)","cited_arxiv_id":null,"evidence_quote":"The base Diffusion Transformer text-to-image model that Med-Art adapts with LoRA."},{"cited_title":"In: Proceedings of the 8th ACM on Multimedia Systems Conference","cited_arxiv_id":null,"evidence_quote":"One of the two evaluation datasets, the Kvasir endoscopic image collection with eight classes."},{"cited_title":"In: International Workshop on Applications of Medical AI","cited_arxiv_id":null,"evidence_quote":"The dermoscopy skin lesion dataset used for FID/HFD and classification tests."},{"cited_title":"Journal of machine learning research21(140), 1–67 (2020) Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation 11","cited_arxiv_id":null,"evidence_quote":"Serves as PixArt-α's text encoder, simplifies overlength captions, and receives LoRA to improve medical text embeddings."}],"review_version":1}