{"id":"ce46d864-2e77-441f-9382-8060e4ecd775","arxiv_id":"2505.10551","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Feasibility of synthetic images has little effect on fine-tuned CLIP accuracy; the edited attribute (background, color, or texture) matters more than whether the attribute is realistic.","lead":"This paper tests whether training images that could never exist in reality (like a purple plane) hurt image classifiers compared with realistic images. It builds a pipeline to create matched real and edited image pairs and finds feasibility barely changes CLIP accuracy, while the type of edit matters more.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The null result is not yet established: Δ1 gaps below 0.3% come from single runs, and Table 2 admits 19.1% of 'infeasible' texture images are judged feasible, so contamination and run-to-run noise could erase a real feasibility effect.","rationale":"The paper's headline claim is a null result: feasibility has no clear impact on LoRA-fine-tuned CLIP classifiers. Two conditions must hold for that claim to be trustworthy: the feasible and infeasible training sets must actually differ only in feasibility, and the evaluation must be sensitive enough to detect meaningful differences if they exist. The weakest point is exactly the intersection of these conditions. First, the human evaluation in Table 2 shows that the infeasible texture condition is labeled feasible 19.1% of the time, meaning the supposedly infeasible set contains a non-negligible fraction of in-distribution examples. This is not an external criticism; the paper itself reports this in Sec. 4.3.1 and Supp. G. Second, the entire comparison rests on one training run per condition, so there is no way to know whether a 0.3% gap is real or noise. The paper's own reporting inconsistency in Sec. 4.2.1 (5/9 stated as 78%) reinforces that the quantitative claims are not being handled with sufficient care. I do not think this warrants rejection: the VariReal pipeline is a useful contribution, the distributional analysis in Table 3 and Figure 5 supports the intended feasibility contrast, and the background-modification results are more robust. The correct stance is to treat the feasibility null as a well-motivated hypothesis with conditional support, exactly as the reader did. The concrete test above would either confirm the null with proper statistical and contamination control or reveal that the observed null is an artifact.","tokens_in":21085,"tokens_out":4213,"duration_ms":44137,"concrete_test":"Compute a seed-robust, contamination-filtered Δ1: for each of the 9 dataset-by-attribute conditions, run the LoRA fine-tuning with at least 10 random seeds and, in parallel, restrict the infeasible training set to images confirmed infeasible by a full human annotation pass (removing the 19.1% of texture images judged feasible). If the resulting 95% confidence intervals for Δ1 exclude differences larger than 0.3% and the contamination-filtered Δ1 remains within 0.3%, the claim survives; if either condition shifts beyond 0.3%, the original null result is an artifact of noise or mislabeled infeasible data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Observation 1: feasibility shows no clear impact) rests on Δ1 = F−IF values in Table 1 that are mostly below 0.3%, each from a single LoRA fine-tuning run with no reported error bars or seed variation. Small point estimates are not evidence of no effect when seed-to-seed variability for CLIP fine-tuning can easily exceed 0.3%. The operationalization problem compounds this: human evaluation (Table 2) shows infeasible texture images are judged feasible 19.1% of the time, and Supplement G attributes this to texture edits that only change color or produce implausible object-background combinations. If a substantial fraction of the 'infeasible' training set is actually feasible, any true feasibility effect is systematically attenuated toward zero. The paper also contains a reporting inconsistency in Sec. 4.2.1: it states that 78% of Δ1 values (5/9) remain within 0.3% after adding real data, but 5/9 is 56%, which further weakens confidence in the quantitative claim. Without seed-level variance estimates or a contamination-filtered analysis, the null conclusion is conditional rather than established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VariReal, a diffusion-based minimal-edit pipeline that creates feasible and infeasible variants of real images along background, color, and texture attributes, and evaluates them on three fine-grained datasets (Oxford Pets, FGVC Aircraft, Stanford Cars) plus WaterBirds for background. CLIP models are LoRA-fine-tuned under synthetic-only and real+synthetic training, comparing feasible, infeasible, and mixed conditions. The main empirical claim is that feasibility has no clear impact on top-1 accuracy (Observation 1), that background edits help while color/texture edits are less effective (Observation 2), and that strictly feasible synthetic data is not necessary (Observation 3). The paper includes human evaluation, MLLM-based filtering, distributional metrics, scaling experiments, and ablations supporting the dataset-generation pipeline.","tokens_in":21361,"tokens_out":4322,"duration_ms":38727,"significance":"If the null result survives proper statistical treatment, the paper makes a useful and somewhat counterintuitive contribution: it challenges the common assumption that infeasible synthetic images are harmful and suggests that synthetic-data pipelines need not strictly enforce feasibility. The VariReal pipeline is a concrete, reusable tool for controlled attribute editing, with code released, and the study design includes human evaluation, automatic filtering, distributional analyses, and scaling/ablation experiments, which are strengths. However, the central evidence consists of small accuracy differences without variance estimates or significance testing, so the contribution is currently conditional on additional statistical validation.","major_comments":[{"comment":"The central claim that feasibility 'shows no clear impact' rests on point estimates from a single LoRA fine-tuning run per condition; no seed variation, confidence intervals, or significance tests are reported. Most Δ1 values are below 0.3%, but per-cell gaps reach 3.2% (Cars, texture, synthetic-only), and the paper provides no estimate of run-to-run variability for this training protocol. Without repeated seeds or an equivalence test with a pre-specified margin, the absence of a visible gap cannot be distinguished from a real effect obscured by noise. Please add multi-seed results with variance reporting and a formal or semi-formal test of the no-effect claim.","section":"§4.2.1, Table 1, Observation 1"},{"comment":"The paragraph reporting the distribution of Δ1 values contains an internal arithmetic error: after incorporating real data, the text states that '78% of Δ1 values (5/9) remain within 0.3%,' but 5/9 is 56%, not 78%. The sentence reports both 78% and the fraction 5/9, so the quantitative summary is internally inconsistent. This needs correction and a re-audit of all percentages and counts in the section.","section":"§4.2.1"},{"comment":"The operationalization of feasibility is not clean enough to support a strong null conclusion. Table 2 reports 80.9% feasibility correctness for infeasible texture images, so 19.1% of that training set is judged feasible by human evaluators; Supp. G attributes this to textures that only convey color and to implausible object-background combinations. If a substantial share of 'infeasible' data is actually in-distribution, any true feasibility effect is attenuated toward zero. Additionally, Supp. Table 5 shows that feasible and infeasible conditions use different generation hyperparameters (guidance scales, strengths, mask dilation/alpha factors), so the comparison varies in more than the target attribute. I request a contamination-filtered analysis on the human-verified subsets and either matched generation hyperparameters or a sensitivity analysis showing robustness to these choices.","section":"§3.2.3, Table 2, Supp. G, Supp. Table 5"}],"minor_comments":[{"comment":"The caption contains a typo: 'Standford-Cars' should be 'Stanford Cars'.","section":"Figure 4 caption"},{"comment":"The caption begins with 'TQualitative results', which appears to be missing a leading word or contains a typo; please correct it.","section":"Supp. Figure 13 caption"},{"comment":"The notation ISyn is introduced in the task formulation but not defined at first use; please define it explicitly.","section":"§3.1"},{"comment":"The formula for Δ2 is written as 'Mix− F+IF/2', which is ambiguous; it should be written as Mix − (F + IF)/2.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked about the feasibility paper (2505.10551). The short version: the paper builds a genuinely useful tool, VariReal, for generating controlled feasible/infeasible minimal edits, and it reports an interesting empirical claim — feasibility of synthetic training images barely moves LoRA-fine-tuned CLIP accuracy. I think the tool is worth having, and the claim is plausible, but as presented the null result is not yet established.\n\nWhat's actually new: VariReal combines inpainting with ControlNet and real priors to isolate background, color, or texture edits while keeping everything else fixed, and it generates matched feasible/infeasible prompt pairs per class. That's a real contribution. The paper then shows background edits help even when infeasible (which loosens ALIA's feasibility constraint), while foreground color/texture edits are less helpful, and mixing feasible and infeasible data doesn't hurt. Those observations are useful and mostly well supported by the tables. They also include a human evaluation of feasibility and naturalness, plus FID/CLIP/DINO/LPIPS distribution analysis. That is more than most synthetic-data papers do.\n\nThe soft spots are in the central quantitative claim. Observation 1 rests on Δ1 differences mostly below 0.3%, but each number comes from a single LoRA run with no error bars or seeds. For CLIP fine-tuning, seed-to-seed variation can easily exceed 0.3%, so the small point estimates don't prove 'no effect.' Compounding that, the infeasible condition is contaminated: Table 2 shows 19.1% of infeasible texture images were judged feasible by human annotators, which would attenuate any true feasibility effect toward zero. That said, the direction is consistent, and the per-cell exceptions (e.g., Cars texture Δ1 = +3.2%) get discussed. The paper also has a typo in Sec. 4.2.1 — it says 78% of Δ1 values (5/9) but 5/9 is 56%; the 78% matches 7/9, so it's a typo, not a substantive error. One Δ2 in Table 1 (Cars texture, +3.0 vs ~2.5) also looks off by rounding. These are minor but should be cleaned up.\n\nScope is narrow: three fine-grained datasets, three attributes, one CLIP architecture, one fine-tuning method. That's fine for an initial study, but it means the negative result shouldn't be over-generalized.\n\nOverall: worth sending to peer review. The VariReal pipeline deserves to be published, and the empirical observations are useful even if the null is framed as 'no large effect' rather than 'no effect.' I'd encourage the authors to add seed variance or confidence intervals and to re-run the analysis excluding images that fail human feasibility checks. If you're doing synthetic augmentation work, this is a relevant citation and a good starting point for a discussion.","headline":"A useful controlled-generation pipeline and a plausible but not yet proven null result — feasibility barely matters for CLIP classification, but single runs and contaminated infeasible sets keep the claim conditional.","tokens_in":21860,"tokens_out":4794,"would_cite":true,"duration_ms":40153,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feasibility of synthetic training images barely changes CLIP accuracy","keywords":["feasibility","synthetic training data","CLIP","LoRA","diffusion models","image editing","fine-grained classification","data augmentation"],"falsifier":"Train LoRA-fine-tuned CLIP on feasible and infeasible synthetic sets that have been censored by a human panel so every image is verified as correctly labeled, across many random seeds. If the top-1 accuracy gap between the two sets consistently exceeds 1% on any of the three attributes—or if the paper's below-0.3% differences fail to replicate under this stricter label control—then the claim that feasibility has no clear impact is falsified.","tokens_in":20896,"feed_emoji":"🖼️","tokens_out":7136,"duration_ms":63109,"temperature":0.7,"pith_summary":"The paper asks whether generated training images must be 'feasible'—able to plausibly occur in the real world—to be useful for training a classifier. It answers no: when a CLIP classifier is fine-tuned with LoRA on synthetic images that differ from real photos only in one edited attribute, swapping feasible attributes for infeasible ones moves top-1 accuracy by less than 0.3 percent in most settings. The attribute edited matters more than feasibility: background changes improve accuracy whether feasible or infeasible, while color and texture changes help less and degrade beyond small scales. Mixed feasible/infeasible training sets perform about as well as pure sets. If the result holds, synthetic-data pipelines need not spend effort enforcing realism for every image.","feed_headline":"Feasibility of synthetic training images barely changes CLIP accuracy","feed_subtitle":"LoRA-finetuned CLIP classifiers move less than 0.3% when synthetic images swap feasible for infeasible attributes.","key_machinery":"The VariReal pipeline generates minimal-change feasible/infeasible image pairs from real images. It uses GPT-4 with in-context examples to produce class-specific feasible and infeasible attribute prompts (background, color, texture), then performs prior-guided diffusion editing: inpainting with a real prior and mask dilation for backgrounds, and a two-stage inpainting-plus-ControlNet with Canny-edge conditioning for foreground color and texture edits. An MLLM (Llava-Next) filters outputs by attribute and feasibility. Performance is measured by LoRA fine-tuning of CLIP, computing $\\Delta_1 = F - IF$ (feasible minus infeasible top-1 accuracy) and $\\Delta_2 = \\text{Mix} - \\frac{F+IF}{2}$ (mixing gain over the average of pure sets). The mechanism that carries the argument is this pairing: because each real image is edited in only one attribute and every image is reused across all prompts, feasible and infeasible training sets differ only in the target attribute, isolating feasibility from other image content.","core_discovery":"The central claim, stated on the paper's own terms, is that feasibility has no clear impact on classification performance for LoRA-fine-tuned CLIP models. Across Oxford Pets, FGVC-Aircraft, and Stanford Cars, most differences $\\Delta_1 = F - IF$ are below 0.3%, with the largest being a 1.8% gain for feasible backgrounds on Aircraft and a 3.2% gain for feasible textures on Cars; the direction is not consistent. The main corollary is that infeasible synthetic images can be used interchangeably with feasible ones: generation need not enforce feasibility, and mixing the two types does not hurt. The paper's second claim is that the attribute type matters: background edits consistently beat real-only training even when infeasible, whereas color and texture edits are less effective and may degrade performance at scale.","pith_inferences":["Editorial inference: the paper's null result is probably a lower bound on feasibility effects: up to 19.1% of 'infeasible' texture images passed human inspection as feasible, so a cleaner operationalization of infeasibility could either shrink the gap further or reveal a genuine effect.","Editorial inference: the pattern that foreground edits underperform despite being distributionally closer to real data suggests classifiers rely on class-relevant foreground cues; feasibility research should separate 'distributional closeness' from 'task-relevance'.","Editorial inference: because the evidence comes from CLIP with LoRA and object-centric datasets, the conclusion may not transfer to other architectures, tasks like detection or segmentation, or safety-critical domains; testing with a convolutional baseline or a medical or autonomous-driving dataset would decide.","Editorial inference: the strongest testable extension is to vary one attribute at a time with a hard feasibility guarantee, such as physically impossible but photorealistic combinations verified by a panel, and measure per-seed accuracy gaps to see whether the below-0.3% null holds outside the paper's generator."],"forward_implications":["Synthetic-data pipelines for object-centric classification can stop treating infeasible outputs as waste; both feasible and infeasible images are usable training material.","Background diversity is the more useful axis of augmentation: both feasible and infeasible background edits outperform real-only training, so constraints like ALIA's feasible-background restriction are unnecessary.","Color and texture edits should be used sparingly as augmentation; their benefit peaks at low synthetic-to-real ratios and turns negative at larger scale.","Mixing feasible and infeasible data is safe and can even add complementary signal in synthetic-only training, since the two types push models toward different correct-prediction sets while reaching similar accuracy."],"supporting_citations":[{"why":"ALIA baseline that augments with feasible backgrounds only; the paper's background experiments directly challenge its restriction.","marker":"[14]"},{"why":"DataDream, the few-shot guided dataset generation work whose CLIP fine-tuning and evaluation protocol the experiments follow.","marker":"[28]"},{"why":"CLIP, the vision-language backbone that is fine-tuned and evaluated for classification.","marker":"[47]"},{"why":"LoRA, the low-rank adaptation method used to fine-tune CLIP's encoders.","marker":"[27]"},{"why":"Stable Diffusion, the latent diffusion model underlying VariReal and the editing baselines.","marker":"[51]"},{"why":"ControlNet, the Canny-edge conditioning used to preserve object shape during color and texture edits.","marker":"[61]"},{"why":"GPT-4, the LLM used to generate and self-filter feasible and infeasible attribute prompts.","marker":"[1]"},{"why":"Llava-Next, the MLLM that automatically filters generated images for attribute and feasibility correctness.","marker":"[33]"},{"why":"Repaint, the inpainting model that provides the realism-preserving editing path in VariReal.","marker":"[39]"}],"fun_headline_variants":["Synthetic data realism barely matters for CLIP training","Feasibility of fake images: no big deal for CLIP","Infeasible synthetic images work just as well for CLIP","Why enforce realism? Synthetic data works without it","CLIP ignores synthetic image feasibility, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that 'feasible' and 'infeasible' are cleanly separated: the whole result rests on prompt generation, manual filtering, and per-attribute editing parameters producing truly infeasible images, yet human evaluation labeled up to 19.1% of infeasible texture images as feasible, so if infeasible data is actually in-distribution or the edits also change naturalness or class-relevant cues, the null finding becomes an artifact of the generator rather than a property of feasibility.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data realism barely matters for CLIP training","Feasibility of fake images: no big deal for CLIP","Infeasible synthetic images work just as well for CLIP","Why enforce realism? Synthetic data works without it","CLIP ignores synthetic image feasibility, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000353,"raw_usage":{"total_tokens":1948,"prompt_tokens":997,"completion_tokens":951,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":871}},"tokens_in":613,"tokens_out":951,"duration_ms":6768,"temperature":1.0,"reasoning_tokens":871,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:07:02.099156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train LoRA-fine-tuned CLIP on feasible and infeasible synthetic sets that have been censored by a human panel so every image is verified as correctly labeled, across many random seeds. If the top-1 accuracy gap between the two sets consistently exceeds 1% on any of the three attributes—or if the paper's below-0.3% differences fail to replicate under this stricter label control—then the claim that feasibility has no clear impact is falsified.","supporting_citations":[{"cited_title":"Gonzalez, and Trevor Darrell","cited_arxiv_id":null,"evidence_quote":"ALIA baseline that augments with feasible backgrounds only; the paper's background experiments directly challenge its restriction."},{"cited_title":"DataDream: Few-shot Guided Dataset Generation","cited_arxiv_id":"2407.10910","evidence_quote":"DataDream, the few-shot guided dataset generation work whose CLIP fine-tuning and evaluation protocol the experiments follow."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"CLIP, the vision-language backbone that is fine-tuned and evaluated for classification."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion, the latent diffusion model underlying VariReal and the editing baselines."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"ControlNet, the Canny-edge conditioning used to preserve object shape during color and texture edits."},{"cited_title":"Repaint: Inpainting using denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Repaint, the inpainting model that provides the realism-preserving editing path in VariReal."}],"review_version":1}