{"id":"05a27e68-b923-4649-a6cd-2073374544c1","arxiv_id":"2411.14807","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Pre-training referring expression comprehension models on a fully synthetic, color-varied dataset generated with GLIGEN improves their fine-tuned accuracy on real benchmarks.","lead":"This paper builds Harlequin, a fully synthetic dataset of over a million referring expression queries plus matching images, by swapping color words in existing captions and regenerating the pictures with a text-to-image model. It then shows that pre-training three visual grounding models on this synthetic data improves their accuracy on real benchmarks after fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GLIGEN fidelity is unmeasured: if colors/boxes are not faithfully rendered, Harlequin's supervision is noisy and the reported gains may only reflect larger-scale pre-training.","rationale":"The reader's conditional verdict is appropriate, and the weakest assumption it identifies is the most load-bearing one. The central empirical claim, that pre-training on Harlequin improves downstream REC, is supported by Table 1, but the proposed explanation ('color-driven generation') presupposes that GLIGEN encodes the altered color attribute into the correct region. The paper never measures this. Fig. 3 is by nature selected examples, and Section 5's statement that boxes are correct follows from the conditioning format, not from verification that the rendered object actually appears in the box. If a fidelity audit shows high accuracy, the concern is resolved and the conditional can be lifted. If fidelity is low, the results may simply show that large-scale pre-training on noisy synthetic data helps, which is a qualitatively weaker and less novel claim. I therefore keep the reader's conditional rather than accepting outright. The real-data pre-training baseline and multiple seeds are also worthwhile, but the proposed fidelity audit is the single test that most directly targets the paper's core mechanism.","tokens_in":11737,"tokens_out":8564,"duration_ms":93406,"concrete_test":"Sample 300 Harlequin annotations stratified by the 12 colors. For each, run an open-vocabulary detector (e.g., Grounding DINO) with the noun phrase and check detection inside the annotated box (IoU >= 0.5); then use a color classifier or CLIP text-image matching to check that the predicted dominant color of the detected region equals the altered color token. Report per-color and per-category fidelity, applying the same protocol to the original Flickr30k Entities subset as control. If mean fidelity is below ~70%, the dataset's supervision is too noisy to support the color-driven mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pipeline's value rests on Section 4.2's assumption that GLIGEN renders the altered color on the referred object inside the given box. The only support is Fig. 3, a few qualitative examples, and the Section 5 assertion that generation 'fixes' annotation errors; no automatic fidelity metric is reported. If the color substitution is often ignored (e.g., 'blue dog' remains brown) or the object appears outside the annotated box, then the 1M+ Harlequin annotations are misaligned with the observed pixels. In that case the Table 1 gains could be due to pre-training on a large corpus with approximate layout-language alignment rather than to accurate color-conditioned supervision. Because the paper's stated mechanism is color-driven generation, the fidelity rate is the load-bearing unverified quantity. The missing real-data pre-training baseline and lack of error bars make attribution less certain, but the fidelity check is the decisive missing measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Harlequin, a synthetic dataset for Referring Expression Comprehension (REC) generated from Flickr30k Entities by replacing color adjectives in referring expressions and captions with alternatives sampled from a 12-color vocabulary, then rendering new images with GLIGEN conditioned on the modified text and the original bounding-box layout. The authors pre-train three REC models (TransVG, VLTVG, LGR-NET) on Harlequin and fine-tune them on RefCOCO, RefCOCO+, and RefCOCOg, reporting accuracy gains over from-scratch training in most splits and larger gains on test subsets whose referring expressions contain a color attribute. The paper argues that this demonstrates that fully synthetic, annotation-free data can serve as effective pre-training for REC and that the generated data can correct some human annotation errors.","tokens_in":11951,"tokens_out":5499,"duration_ms":56934,"significance":"If the central claim is corroborated, the work is significant: it offers a scalable, low-cost route to REC training data, a reusable recipe based on color substitution and grounded text-to-image generation, and a concrete dataset (286k images / 1.09M queries) with released code. The paper deserves credit for evaluating three different architectures across three benchmark families and for including a color-subset analysis. However, the paper currently does not measure whether GLIGEN actually renders the substituted color on the referred object inside the provided box, and it lacks both a real-data pre-training control and variance/significance reporting. Those gaps are load-bearing for the attribution of the observed gains to the proposed synthetic generation mechanism.","major_comments":[{"comment":"The pipeline's stated mechanism is color-driven generation, and the quality of the supervision signal depends on GLIGEN faithfully rendering the substituted color on the referred object and placing that object inside the given bounding box. The only evidence offered is the qualitative examples in Fig. 3 and the assertion in Section 5 that generation 'fixes some errors in the human-annotated labels.' The paper should report a quantitative fidelity measurement on a random sample of Harlequin, for example the fraction of generated images in which (a) the referred object's box has at least a threshold IoU with the annotation and (b) the replaced color is recognized by a color classifier or human judgment. Without such a measurement, the 1M+ annotations may be systematically misaligned with the pixels, and the Table 1 gains could be explained by pre-training on a large but noisy corpus rather than by accurate color-conditioned synthetic supervision.","section":"Section 4.2 and Section 5, Fig. 3"},{"comment":"Table 1 compares Synth→Real fine-tuning only against a from-scratch baseline (Real). It does not compare against pre-training on real data, such as Flickr30k Entities, from which Harlequin is derived. Because Harlequin contains about four times as many referring expressions as Flickr30k Entities and the authors pre-train for 60 epochs, the observed improvements (for example +4.11 for LGR-NET on RefCOCOg test) could stem from additional pre-training data or iterations rather than from the synthetic color-driven generation itself. A controlled baseline that pre-trains on Flickr30k Entities with the same schedule, and ideally a matched amount of data, is necessary to attribute the gains to the proposed generation pipeline.","section":"Section 6.3, Table 1"},{"comment":"The paper reports single-run accuracy values, yet several differences between Real and Synth→Real are small (e.g., TransVG −0.10 on RefCOCO+ testB, VLTVG −0.53 on RefCOCOg val, LGR-NET +0.29 on RefCOCO+ val). With no error bars, no repeated-seed experiments, and no significance testing, the conclusion that 'pre-training on artificial data is beneficial' is not yet statistically supported. At minimum, the authors should report mean and standard deviation over at least three seeds per condition and state whether the gains are consistent across seeds.","section":"Section 6.3, Table 1, and Section 6.4, Table 2"}],"minor_comments":[{"comment":"The abstract says the approach 'eliminates manual data collection and annotation,' but the pipeline uses Flickr30k Entities human annotations as seeds for the Annotation Generation Engine. The claim should be qualified to clarify that it eliminates manual annotation of the newly generated synthetic images while still relying on seed annotations from an existing dataset.","section":"Abstract and Section 4"},{"comment":"The paper repeatedly calls Harlequin 'the first totally synthetic generated dataset' for REC. This claim is hard to verify and is not load-bearing for the main contribution. It would be safer to state that Harlequin is, to the authors' knowledge, the first fully synthetic REC dataset of this kind, or to cite any prior synthetic REC datasets explicitly.","section":"Section 5 and Conclusion"},{"comment":"The implementation-details paragraph mixes pre-training, fine-tuning, and baseline training schedules in one long sentence. Separating the three settings into distinct descriptions would improve readability and reproducibility.","section":"Section 6.1"},{"comment":"The sentence 'These results were expected' and the subsequent assertion that the improvement is 'solely guided by learning patterns from data' are not established by the experiments. The authors should either provide an analysis supporting this mechanism or soften the claim.","section":"Section 6.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the dataset/code release is a tangible contribution. My main concern is not the direction of the work but the missing fidelity measurement and the absence of a real-data pre-training control; both are fixable in revision. I would advise the editor that the 'first synthetic dataset' novelty claim should not be the basis for acceptance, since the more durable contribution is the pipeline and the controlled empirical evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know this paper's central result is probably right: pre-training TransVG, VLTVG, and LGR-NET on Harlequin before fine-tuning on RefCOCO family gives consistent gains of 1-4 accuracy points, and the color-subset ablation shows the largest gains on expressions containing color. The first contribution is genuinely new: it generates both the referring expression and the image for REC, where prior work (Pseudo-Q, COPS-REF) only altered the query on existing images. The release of the dataset and code is a plus.\n\nWhat the paper does well: the pipeline is simple and the evaluation is more thorough than the norm. Three models, three benchmarks, plus an ablation targeted at the proposed mechanism. The cross-dataset setting is clean (no overlap between synthetic pre-training and fine-tuning data). The authors also flag the limitation that color swapping can produce unrealistic objects, and they show it still helps.\n\nWhere it is soft: the biggest gap is the missing fidelity measurement. The paper asserts GLIGEN renders the swapped color on the referred object inside the given box, and supports it only with qualitative examples in Fig. 3 and a claim that generation fixes annotation errors. If GLIGEN often ignores the color or places the object outside the box, the synthetic labels are noisy and the measured gains may come mostly from pre-training on a large corpus with approximate layout-language alignment rather than from accurate color-conditioned supervision. The color ablation mitigates this concern somewhat, since gains concentrate on color expressions, but it is not a direct fidelity check.\n\nSecond, there is no real-data pre-training baseline. A comparison against pre-training on the same number of Flickr30k images (or another real dataset) would help attribute the gains to the synthetic generation rather than simply to having more data. Third, no error bars or significance tests; some individual splits show small negative deltas (e.g., VLTVG on RefCOCOg). Finally, the abstract says the approach eliminates manual data collection and annotation, but the pipeline starts from human-annotated Flickr30k Entities, so it reduces rather than eliminates human effort.\n\nThese are all addressable, and none of them breaks the central claim that synthetic data of this kind is useful for REC. Who is it for? Anyone working on visual grounding, dataset generation, or using diffusion models as a data engine. It deserves a serious referee, especially one who asks for the fidelity metric and the real-data baseline.","headline":"A believable synthetic-data result for REC, with an unmeasured generation-fidelity link and no real-data pre-training baseline; the core finding is solid but the mechanism is not fully pinned down.","tokens_in":12429,"tokens_out":2367,"would_cite":true,"duration_ms":24013,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fully synthetic color-swapped training data matches or improves referring-expression models on real benchmarks.","keywords":["synthetic data generation","referring expression comprehension","visual grounding","text-to-image diffusion","color attribute augmentation","GLIGEN","pre-training","Flickr30k Entities"],"falsifier":"Take a random sample of Harlequin images and run an attribute-aware detector to verify, for each query, that the object inside the bounding box carries the swapped color and that the unswitched context objects are still present. If a substantial share of images fails this fidelity check while the fine-tuning gains remain, then accurate rendering is not the active ingredient; if the gains disappear when Harlequin images are replaced by unrelated images paired with the same queries, rendering fidelity is the mechanism.","tokens_in":11577,"feed_emoji":"🎨","tokens_out":7093,"duration_ms":67675,"temperature":0.7,"pith_summary":"The paper tries to establish that Referring Expression Comprehension (REC), locating the object named by a natural-language phrase, can be trained with synthetic data that requires no human annotation. The authors build Harlequin, a dataset of 286,948 images and 1,093,181 queries, by taking color-bearing referring expressions from Flickr30k Entities, swapping the color token for one from a 12-color vocabulary, and rendering a new image with GLIGEN conditioned on the edited caption and box-level entity descriptions. Pre-training TransVG, VLTVG, and LGR-NET on Harlequin before fine-tuning on RefCOCO, RefCOCO+, and RefCOCOg gives accuracy equal to or better than training from scratch on real data, with the largest gains on RefCOCOg for LGR-NET (+4.11 on the test split). If this holds, expensive manual annotation is no longer a hard bottleneck for REC, and synthetic benchmarks of arbitrary size and controlled complexity become feasible.","feed_headline":"Synthetic color-swapped images boost referring-expression models","feed_subtitle":"Pre-training on over a million generated queries matches or beats fine-tuning on real data across three benchmarks.","key_machinery":"The load-bearing mechanism has two parts. The Annotation Generation Engine is a function $\\varphi: A \\to A$ that edits an annotation by replacing the color attribute in the chosen referring expression with a color sampled from a fixed vocabulary, leaving the bounding boxes and the other referring expressions untouched. The Image Generation Engine is a function $\\psi: A \\to I$ implemented with GLIGEN, a grounded text-to-image model that renders an image from the edited caption together with the box-plus-text entities. What makes the pair work is that the textual change is chosen to preserve layout while forcing a visible appearance change, so the synthetic labels attach a discriminative color signal to a known location.","core_discovery":"On the paper's own terms, the discovery is that a single attribute, color, is enough to generate synthetic REC supervision that transfers to real benchmarks. The pipeline treats each seed annotation as a caption plus a set of (referring expression, bounding box) pairs; it replaces the color in one referring expression and the matching caption span, keeps every box and every other expression fixed, and asks GLIGEN to render an image consistent with both the edited caption and the entity boxes. The resulting dataset, Harlequin, contains over one million queries, and pre-training on it improves or matches the from-scratch baselines for all three models and all three benchmark families; on color-only test subsets, TransVG improves by up to +5.15 on RefCOCO val. The authors conclude that the artificial character of the data is overcome when the generative model is controlled tightly, and that fully automatic annotation for REC is achievable.","pith_inferences":["Editorial inference: the same color-swap setup could be used as a controlled diagnostic for whether a REC model leans on color versus object identity, since one scene can be re-rendered many times with only the expression's color changed.","Editorial inference: extending the variation function to attributes that change boxes, such as size or location, could convert this pipeline from appearance-only augmentation into a generator of compositional-reasoning benchmarks.","Editorial inference: a direct test of whether the gains come from accurate synthetic supervision or simply from more data would be to pre-train on Harlequin queries paired with randomly shuffled images and compare against the faithful-image setting."],"forward_implications":["Pre-training on Harlequin works as a drop-in initialization: TransVG, VLTVG, and LGR-NET need no architectural changes to benefit from it.","Because the pipeline is automated, REC training sets of arbitrary size can be produced without human collection or labeling; Harlequin already contains more than a million queries.","Since bounding boxes and non-target expressions are inherited from Flickr30k Entities, the synthetic data keeps spatial semantics such as perspective and object relations while adding color diversity and fresh backgrounds.","The rendering step can repair human annotation errors by generating images in which the referred object actually appears inside its stated box.","Even unnatural combinations such as a blue dog yield usable supervision, indicating that models can treat color as a disambiguating cue rather than relying on memorized color-object co-occurrences."],"supporting_citations":[{"why":"Implements the Image Generation Engine: renders each synthetic image from the edited caption and box-plus-text entities.","marker":"[21]"},{"why":"Supplies the seed annotations (captions, referring expressions, bounding boxes) that the pipeline varies.","marker":"[27]"},{"why":"TransVG is the first evaluated model; pre-training it on Harlequin produces consistent gains on all three benchmarks.","marker":"[7]"},{"why":"VLTVG is the second evaluated model, used to show the pre-training transfers to a verification-style architecture.","marker":"[36]"},{"why":"LGR-NET is the third evaluated model and gives the largest reported gains, especially on RefCOCOg.","marker":"[25]"},{"why":"Provides the RefCOCO and RefCOCO+ benchmarks used for fine-tuning and evaluation.","marker":"[43]"},{"why":"Provides the RefCOCOg benchmark with richer natural descriptions.","marker":"[26]"}],"fun_headline_variants":["Color swaps create a million synthetic queries for REC","Single-attribute edits create synthetic REC data that transfers","Color-driven variations generate reusable REC pretraining data","One color change yields a million training queries for REC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on GLIGEN faithfully painting the swapped color onto the referred object inside its box: if the generator often ignores the color or location guidance, Harlequin's labels are noisy and the measured gains could come from corpus size rather than accurate synthetic supervision.","fun_headline_variants_meta":{"raw":{"variants":["Color swaps create a million synthetic queries for REC","Single-attribute edits create synthetic REC data that transfers","Color-driven variations generate reusable REC pretraining data","One color change yields a million training queries for REC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000392,"raw_usage":{"total_tokens":2047,"prompt_tokens":916,"completion_tokens":1131,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1070}},"tokens_in":532,"tokens_out":1131,"duration_ms":10100,"temperature":1.0,"reasoning_tokens":1070,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:51:14.248258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Harlequin images and run an attribute-aware detector to verify, for each query, that the object inside the bounding box carries the swapped color and that the unswitched context objects are still present. If a substantial share of images fails this fidelity check while the fine-tuning gains remain, then accurate rendering is not the active ingredient; if the gains disappear when Harlequin images are replaced by unrelated images paired with the same queries, rendering fidelity is the mechanism.","supporting_citations":[{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Implements the Image Generation Engine: renders each synthetic image from the edited caption and box-plus-text entities."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Supplies the seed annotations (captions, referring expressions, bounding boxes) that the pipeline varies."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"TransVG is the first evaluated model; pre-training it on Harlequin produces consistent gains on all three benchmarks."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"VLTVG is the second evaluated model, used to show the pre-training transfers to a verification-style architecture."},{"cited_title":"IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)in press (2024) 3, 10, 11","cited_arxiv_id":null,"evidence_quote":"LGR-NET is the third evaluated model and gives the largest reported gains, especially on RefCOCOg."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides the RefCOCO and RefCOCO+ benchmarks used for fine-tuning and evaluation."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides the RefCOCOg benchmark with richer natural descriptions."}],"review_version":1}