{"id":"97ba944e-1f08-448e-8bb1-d2d8418dba96","arxiv_id":"2508.08754","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Palette embeddings fused into a diffusion model's cross-attention guide image colorization toward a desired color scheme more closely than text-only or palette-in-prompt conditions.","lead":"This paper adds a color palette as a separate steering signal, together with text, when a diffusion model colorizes grayscale images. It reports that learned palette embeddings match desired color schemes more accurately than text-only or palette-in-prompt guidance, which could help designers control image color themes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative evaluation only uses palettes/reference images extracted from the target image and never tests user-specified palettes; claimed user-customized palette benefit is unsupported.","rationale":"Both the reader and I identify a missing evaluation, but my concern is more specific: the quantitative setup conflates 'palette guidance' with oracle information. Because palettes in Tables 3/4 are extracted from the ground-truth images, all palette-based methods receive the answer. That is acceptable for a reconstruction-style colorization metric, but it does not demonstrate the claimed user-customization capability. The image-palette variant is especially susceptible: if the reference image is the target image, the CLIP image encoder sees the full-color target, making the color-conditioning effectively leakage. This undermines the strongest quantitative evidence. However, the palette-only and text-palette variants also show gains, so the central mechanism may still be viable. The user study is encouraging but it uses preference only and does not test hand-authored palettes either. Thus the paper should remain CONDITIONAL, conditioned on showing generalization to user-provided palettes/references.","tokens_in":11483,"tokens_out":8229,"duration_ms":84659,"concrete_test":"Run the text-palette and palette-only models on 100 held-out grayscale image+prompt pairs using hand-authored palettes (e.g., from Adobe Color) instead of the target-derived K-means palette, and compare DCCW/Hist_Bha to the results in Table 3 (e.g., text-palette DCCW 16.07). If DCCW increases substantially (>20, say), the model has not generalized to user-customized palettes. For the image-palette model, extract the palette from a different reference image than the target and evaluate whether the generated image matches that reference palette; this tests whether the reported gains depend on conditioning on the ground-truth image.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that palette guidance improves color control for user-selected palettes or reference-derived palettes. The quantitative evaluation (Tables 3–4) only uses palettes extracted by K-means from the target image itself (Section 4.1, 4.4). For the image-palette model, the reference image appears to be the ground-truth color image (since PSNR/SSIM against ground truth are reported), so the model receives oracle color information through both the palette and the reference-image encoder. This makes the strong gains of the image-palette variant uninterpretable: it is unclear whether the improvement comes from the learned palette embedding or from the target image being provided as a reference. More importantly, the claimed 'manually selected palettes' scenario (Figure 1, Section 3.1) is never evaluated; all test palettes are natural-image K-means palettes. If the representation model overfits to the statistics of such palettes (e.g., CIELAB 16^3 binning, Section 3.3), hand-authored palettes with out-of-distribution colors may fail, undercutting the practical user-control claim. The paper even acknowledges related gaps (color bleeding, text-palette conflict) in Section 5.4 but does not acknowledge the lack of user-palette evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a palette-based guidance mechanism for diffusion-based image colorization. The authors construct two palette-text-image datasets (LAION-22k-PTI and COCO-Stuff-PTI) and train three palette representation models — palette-only, text-palette (with CLIP or SBERT text encoders), and image-palette (with CLIP or DINOv2 image encoders) — based on an extended masked color model. These representations are fused with text embeddings and injected into a Stable Diffusion-based colorization model via cross-attention, while the grayscale input is fed through a ControlNet branch. The main claim (Section 5.2, Tables 3 and 4) is that learned palette embeddings significantly improve color control over text-only and palette-in-prompt baselines on color-level metrics (Hist_Bha, DCCW), with a user study involving 33 designers supporting the qualitative preference for the proposed method.","tokens_in":11841,"tokens_out":6128,"duration_ms":63143,"significance":"If the results hold, this is a useful contribution to controllable colorization: it introduces a new conditioning modality (palette) and systematically compares several palette representation strategies, including a user study with professional designers. The datasets and the exploration of text/image encoders for palette representation are valuable assets. The paper is clearly motivated and the experimental design is mostly appropriate, but the evaluation has a few load-bearing gaps (oracle reference for the image-palette model, no hand-authored palettes, no statistical significance tests, and no simple learned-palette baseline). These issues prevent the paper from fully supporting its central claims in its current form, but they are addressable with additional experiments.","major_comments":[{"comment":"The image-palette model is evaluated with the ground-truth color image as the reference image. This follows from the dataset construction (palettes are K-means extracted from the original image) and from the reported PSNR/SSIM/LPIPS against ground truth. Consequently, the large gains of the image-palette variant in Tables 3 and 4 (e.g., DCCW 14.12 vs. 16.07 on LAION) conflate the effect of the learned palette embedding with the oracle reference-image condition. The claim that the image-palette representation is superior is uninterpretable unless a non-oracle reference image is used at inference, or an ablation removes the reference-image branch while keeping the palette.","section":"Section 4.4 / Figure 2 caption"},{"comment":"All evaluation palettes are K-means clusters extracted from natural images, and the claimed 'user-selected palette' scenario (Figure 1, Section 3.1) is never tested. The model is trained and evaluated on the same distribution of natural-image K-means palettes, so the paper does not support the practical claim that hand-authored palettes with out-of-distribution colors (e.g., strong neon or pastel palettes) will be faithfully followed. I recommend adding an evaluation with user-designed palettes or at least synthetic palettes sampled from a broader CIELAB distribution, and reporting whether the colorization outputs align with those palettes on Hist_Bha and DCCW.","section":"Section 4.1 / Section 5.2"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any colorization metric. Since the generation process is stochastic (DDIM sampling) and several differences are small (e.g., LPIPS 0.3601 vs. 0.3671 on LAION; SSIM consistently lower than L-CAD), the abstract's claim of 'significantly improves' is not statistically supported. Please report means and standard deviations over multiple seeds or inference runs, and perform significance tests (e.g., paired bootstrap or Wilcoxon) for the key color-level metrics.","section":"Tables 3 and 4"},{"comment":"The only palette-conditioned baselines are palette descriptions inserted into the text prompt, which is a deliberately weak baseline given CLIP's poor handling of raw RGB values. To isolate the contribution of the masked-color representation, the paper should include a simple learned baseline, such as a linear or MLP projection of the five palette RGB values into the same cross-attention conditioning space. Without this, it is unclear whether the observed improvement comes from the proposed palette representation or merely from using any learned embedding of palette values.","section":"Section 4.3 / Tables 3 and 4"}],"minor_comments":[{"comment":"Typo: 'boarder' should be 'broader'.","section":"Figure 3 caption"},{"comment":"The concatenation of palette and text embeddings is described only as 'with aligned dimensions' and 'projected into the same space.' Please specify whether the concatenation is along the sequence dimension (adding tokens) or the feature dimension (pooled vector), and give the resulting conditioning vector shape in the cross-attention layers.","section":"Section 3.2"},{"comment":"Hyperparameters for the palette representation models (learning rate, batch size, number of epochs, sequence length, token vocabulary size) are not reported. At least the sequence length and vocabulary size should be stated for reproducibility.","section":"Section 4.2"},{"comment":"The text says 33 designers were recruited, but Table 5 reports 28 responses. Please clarify whether 5 did not complete the questionnaire or were excluded.","section":"Section 5.3"},{"comment":"The phrase 'textural data' should likely be 'textual data' when referring to Sentence-BERT training.","section":"Section 3.3"},{"comment":"The DCCW metric is used both for palette-model evaluation and for colorization evaluation. It would help to state explicitly whether DCCW is computed between the predicted/generated palette and the reference palette after dynamic warping, and whether the generated palette is also K-means extracted with the same K=5.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is solid and the datasets and user study are valuable, but the evaluation has a few important gaps: the image-palette experiment is confounded by the oracle reference input, the user-customized-palette scenario is never tested, and no statistical support is given for the 'significant' claims. These are fixable with additional experiments and do not require a redesign of the method. The manuscript would be a good fit for the journal if the authors address these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable, incremental paper on using learned color-palette embeddings as an extra conditioning signal in diffusion models, tested through image colorization. The core comparison of palette-only, text-palette, and image-palette representations is useful, and the results support the idea that learned embeddings beat simply writing RGB values into the prompt. But the quantitative evaluation has two real blind spots that the authors don't address: all test palettes are K-means extracted from natural images, and the image-palette variant appears to feed the ground-truth color image as the reference at test time. That makes the strongest-sounding results hard to interpret and leaves the 'user-customized palette' selling point untested.\n\nWhat's new and good: the combination of masked-color-model embeddings from the authors' prior work with diffusion cross-attention conditioning is not something I've seen in the cited literature. The systematic ablation of palette representation modalities and encoder choices (CLIP vs. SBERT, CLIP vs. DINOv2) is useful and gives practical guidance. The user study with professional designers is a plus, and the paper is honest about limitations like color bleeding and text-palette conflict.\n\nSoft spots: no significance tests or error bars, and checkpoints are selected based on performance, so the numbers may be optimistic. The image-palette setup is ambiguous — if the reference is the ground truth, the PSNR/SSIM gains are oracle-conditioned and tell us little about actual reference-based color transfer. The stress-test note is right: user-authored palettes are never evaluated, so the 'designer picks their own colors' claim is unsupported, even if it might hold in practice. The reliance on the authors' own masked color model [22] is transparent, but there is no code or data release, so reproducibility is limited.\n\nWho this is for: researchers working on color control in generative models or design tools. Not a breakthrough, but a competent empirical step worth engaging with. I would send it to peer review, with requests to clarify the image-palette test setup, add significance testing, and test at least a handful of hand-authored palettes to back the central use case.","headline":"A solid but incremental empirical paper on learned palette embeddings for diffusion color control, with two evaluation blind spots: oracle-conditioned image-palette results and no user-defined palettes tested.","tokens_in":12252,"tokens_out":2683,"would_cite":false,"duration_ms":26326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that encoding color palettes as learned embeddings and fusing them with text in cross-attention gives diffusion models global color-scheme control that text prompts alone cannot provide.","keywords":["color guidance","diffusion models","palette representation","image colorization","cross-attention conditioning","text-to-image generation","color palette","user study"],"falsifier":"Take hand-authored palettes with deliberately non-natural color combinations, such as a brand palette or a neon-only palette, run the text-palette model and the palette-in-prompt baselines on the same grayscale images, and compare DCCW and histogram distance on the outputs; if the learned-embedding advantage disappears or reverses on these palettes, the claim of general palette-guided color control is falsified.","tokens_in":11423,"feed_emoji":"🎨","tokens_out":7317,"duration_ms":69525,"temperature":0.7,"pith_summary":"This paper tries to establish that a color palette should be a conditioning input in its own right for diffusion-based image generation, rather than being described in words or appended to the prompt. It proposes a palette-guided colorization framework in which a learned palette embedding is fused with the text embedding and injected through cross-attention, while the grayscale image is supplied through a separate structural branch. If the claim holds, users gain global control over the color scheme of a generated image, including backgrounds and objects never named in the prompt, by choosing a palette or pointing to a reference image. The paper reports that this approach outperforms text-only and palette-in-prompt baselines on color-alignment metrics and in a preference study with professional designers.","feed_headline":"Palette guidance beats text prompts for diffusion color control","feed_subtitle":"Learned palette embeddings, fused with text in cross-attention, align generated colors with target schemes.","key_machinery":"The load-bearing component is the learned palette embedding. A palette is tokenized as discrete color codes in a $16\\times16\\times16$ CIELAB histogram, embedded, and processed by a masked color model that predicts masked color tokens; optional text or image conditions are mixed in through cross-attention. The resulting palette embedding is projected into the same space as the text embedding, concatenated with it, and fed into the cross-attention layers of the diffusion U-Net. This makes the palette a first-class conditioning signal, so the model can apply the palette globally rather than only to objects named in the prompt.","core_discovery":"The central discovery is that raw palette colors only help a diffusion model when they are converted into a learned embedding; writing RGB values into the prompt is not just weak, it can degrade output. The paper demonstrates this in an image-colorization setting: a grayscale image enters through a spatial-control branch, while a concatenation of text and palette embeddings is injected through cross-attention in the U-Net. Palette embeddings are produced by a masked color model trained over color tokens in CIELAB space, optionally conditioned on text or on a reference image. Across the LAION-derived and COCO-Stuff-derived test sets, palette-guided models improve histogram distance and palett","pith_inferences":["A direct next step is to make text-versus-palette priority user-controllable; the paper documents cases where text wins over palette but leaves the arbitration mechanism unspecified.","Because the palette and text embeddings are concatenated before cross-attention, the fusion should transfer to text-to-image generation without the grayscale branch; that is a testable extension the paper motivates but does not run.","Training the palette representation on hand-authored palettes rather than only K-means-extracted natural-image palettes would test whether the method serves professional design palettes; the current datasets leave this open.","The same masked color model could double as an automatic color-harmony checker, since it already predicts plausible palette colors from partial context."],"forward_implications":["Users could control the color mood of an entire image, backgrounds and minor objects included, by selecting a palette without having to name every colored element in the prompt.","Palette information must be embedded, not spelled out: adding RGB values to prompts can make color alignment worse than using no palette at all.","Palettes extracted from reference images transfer color style while leaving texture alone, which style-transfer methods that entangle color and texture do not offer.","Conditioning palette representations on text or images, especially with cross-modal encoders, yields sharper color predictions than palette-only representations.","The same palette-text fusion can be shifted to other color-sensitive generation tasks, such as text-to-image generation, where global color control is also missing."],"supporting_citations":[{"why":"Supplies the cross-modal text encoder that produces the prompt embeddings used in conditioning and in the text-palette representation.","marker":"[24]"},{"why":"Provides the structural-conditioning architecture the grayscale branch is based on.","marker":"[38]"},{"why":"State-of-the-art text-driven colorization baseline that must be beaten; also the source of the segmentation-based color-bleeding remedy discussed as future work.","marker":"[33]"},{"why":"Base masked color model that is extended to produce palette embeddings conditioned on text or images.","marker":"[22]"},{"why":"Supplies the COCO-Stuff images used to build the second palette-text-image evaluation dataset.","marker":"[5]"},{"why":"Supplies the large image-text collection from which the LAION-22k-PTI training set is sampled.","marker":"[28]"},{"why":"Defines the Dynamic Closest Color Warping metric used to measure palette similarity in both representation and colorization evaluations.","marker":"[15]"},{"why":"Text-only sentence encoder used as the comparison point showing cross-modal text features help palette representation.","marker":"[25]"},{"why":"Self-supervised image encoder used as the comparison point showing cross-modal image features help palette representation.","marker":"[20]"}],"fun_headline_variants":["Learned palette embeddings, not raw RGB, steer diffusion color","Palette embeddings, not text, give diffusion color control","Diffusion colorization: palette embeddings beat raw color codes","Embedded palettes outperform raw colors in diffusion guidance","Color palette embeddings improve diffusion image colorization"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The method is trained and evaluated only on five-color palettes extracted by K-means from natural images, so the promised benefit for user-customized palettes depends on the learned embeddings generalizing to palettes unlike those found in natural photos.","fun_headline_variants_meta":{"raw":{"variants":["Learned palette embeddings, not raw RGB, steer diffusion color","Palette embeddings, not text, give diffusion color control","Diffusion colorization: palette embeddings beat raw color codes","Embedded palettes outperform raw colors in diffusion guidance","Color palette embeddings improve diffusion image colorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00059,"raw_usage":{"total_tokens":2565,"prompt_tokens":663,"completion_tokens":1902,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":1824}},"tokens_in":407,"tokens_out":1902,"duration_ms":14914,"temperature":1.0,"reasoning_tokens":1824,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:20:52.113464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take hand-authored palettes with deliberately non-natural color combinations, such as a brand palette or a neon-only palette, run the text-palette model and the palette-in-prompt baselines on the same grayscale images, and compare DCCW and histogram distance on the outputs; if the learned-embedding advantage disappears or reverses on these palettes, the claim of general palette-guided color control is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the structural-conditioning architecture the grayscale branch is based on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"State-of-the-art text-driven colorization baseline that must be beaten; also the source of the segmentation-based color-bleeding remedy discussed as future work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Base masked color model that is extended to produce palette embeddings conditioned on text or images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the COCO-Stuff images used to build the second palette-text-image evaluation dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Dynamic Closest Color Warping metric used to measure palette similarity in both representation and colorization evaluations."}],"review_version":1}