{"id":"6f3d8cee-9cfb-41e5-947a-3c4c3f3724d7","arxiv_id":"2412.02929","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A dual-diffusion model that jointly denoises an image and a bit-encoded panoptic segmentation map from a text prompt, with multi-scale patching for high-resolution maps.","lead":"This paper trains diffusion models to output both an image and a matching panoptic segmentation map from a single text prompt, so the map guides the image during the same denoising process. The authors report FID scores worse than their own U-ViT baseline and compare against no controllable-generation baseline, so the state-of-the-art claim is not supported.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central co-generation claim is unsupported: MCD (Sec. 3.6) only compares category-frequency histograms, and the reported values (1.52–1.64 on [0,2]) indicate large layout mismatch; with the train/inference map-noise inconsistency, the map stream may not provide the claimed scene control.","rationale":"I read the paper as claiming two things: (i) a mechanism that co-generates an image latent and a bit-encoded panoptic map through a joint DPM-solver reverse process, and (ii) state-of-the-art image generation with 'implicit scene control.' For (i) to hold, the map stream must decode into valid panoptic layouts. For (ii) to hold, the map channel must measurably improve or control image generation. The paper provides no quantitative evidence for either: Table 1 shows both PDM variants have worse FID than the plain U-ViT baseline, and no relevant scene-control baselines (UniGS, SpaText, ControlNet, DiffuMask) are compared. I agree with the reader's rejection. My most load-bearing concern is slightly different from the reader's identified weakest assumption: rather than the N(0,1) initialization per se, it is the absence of any spatial validation of the generated maps. The MCD metric is a histogram distance; it can be low only if category frequencies match, but it cannot detect wrong object locations, missing instances, or implausible shapes. The reported MCD values are high on a [0,2] scale, so even the weak metric is not satisfied well. The train/inference noise inconsistency is a concrete implementation flaw that reinforces the concern and may explain poor map behavior. A focused test—computing PQ and bit-error rates—would settle whether the map stream works. If it does not, the co-generation premise collapses, independent of the FID comparison. I would credit the qualitative examples and the zero-shot CIFAR experiment as suggestive, and the B.2 zero-map ablation does show that the model can still generate images when map inputs are zeros; that further undercuts the claim that the map stream provides the required scene control rather than being ignored or acting as noise.","tokens_in":14301,"tokens_out":9541,"duration_ms":92031,"concrete_test":"Re-evaluate the 5,000 COCO validation generated maps with the standard panoptic metrics (PQ, SQ, RQ) and per-class IoU against the ground-truth panoptic annotations, and separately decode the raw map latents to compute the per-pixel bit-error rate for the 8-bit category encoding. If PQ is near zero, per-class IoU is near chance, or bit-error rate is high, the map stream is not generating valid panoptic layouts and the co-generation/SOTA claim does not stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim—that PDM co-generates images and panoptic segmentation maps from text and uses the map as built-in scene control—the generated maps must actually be valid panoptic segmentations. The paper never demonstrates this. The proposed MCD metric (Sec. 3.6) only compares per-category pixel counts; it does not measure spatial layout, instance boundaries, or semantic correctness. Table 2 reports MCD values around 1.52–1.64 on a [0,2] scale, close to the upper end of the range, meaning the category-frequency histograms of generated and ground-truth maps differ by a large fraction of the pixels. Qualitative figures are not sufficient to overturn this. A related, concrete implementation flaw is the train/inference mismatch: training with map noise is described as N(0,2I) in Eq. 3, Algorithm 1 uses normal(std=2), while Algorithm 2 initializes M_t from N(0,1), and no analysis of bit-error rates or category confusion is provided. If the map stream is not recovering meaningful bits, the image stream sees noisy or incorrect layout tokens, so the headline 'implicit scene control' is not established. The load-bearing premise—that the bit-encoded map channel denoises into usable panoptic layouts—is thus unsupported by the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Panoptic Diffusion Models (PDM), which aim to generate an image and a panoptic segmentation map jointly from a text prompt. Two architectures are presented: a one-stream diffusion transformer based on U-ViT and a two-stream transformer with a pretrained image stream and a fine-tuned map stream. The method encodes panoptic maps as 8-bit analog bit-planes, adds Gaussian noise, and uses a modified DPM-solver to denoise image latents and maps in parallel. A Multi-Scale Patching mechanism is proposed to produce maps at resolutions up to four times the image latent, and a new metric, Mean Count Difference (MCD), is introduced to evaluate generated maps. The stated contributions are co-generation of images and panoptic maps, fast simultaneous denoising, a two-stream fine-tuning scheme, and state-of-the-art image generation with implicit scene control.","tokens_in":14520,"tokens_out":8097,"duration_ms":77153,"significance":"If the central claim were correct, PDM would be a notable contribution: a single text-conditioned diffusion model that produces both a photorealistic image and a structurally valid panoptic segmentation map, with the map providing built-in layout control. The idea of jointly denoising image and map streams is interesting, and the reported qualitative examples suggest a plausible direction. However, the paper as written does not establish the central claim. The quantitative image-generation results are worse than the U-ViT baseline, the map evaluation metric ignores spatial structure, and the map-noise schedule in training is inconsistent with the inference algorithm. These issues affect the load-bearing parts of the contribution. The manuscript also omits comparisons with established controllable-generation methods, so the 'state-of-the-art' claim is not substantiated.","major_comments":[{"comment":"The abstract claims that PDM 'achieves state-of-the-art results in image generation with implicit scene control,' but Table 1 reports FID 18.52 for one-stream PDM and 10.99 for two-stream PDM, both worse than the U-ViT baseline at 8.29. Only the one-stream PDM with ground-truth maps (8.21) is slightly better than the baseline, and the two-stream PDM with maps is worse (11.61). The claim of 'comparable fidelity' is therefore contradicted by the paper's own numbers. No controlled-generation baselines (e.g., ControlNet, SpaText, UniGS) are included, so the 'state-of-the-art for implicit scene control' claim is unsupported. The authors should either temper the claims to match the reported results or add a proper comparison on the claimed task.","section":"Abstract; Table 1"},{"comment":"The proposed MCD metric only compares category-frequency histograms, not spatial layout, instance boundaries, or semantic correctness. The reported MCD values (1.522–1.638 on a [0,2] scale) indicate that the generated maps differ from ground-truth maps in the pixel counts of most categories, which does not demonstrate that the maps are valid panoptic segmentations. The qualitative figures are not sufficient to establish map quality. The authors should report standard panoptic segmentation metrics such as Panoptic Quality (PQ), or at least IoU, for the co-generated maps. This is load-bearing because the central claim is that PDM produces usable segmentation maps that control image generation.","section":"Sec. 3.6; Table 2"},{"comment":"There is a concrete train/inference mismatch in the map noise scale. Training uses ϵM ∼ N(0, 2I) (Eq. 3 and Algorithm 1), while Algorithm 2 initializes M_t from N(0, 1). Appendix B.3 states that maps are not properly denoised when the noise variance is N(0, I), so the inference initialization in Algorithm 2 is inconsistent with the trained model. Furthermore, the DPM-solver update equations in Sec. 3.3 are derived under the standard assumption that the perturbation noise has unit variance; applying them directly to a forward process with non-unit noise (std=2) is not mathematically justified. The authors need to either use unit-variance map noise (if the bit-flipping motivation permits) or derive the correct solver for the non-unit-noise forward process, and then verify the generated maps are valid.","section":"Sec. 3.2; Algorithm 1; Algorithm 2"},{"comment":"The experiments compare PDM only with text-to-image baselines (GLIDE, Imagen, VQ-Diffusion, UViT). Since the paper's claimed contribution is 'image generation with implicit scene control,' a fair evaluation must include methods that use spatial control, such as ControlNet, SpaText, or UniGS, or at least a sequential baseline that first generates a map and then generates an image conditioned on the map. Reporting raw FID/CLIP against text-only baselines does not evaluate the proposed method's claimed advantage of built-in layout control. Without such comparisons, the 'state-of-the-art' claim cannot be assessed.","section":"Sec. 4; Table 1"},{"comment":"Equation 6 is simply the chain rule of probability, not a derivation that joint prediction is feasible or beneficial. The text argues that 'the intermediate features already contain the segmentation information' and that self-attention conditions x_{t-1}, but this is an architectural assumption. The paper does not compare the joint model with a sequential model that first generates M0 and then generates x0 conditioned on the map. An ablation isolating the benefit of joint prediction would be needed to support the claim that co-generation is advantageous over a two-stage pipeline.","section":"Sec. 3.4; Eq. 6-7"}],"minor_comments":[{"comment":"Equation 1 uses βt as the noise coefficient without a square root, whereas the standard DDPM forward process uses sqrt(βt) for the noise term; please correct the notation or clarify the schedule.","section":"Sec. 3.2, Eq. 1"},{"comment":"The text states 'We will include a comparison of the number of parameters to highlight this advantage,' but no such parameter-count comparison appears in the experiments.","section":"Sec. 2.4"},{"comment":"The sentence 'We generate 30,000 images and segmentation maps from 5,000 text files in the COCO dataset’s validation set, with each file containing five captions' is ambiguous and the arithmetic appears inconsistent; please specify the exact evaluation protocol.","section":"Sec. 4.1"},{"comment":"Variable naming is inconsistent: 'Xt' in the initialization line and 'xt' in the loop; unify the notation to avoid confusion.","section":"Algorithm 2"},{"comment":"The paper notes that COCO has only 133 panoptic classes but category IDs range up to 200; please clarify how the 8-bit encoding maps the actual 133 classes and how unused IDs are handled.","section":"Appendix D"},{"comment":"There are several typos, e.g., 'Evaluaiton' in Sec. 4.2 and 'segmanation' in Sec. 3.4; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claims are not supported by its own reported numbers, and the map-denoising procedure has a train/inference inconsistency that is not a minor typo: it affects the validity of the co-generation mechanism. The authors would need to redesign the noise schedule or the solver, re-run experiments, add appropriate baselines, and evaluate map quality with standard panoptic metrics. Given the scope of these changes, I cannot recommend acceptance even with major revision; the current draft is not ready for publication in a top-tier venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth taking seriously: instead of conditioning on a segmentation map, denoise an image and a bit-encoded panoptic map in parallel, letting each guide the other. I haven't seen that exact formulation in the prior work, and the two-stream variant (frozen image stream, trainable map stream with zero-conv injection) is a sensible way to bolt this onto a pretrained model. The multi-scale patching trick for higher-res maps is also a practical contribution, and the sanity check in Table 1—that given ground-truth maps the one-stream model's FID improves to 8.21, matching the U-ViT baseline—shows the map information can genuinely steer image generation when it's accurate.\n\nThe problems are mostly in how the results are presented and one concrete implementation bug. The abstract and Section 4.1 claim \"comparable fidelity\" and \"state-of-the-art\" while Table 1 shows both PDM variants worse than the plain U-ViT baseline (18.52 and 10.99 vs 8.29). That's not a marginal gap, and calling it comparable is hard to justify. The relevant controllable-generation baselines (ControlNet, SpaText, UniGS) are never evaluated, so the SOTA claim is simply untested. The MCD metric is weak: it only compares per-category pixel counts, not layout or boundaries, and the reported values around 1.5–1.6 on a [0,2] scale are close to a large-divergence regime, not a pass.\n\nMost importantly, there is a train/inference mismatch that looks like a genuine bug. Training corrupts maps with epsilon_M ~ N(0, 2I) (Eq. 3 and Algorithm 1), and the paper argues variance >1 is necessary for the analog bits to flip. Yet Algorithm 2 initializes the inference map from N(0, 1). The authors never analyze bit-error rates or category confusion, and if the map channel is starting from the wrong noise scale, that alone could explain the poor map quality and the downstream FID regression. This is fixable, but it is load-bearing for the central claim that the model co-generates usable layouts.\n\nWho benefits from this paper? A reader working on joint scene generation and understanding might find the formulation inspiring, and the noise-mismatch fix plus honest baseline comparison could turn it into a useful result. But as it stands, the evidence does not support the headline claims. If I were handling it, I'd want the authors to run the corrected version, add the missing baselines, and either improve MCD or supplement it with something layout-aware. The idea deserves serious referee time, but the current version should not be accepted.","headline":"The co-generation idea is genuinely new, but this version's claims outrun its evidence: the train/inference noise mismatch is a load-bearing bug, and the evaluation doesn't back the SOTA claim.","tokens_in":15136,"tokens_out":2930,"would_cite":false,"duration_ms":29867,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single diffusion model can co-generate a photorealistic image and an 8-bit-encoded panoptic segmentation map from one text prompt, with the map acting as built-in guidance so object shapes and scene layout follow…","keywords":["panoptic segmentation","diffusion models","text-to-image generation","co-generation","analog bits","DPM-solver","classifier-free guidance","multi-scale patching"],"falsifier":"Run the published inference procedure but initialize the map stream with $\\mathcal{N}(0,2I)$ noise instead of the $\\mathcal{N}(0,1)$ used in Algorithm 2, and compare MCD and category-wise bit-error rates on COCO validation prompts; if the mismatch matters, the generated maps will change measurably. A second decisive check is to feed the generated maps to a standard panoptic-quality evaluator and ask whether instances in the map align with objects in the co-generated image, since MCD cannot detect mislocated categories.","tokens_in":13981,"feed_emoji":"🖼️","tokens_out":9236,"duration_ms":87634,"temperature":0.7,"pith_summary":"The paper introduces Panoptic Diffusion Models (PDM), which aim to be the first text-to-image diffusion models that generate an image and a panoptic segmentation map at the same time from the same prompt. This matters because diffusion models usually have no built-in notion of object shape or scene layout, so they distort objects and omit categories named in the prompt. PDM's answer is to make the segmentation map an internal, co-generated condition: image and map denoise together and influence each other at every step. The paper reports that this yields image fidelity comparable to a text-only diffusion transformer and improved text-image relevance, while also emitting a pixel-level map, and calls the result state of the art for image generation with implicit scene control. When ground-truth maps are supplied, the same model becomes a text-guided image-to-image generator.","feed_headline":"First diffusion model generates an image and its panoptic map together","feed_subtitle":"Co-generating a labeled blueprint with the image lets the model enforce shapes and scene layout from the text prompt.","key_machinery":"The load-bearing object is the joint reverse diffusion step over the pair consisting of the image latent $x_t$ and the bit-encoded map $M_t$. Maps are first converted to 8-bit analog bits scaled to $[-1,1]$ so that discrete categories become continuous data, and during training noise $\\epsilon_M \\sim \\mathcal{N}(0,2I)$ is added so the noise can flip bits. The model outputs both the image-noise estimate and a direct estimate of the clean map $M_0$, and the two outputs are combined in a DPM-solver-style update, the same fast ODE solver used for image denoising, extended to the map stream. In the transformer, image and map tokens are concatenated and processed by self-attention, which lets each modality condition the other; in the two-stream version, a zero-initialized convolution injects map-derived features into the frozen image stream. Multi-scale patching lets the map be predicted on a finer grid than the image latent without a separate decoder.","core_discovery":"The central claim is that image generation and panoptic segmentation can be posed as one joint diffusion problem rather than a generate-then-segment or segment-then-generate pipeline. PDM encodes each category ID on the map as eight analog bits scaled to $[-1,1]$, corrupts the map with Gaussian noise of variance 2 during training, and trains a transformer to predict the clean map and the image noise simultaneously. Because the map and image tokens attend to each other in the same transformer, the model learns the correlation among text, image, and layout; at inference the reverse process is run with a fast DPM-solver-style update for both streams, so image and map emerge together. The authors demonstrate this in a one-stream diffusion transformer and in a two-stream model built from a frozen pretrained image stream plus a tunable map stream connected by zero-initialized convolutions. They also introduce multi-scale patching to emit maps at two or four times the latent resolution and a Mean Count Difference (MCD) metric for evaluating the generated maps.","pith_inferences":["Inference: because MCD only compares per-category pixel counts, two maps with identical category budgets but different spatial layouts score the same; a spatial metric such as Panoptic Quality would test whether co-generated maps actually align with objects in the co-generated image.","Inference: the same joint-denoising idea could be applied to other discrete scene-structure outputs such as depth, edge maps, or keypoints, turning them into learned internal controls rather than external inputs.","Inference: inference initializes the map stream from $\\mathcal{N}(0,1)$ while training corrupts maps with $\\mathcal{N}(0,2I)$, so switching the initialization to $\\mathcal{N}(0,2I)$ and measuring MCD would test the consistency of the pipeline, a comparison the paper does not report.","Inference: the paper's scaling remark predicts that map quality and image quality improve together as model size grows, which could be checked by varying the hidden dimension and tracking FID and MCD jointly."],"forward_implications":["If the central claim holds, text-to-image models gain an internal scene layout: a prompt like an upside-down stop sign will tend to produce an octagonal stop-sign mask and image, because the map stream is trained to respect shape categories.","Co-generation removes the need for an external segmentation model or a user-provided map at inference, while still allowing ground-truth maps to be supplied for text-guided image-to-image editing.","The frozen-backbone two-stream design offers a parameter-efficient route to adding layout awareness to an existing diffusion model by fine-tuning only the map stream.","Multi-scale patching implies that high-resolution segmentation maps can come from the same latent budget, with a trade-off between map resolution and map quality at patch factor four."],"supporting_citations":[{"why":"Supplies the analog-bits encoding that converts discrete category IDs into continuous 8-bit data a diffusion model can denoise.","marker":"Chen et al. (2022)"},{"why":"Provides the DPM-solver fast ODE solver that the paper extends to the joint image-map reverse process.","marker":"Lu et al. (2022; 2023)"},{"why":"Contributes the zero-initialized convolution conditioning mechanism used in the two-stream PDM.","marker":"Zhang et al. (2023)"},{"why":"Provides the U-ViT diffusion transformer backbone on which both PDM variants are built.","marker":"Bao et al. (2023)"},{"why":"Supplies the VAE latent space and text conditioning setup used for the image stream.","marker":"Rombach et al. (2022)"},{"why":"Provides the COCO2017 dataset with panoptic maps and captions used for training and evaluation.","marker":"Lin et al. (2015)"},{"why":"Defines the DDPM noise schedule and forward-reverse formulation that the joint process builds on.","marker":"Ho et al. (2020)"},{"why":"Supplies classifier-free guidance, which the paper adapts by dropping only the text condition while keeping map guidance.","marker":"Ho & Salimans (2022)"},{"why":"Defines panoptic segmentation and Panoptic Quality, the task and the metric against which the new MCD metric is positioned.","marker":"Kirillov et al. (2018)"}],"fun_headline_variants":["One model, two outputs: image plus its panoptic map","Joint diffusion for images and segmentation maps","Draw the image and label it in one pass","Panoptic diffusion: image and layout together","First co-generation of image and panoptic map"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a bit-encoded segmentation map, corrupted with $\\mathcal{N}(0,2I)$ noise during training, can be recovered by the same fast ODE solver used for images when inference starts from $\\mathcal{N}(0,1)$ noise as written in the algorithm; if the map stream instead accumulates bit errors, the image stream is being guided by a garbled layout.","fun_headline_variants_meta":{"raw":{"variants":["One model, two outputs: image plus its panoptic map","Joint diffusion for images and segmentation maps","Draw the image and label it in one pass","Panoptic diffusion: image and layout together","First co-generation of image and panoptic map"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1216,"prompt_tokens":959,"completion_tokens":257,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":575,"tokens_out":257,"duration_ms":3330,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:56:35.923439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published inference procedure but initialize the map stream with $\\mathcal{N}(0,2I)$ noise instead of the $\\mathcal{N}(0,1)$ used in Algorithm 2, and compare MCD and category-wise bit-error rates on COCO validation prompts; if the mismatch matters, the generated maps will change measurably. A second decisive check is to feed the generated maps to a standard panoptic-quality evaluator and ask whether instances in the map align with objects in the co-generated image, since MCD cannot detect mislocated categories.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the zero-initialized convolution conditioning mechanism used in the two-stream PDM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the U-ViT diffusion transformer backbone on which both PDM variants are built."}],"review_version":1}