{"id":"966c5cf4-6eb1-4762-ab86-bca797350148","arxiv_id":"2508.20020","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Direct label-space diffusion, conditioned on image and text, achieves state-of-the-art 69.7 Average Recall on Panoptic Narrative Grounding.","lead":"GS trains a diffusion model to generate segmentation masks directly from noise, conditioned on the input image and a natural language description. On the Panoptic Narrative Grounding benchmark it reports the highest Average Recall to date (69.7), positioning generative mask synthesis as a viable alternative to discriminative segmentation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 69.7 SOTA claim is not yet supported: hyperparameters were tuned on a 7% slice of the test set (§4.4), and the 2.6-point margin over EIPA+MLMA could be a selection artifact.","rationale":"The reader's weakest_assumption focuses on the VAE latent geometry; while that ambiguity is real and should be fixed, the more decisive issue for the paper's stated contribution is the validity of the headline number. The paper's own §4.4 gives an explicit, quotable admission that the hyperparameters were selected on the test set. Hyperparameter selection on the test split is a methodological error independent of any modeling choice; it directly undermines the comparison to prior methods in Table 2 because the selected configuration was chosen with access to the answer. In addition, the full paper does not supply code or a validation split, so no external check can currently separate a true 69.7 from an artifact of selection. I do not see internal mathematical inconsistency in the label-diffusion construction, and the dual-conditioning idea is coherent, but the evidence as presented is insufficient to support 'significantly outperforms' as a reproducible claim. Thus I keep the conditional verdict: the method is plausible, but acceptance should require a clean validation protocol or code release with validation-based selection. I am not accusing the authors of misconduct; the protocol written in §4.4 is simply not adequate for the strength of the claim.","tokens_in":11809,"tokens_out":6627,"duration_ms":66959,"concrete_test":"Hold out a validation split from PNG (or use a separate subset of the 8,533 test images only for final evaluation). Select image size, DDIM steps, and guidance scale using validation AR; then run the selected model once on the full test set and compare to 67.1. If the resulting overall AR is still at least 67.1 (ideally close to 69.7), the SOTA claim survives; if it drops below 67.1, test-set tuning is the cause and the paper's headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 states that the ablation for image size, DDIM steps, and guidance scale 'was conducted on a subset of the test set, comprising approximately 7% of its total size.' These are the settings used in the headline Table 2 result (1024×1024, 50 DDIM steps, guidance 7.5). Selecting hyperparameters on a test subset invalidates the reported 69.7 overall AR as an unbiased estimate; the margin over the prior best method, EIPA+MLMA at 67.1, is only 2.6 points, and Tables 3–5 show the metric changes by multiples of that across choices (e.g., image size 512 gives overall 32.7; 1024 gives 71.0 on the subset). The paper provides no validation split, no fixed pre-registered configuration, and no released code to check whether the selected settings were the only ones tried. Under standard evaluation protocol for PNG, all hyperparameters must be selected on training/validation data, and the test set used at most once. As written, the central quantitative assertion is not independently verifiable and may be inflated by selection bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GS (Generative Segmentation), a framework that treats language-driven segmentation as a generative task by diffusing directly in the label space. Given an input image and a noun phrase, a U-Net conditioned on both the image latent (via channel-wise concatenation) and text features (via token-level adapter injection and cross-attention) denoises a latent representation of the segmentation mask. Training uses a standard noise-prediction objective with conditional dropout for classifier-free guidance; inference applies CFG and then decodes/interpolates the predicted latent. The method is evaluated on Panoptic Narrative Grounding (PNG), where Table 2 reports an overall Average Recall of 69.7, claimed to be a new state of the art over the previous best EIPA+MLMA at 67.1. The paper also includes ablations on image size, DDIM steps, and guidance scale, plus qualitative visualizations.","tokens_in":12024,"tokens_out":3807,"duration_ms":38503,"significance":"If the reported numbers withstand a clean evaluation, the paper would make a useful conceptual contribution: it reframes segmentation as a primary generative objective rather than an auxiliary use of image diffusion, and it demonstrates end-to-end training without panoptic segmentation pretraining. The dual-branch conditioning mechanism (spatial concatenation plus token-level text injection) is plausible, and the idea of generating masks directly from noise is interesting and clearly presented. The main strength is the novelty of the formulation; the main weakness is that the central empirical claim is not yet supported by a valid evaluation protocol, and one load-bearing architectural assumption about the VAE latent space is not validated. The paper does not provide machine-checked proofs or reproducible code within the manuscript, so the reported performance numbers are the only support for the SOTA claim.","major_comments":[{"comment":"The ablation study is run on a subset of the test set comprising approximately 7% of its size, and the settings selected there (1024×1024 image size, 50 DDIM steps, guidance 7.5) are exactly the settings used for the headline result in Table 2. Selecting hyperparameters on the test set invalidates the 69.7 overall AR as an unbiased estimate of test performance, and the 2.6-point margin over EIPA+MLMA is within the range of plausible selection effects: Table 3 alone shows a 6.4-point drop when moving from 1024 to 768 resolution, and a 38.3-point drop from 1024 to 512. Please re-select hyperparameters on a validation split (or pre-register a fixed configuration before touching the test set), report the resulting test-set AR, and if the margin changes materially, temper the SOTA claim accordingly.","section":"§4.4, Tables 3–5"},{"comment":"The clean signal x0 is assumed to be a 1-channel segmentation mask in the latent space of the SDXL VAE, which was trained on RGB images with a 4-channel latent. No experiment or reconstruction analysis demonstrates that binary masks are faithfully representable in this latent geometry. The paper itself notes in Fig. 4 that VAE decoding 'introduces undesired artifacts' and that bilinear interpolation of the latent is used for the final mask, which conflicts with §3.5's statement that x0 is decoded and then resized. Please either provide round-trip reconstruction statistics for masks through the SDXL VAE, or clearly state that final masks are obtained by interpolating the denoised latent without VAE decoding, and update the inference description in §3.5 accordingly.","section":"§3.3, Eq. (2), §3.5, Fig. 4"},{"comment":"The supplementary text says that source code and deployment instructions are provided, but the manuscript as submitted contains no code, checkpoints, or explicit definition of which split is used for the reported test numbers. Given that the central claim is an empirical performance number and the evaluation protocol uses test-data tuning, release of code and a precise statement of the validation/test split are necessary for independent verification. Please make the code available and describe the split and evaluation pipeline in enough detail that the reported AR can be recomputed.","section":"§4 and Appendix A"}],"minor_comments":[{"comment":"The heading 'Appliable Efficiency' contains a typo; it should be 'Applicable Efficiency' or simply 'Efficiency'.","section":"§4.5"},{"comment":"The relationship between the predicted latent and the final binary mask should be specified precisely: the text mentions decoding x0, while Fig. 4 compares bilinear interpolation, nearest-neighbor interpolation, and VAE decoding, with the caption indicating that bilinear interpolation is preferred. Please clarify which operation is actually used in the reported quantitative results.","section":"§3.5 and Fig. 4"},{"comment":"The dataset description says there are 133,103 training images and 8,533 validation images, but the paper repeatedly refers to a 'test set' and says images in the figures 'belong to the test set.' Please state explicitly whether the official PNG validation split is used as the test set, and whether any separate held-out test split exists.","section":"§4.1"},{"comment":"The phrase 'our method without pretraining outperforms other pretrained counterparts' is potentially misleading because the model is initialized from a pretrained SDXL inpainting backbone; what is meant is likely 'without panoptic segmentation pretraining on COCO.' Please clarify this in the text and table caption.","section":"Table 2 and §4.3"},{"comment":"The claim that higher resolution will 'substantially enhance' the things metric is speculative, since only one resolution (1024×1024) is tested as the maximum and the trend could saturate; please soften this statement or add experiments at a larger resolution.","section":"§4.3 and Table 3"}],"recommendation":"major_revision","confidential_remarks":"The conceptual contribution is interesting and the paper is clearly written, but the central SOTA claim rests on test-set hyperparameter selection and on an unvalidated assumption about the SDXL VAE's ability to represent binary masks. I would ask the authors to re-run the evaluation with hyperparameters chosen on a validation split and to either validate or remove the VAE-decoding assumption. If the clean evaluation still shows a margin of roughly 2.6 AR over EIPA+MLMA, the paper would be a reasonable candidate for acceptance; if the margin shrinks or reverses, the SOTA claim must be removed. The absence of code in the submission also makes it difficult to verify the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is the first paper I've seen that runs a text-conditioned diffusion process directly in label space for panoptic narrative grounding, and the headline number (69.7 AR) beats the previous best by 2.6 points. The idea is coherent and the paper is readable. Second thing: I wouldn't trust that 69.7 yet. The ablations in Section 4.4 — image size, DDIM steps, guidance — were done on a 7% slice of the test set, and the settings that won on that slice (1024x1024, 50 steps, guidance 7.5) are exactly the ones used in Table 2. That means the reported margin over EIPA+MLMA could be selection bias. The paper has no validation split, no pre-registration, and no code, despite the appendix saying code is provided.\n\nWhat is genuinely good: the dual conditioning mechanism is plausible; injecting text into self-attention via an adapter is not new, but it's applied cleanly here. The qualitative results include failed cases in the appendix, which is honest. The paper also correctly identifies that prior diffusion-based segmentation uses diffusion as feature extractor, data generator, or inversion, and that none directly generate masks. That novelty is real, though the mechanism itself predates this paper: SegDiff and similar direct mask-diffusion models are not cited. So the novelty is in the text-conditioned application and the PNG benchmark, not the generative-over-labels idea.\n\nSoft spots, in proportion: the mask-to-latent representation is underspecified. They say x0 is a 1-channel latent in SDXL's VAE space, but the SDXL VAE is trained for RGB images; they don't say how a binary mask is encoded into that space, and their own Fig 4 shows VAE decoding gives artifacts, so the final mask decoding path is unclear. No code means I can't check whether the 7% subset results generalize. And the \"things\" subcategory is actually lower than several baselines (65.6 vs 67.9 for EIPA+MLMA), so the state-of-the-art claim is really driven by stuff and plurals.\n\nWho this is for: anyone working on diffusion for structured prediction or language-driven segmentation. The paper is worth debating because the paradigm is interesting, but the evidence as presented is not enough to support the central quantitative claim. I'd send it to referees, but with a clear expectation that the evaluation protocol be fixed: hyperparameters selected on a proper validation split, test set used once, code released, and mask latent encoding/decoding specified precisely. Also, a citation to SegDiff is required. This is a conditional accept territory, not a desk reject, but it needs real revision.","headline":"Direct mask diffusion for language-driven segmentation is a real idea worth refereeing, but the SOTA claim is contaminated by test-set hyperparameter selection and missing code.","tokens_in":12559,"tokens_out":2406,"would_cite":false,"duration_ms":23310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that language-driven segmentation is best formulated as a generative problem: directly generating the mask from noise with a conditioned label-diffusion model, and reports state-of-the-art Average Recall of 69.7 on…","keywords":["label diffusion","generative segmentation","panoptic narrative grounding","language-driven segmentation","classifier-free guidance","latent diffusion models","text-to-mask generation"],"falsifier":"Encode ground-truth PNG masks with the SDXL encoder and immediately decode them, then measure pixel-wise IoU against the originals; if autoencoding visibly distorts the masks, the reported Average Recall is not measuring faithful segmentation and the central claim weakens. A complementary check is to replace VAE decoding of the denoised latent with direct bilinear upsampling of a soft prediction and quantify the Average Recall gap, separating the generative signal from the decode step.","tokens_in":11560,"feed_emoji":"🧩","tokens_out":8158,"duration_ms":67637,"temperature":0.7,"pith_summary":"This paper proposes GS, a framework that treats language-driven segmentation as a generative problem: instead of classifying pixels or extracting attention maps from an image diffusion model, a U-Net denoises a segmentation mask from Gaussian noise, conditioned on the input image and a noun-phrase description. The authors claim that this 'label diffusion' reversal is not just viable but superior, reporting 69.7 Average Recall on Panoptic Narrative Grounding against 67.1 for the best prior method, EIPA+MLMA, without panoptic segmentation pretraining. If correct, the result would establish that dense structured prediction can be achieved by modeling the label distribution itself with a conditioned diffusion process, with the label space rather than the image as the primary generative target. The paper also shows that performance improves with larger generation resolution and more DDIM sampling steps, and that classifier-free guidance strengthens text control.","feed_headline":"Mask-from-noise beats pixel classification for language segmentation","feed_subtitle":"GS's label diffusion hits 69.7 Average Recall on panoptic narrative grounding with no panoptic pretraining.","key_machinery":"The load-bearing object is the conditional label-space diffusion process: a forward noising schedule applied to a one-channel latent mask and a reverse denoising U-Net $\\Phi$ whose input is the channel-wise concatenation of the noisy label $x_t$ and the image latent $\\varepsilon(I)$. This machinery makes the mask itself the generative variable and lets the same VAE geometry that encodes images carry the label signal, with text injected through cross-attention and multi-scale adapter tokens. Classifier-free guidance combines conditional and unconditional noise predictions at inference to sharpen textual control.","core_discovery":"GS directly generates segmentation masks from noise in the latent space of a variational autoencoder, conditioned jointly on the image and on text. The clean signal $x_0$ is a one-channel latent encoding of the ground-truth mask, and the noisy label map is concatenated channel-wise with the four-channel image latent and fed to a Stable Diffusion XL denoising U-Net; text enters twice, through global CLIP embeddings with cross-attention and through token-level adapter features injected into self-attention. Training uses the standard noise-prediction objective with conditional dropout, and inference applies classifier-free guidance. The paper reports that this end-to-end generative pipeline achieves state-of-the-art Average Recall on the PNG benchmark, 69.7 overall, with leading scores on stuff (76.5), singulars (71.3), and plurals (63.7), while the things subcategory lags and is attributed to limited latent resolution.","pith_inferences":["Beyond the reported experiments, the same label-diffusion formulation could be pointed at other dense prediction targets such as depth, edges, or saliency by swapping the one-channel target while keeping image-text conditioning; the paper remarks on generality but does not test it.","The unvalidated use of the SDXL VAE for masks is independently testable: if mask autoencoding is lossy, some of the measured gain may come from the bilinear post-processing rather than from the generative formulation itself.","The supplement's failure cases, such as segmenting nearly the whole scene as 'bench', look like hallucination modes of generative sampling; stronger spatial conditioning or consistency terms may be needed before the approach is reliable for small or ambiguous phrases."],"forward_implications":["Language-driven segmentation can be cast as conditional mask synthesis, so neither panoptic pretraining on COCO nor handcrafted post-processing is required to reach competitive performance.","Resolution is a first-order factor: in the paper's ablation, raising generation size from 512 to 1024 lifts overall Average Recall from 32.7 to 71.0.","More DDIM sampling steps give finer masks (50 steps: 71.0; 20 steps: 68.7), so inference cost can be traded against accuracy.","Classifier-free guidance with text dropout transfers from image generation to mask generation and is needed for strong conditioning.","Because training is end-to-end on masks, the formulation can in principle apply to other language-driven segmentation tasks beyond the PNG benchmark."],"supporting_citations":[{"why":"Supplies the DDPM forward and reverse noising equations that GS transfers to mask latents.","marker":"(Ho, Jain, and Abbeel 2020)"},{"why":"Provides the Stable Diffusion XL latent diffusion backbone and VAE whose latent space carries both the image condition and the mask signal.","marker":"(Rombach et al. 2022)"},{"why":"Introduces classifier-free guidance, which GS uses to amplify text conditioning at inference.","marker":"(Ho and Salimans 2022)"},{"why":"Defines the Panoptic Narrative Grounding benchmark, its dataset, and the Average Recall metric used for evaluation.","marker":"(González et al. 2021)"},{"why":"Supplies the token-level adapter injection into self-attention that GS uses for multi-scale text conditioning.","marker":"(Choi et al. 2024)"},{"why":"EIPA+MLMA, the strongest prior baseline in the Average Recall comparison that GS reports beating.","marker":"(Li et al. 2024)"},{"why":"ODISE, a diffusion-based panoptic segmentation baseline that represents the feature-extraction approach GS compares against.","marker":"(Xu et al. 2023)"}],"fun_headline_variants":["Label diffusion: masks generated, not classified, for panoptic grounding","GS turns segmentation into generation, hits 69.7 AR on PNG","From noise to mask: generative segmentation beats discriminative baselines","No panoptic pretraining, just direct mask diffusion: 69.7 AR","Mask diffusion outperforms pixel classification on narrative grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a binary segmentation mask can be faithfully represented as a one-channel clean signal in the Stable Diffusion XL variational autoencoder's latent space, so that decoding the denoised latent with the VAE decoder yields a usable mask; the paper adopts this without validating the reconstruction fidelity.","fun_headline_variants_meta":{"raw":{"variants":["Label diffusion: masks generated, not classified, for panoptic grounding","GS turns segmentation into generation, hits 69.7 AR on PNG","From noise to mask: generative segmentation beats discriminative baselines","No panoptic pretraining, just direct mask diffusion: 69.7 AR","Mask diffusion outperforms pixel classification on narrative grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001065,"raw_usage":{"total_tokens":4480,"prompt_tokens":975,"completion_tokens":3505,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":3415}},"tokens_in":591,"tokens_out":3505,"duration_ms":21404,"temperature":1.0,"reasoning_tokens":3415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:48:49.531625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Encode ground-truth PNG masks with the SDXL encoder and immediately decode them, then measure pixel-wise IoU against the originals; if autoencoding visibly distorts the masks, the reported Average Recall is not measuring faithful segmentation and the central claim weakens. A complementary check is to replace VAE decoding of the denoised latent with direct bilinear upsampling of a soft prediction and quantify the Average Recall gap, separating the generative signal from the decode step.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the token-level adapter injection into self-attention that GS uses for multi-scale text conditioning."}],"review_version":1}