{"id":"a40bd8f2-3fcd-46d1-9684-016e501f4809","arxiv_id":"2411.13619","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"NCIS, which samples outliers from a Stable Diffusion embedding space using a conditional volume-preserving network, achieves state-of-the-art OOD detection on CIFAR-100 and ImageNet-100.","lead":"This paper introduces NCIS, a method that generates synthetic out-of-distribution images by sampling from a diffusion model's own embedding space and fitting class-specific shape models. It sets new accuracy records on two standard OOD detection benchmarks and shows that simple image resizing choices can strongly affect detector performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The near-OOD boundary assumption in Section 3.4 is unvalidated: low-likelihood samples from the cVPN-fitted Gaussian may map back to embeddings that Stable Diffusion renders as arbitrary corruptions, rather than boundary images, making the SOTA claim depend on an untested geometric property.","rationale":"The reader's conditional verdict identifies the diffusion-space semantic structure and the low-likelihood-to-boundary mapping as the weakest assumption. I agree with that identification, and the concern is load-bearing because the entire method is a pipeline in which each component's benefit is only meaningful if the final generated outliers actually sit at the ID/OOD boundary. The paper's ablations show that each component helps, but they do not directly validate the geometric assumption that low-likelihood points in invariant space invert to boundary embeddings. The 'Both' resizing augmentation and the choice of lambda are tuned on the same benchmarks, so the empirical SOTA claim is somewhat confounded. However, the existing ablations (Table 3) do show large gains from the diffusion embeddings and the cVPN even under fixed bilinear resizing, which gives independent support that the outlier quality matters. Thus the concern does not rise to rejection; it warrants a specific quantitative validation of the near-boundary property before the mechanism can be fully accepted. The verdict should remain conditional, pending such a check.","tokens_in":14992,"tokens_out":9727,"duration_ms":101523,"concrete_test":"Use a pretrained feature extractor (e.g., CLIP or a classifier trained on the ID data) to compute OOD scores (Mahalanobis distance or energy) for four sets on CIFAR100: (a) real ID test images, (b) real OOD test images, (c) NCIS-generated outliers, and (d) corrupted versions of ID images (e.g., Gaussian noise, random crops). If the NCIS outliers do not have scores intermediate between ID and true OOD, and instead overlap with arbitrary corruptions, the near-boundary claim in Section 3.4 is false. Report the score distributions for the default lambda = 1e-5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of NCIS is that sampling a low-likelihood point v' from the class-conditional Gaussian (Eq. 8) in the invariant space and mapping it back through the inverse cVPN f^{-1}(v', y) (Section 3.4) yields an embedding e' lying in the ID/OOD boundary region of the diffusion conditioning space, so that Stable Diffusion produces useful near-OOD images. This requires two unverified properties: (i) the fitted density p_v accurately models the true distribution of diffusion embeddings, and (ii) the off-manifold behavior of the volume-preserving inverse is semantically meaningful, i.e., large invariant coordinates correspond to semantic OOD attributes rather than arbitrary directions in text-embedding space. The training objective (Eq. 4) only constrains f to map ID embeddings to near-zero invariants; it places no constraint on f^{-1} outside the training set. Consequently, e' can lie far outside the training embedding distribution in ways the CLIP text encoder has never seen, and Stable Diffusion may emit images that are corruptions or unrelated categories rather than near-OOD samples. The paper's evidence for the near-boundary property is qualitative (Fig. 5) and indirect (ablations, Table 3); no quantitative measure of the semantic distance between generated outliers and the true ID boundary is provided. If this assumption fails, the reported FPR95 improvements could stem from the resizing augmentation or from any generic distributional shift, rather than from non-linear outlier synthesis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NCIS, a method for out-of-distribution (OOD) detection that generates synthetic outliers using Stable Diffusion. Each training image is embedded into the diffusion conditioning space by taking three gradient steps of the noise-prediction loss starting from the class label embedding (Algorithm 1, Eq. (1)). A class-conditional volume-preserving network (cVPN) maps these embeddings into an invariant space where class-conditional Gaussians are fitted (Eqs. (4), (8)). Low-likelihood samples are drawn from these Gaussians, mapped back through the inverse cVPN, and rendered by Stable Diffusion into synthetic OOD images, which are then used to regularize a classifier with an energy-based OOD loss (Eqs. (12)-(13)). Experiments on CIFAR-100 and ImageNet-100 report state-of-the-art FPR95/AUC, with ablations of the embedding, the sampling distribution, the resizing strategy, and hyperparameters, plus experiments across architectures and outlier counts.","tokens_in":15386,"tokens_out":4500,"duration_ms":45868,"significance":"If the empirical claims hold under independent validation, this is a strong contribution to synthetic-outlier OOD detection: the reported gains over Dream-OOD are large on CIFAR-100, the method generalizes across architectures, the code is publicly available, and the analysis of resizing/interpolation sensitivity is a useful and somewhat surprising finding. The cVPN contribution is a natural extension of the authors' prior NL-Invs work and is clearly motivated. However, the central state-of-the-art claim currently rests on test-set hyperparameter selection and on copied baseline numbers without variance, and the geometric assumption that low-likelihood invariant-space samples map to useful near-OOD images is not quantitatively validated.","major_comments":[{"comment":"The hyperparameters lambda and p are selected by evaluating FPR95 on the same CIFAR-100 OOD test sets that are used in the main comparison of Table 1. Since lambda directly controls the 'degree of out-of-distributionness' of the generated outliers (Section 3.3), tuning it on the test OOD distribution can substantially inflate the reported average FPR95 of 10.84. Please either select these hyperparameters on a held-out validation split and then evaluate once on the test benchmarks, or report results for a fixed default choice across all benchmarks, clearly stating which numbers are the result of test-set selection.","section":"Section 5.2, Fig. 6"},{"comment":"All baseline numbers are copied from Dream-OOD [12] with no error bars, while NCIS results are averaged over three seeds. On ImageNet-100 the reported improvement over Dream-OOD is only 4.87 FPR95 (33.89 vs. 38.76), which is within the plausible run-to-run variance of a single training run. The claim of state-of-the-art on ImageNet-100 therefore needs either multiple seeds for the baselines under the same codebase, or at least a statement of Dream-OOD's reported variance if it exists in [12].","section":"Tables 1 and 2"},{"comment":"The method's core assumption is that a low-likelihood point v' in the invariant space, when mapped back through the inverse cVPN, lands near the ID/OOD boundary of the diffusion conditioning space. However, Eq. (4) only constrains the forward map on ID embeddings; the inverse map f^{-1} is unconstrained outside the training support. The paper provides only qualitative examples (Fig. 5) and indirect ablations (Table 3) as evidence. Please add a quantitative validation of the near-boundary property, for example by measuring the distance of generated embeddings e' to the ID embedding manifold, or by comparing the effect of NCIS-generated outliers with that of random corruptions or random perturbations of ID embeddings under the same resizing and training protocol.","section":"Section 3.4, Fig. 5"},{"comment":"The text calls the embedding e a maximum-a-posteriori estimate, but the algorithm initializes e at the label embedding and performs only three gradient steps on the first term of Eq. (1). No evidence is provided that three steps approximate the MAP solution, nor is the iteration count ablated. Since the quality of all downstream modeling depends on these embeddings, please either show convergence behavior (e.g., loss curves or embedding drift over steps) or explicitly state that three steps is a heuristic and measure its sensitivity.","section":"Section 3.1, Algorithm 1"},{"comment":"The rejection sampling procedure is not specified: the paper says 'we apply rejection sampling in the invariant space to obtain an outlier v' from the low-likelihood regions' but does not define the acceptance threshold, the quantile, or the number of samples tried. This is a load-bearing step of the generation pipeline and must be described precisely for reproducibility.","section":"Section 3.4"}],"minor_comments":[{"comment":"The regularization term R(e, ey) appears in the objective but is never explicitly defined; the text later states it is implemented by initialization to ey, but the form of R should be stated or the notation should be removed to avoid confusion.","section":"Eq. (1)"},{"comment":"The row label 'Both cVPN Ours' is cryptic; please clarify that 'Both' refers to combining both resizing augmentations with the cVPN-based sampling.","section":"Table 3"},{"comment":"The sentence 'all other baselines are unaffected by the interpolation method' is asserted without supporting evidence; if retained, please provide the corresponding experiment or cite a table.","section":"Section 5.2"},{"comment":"The per-architecture results are reported without error bars or the number of seeds; adding these would strengthen the claim of robustness across architectures.","section":"Table 4"},{"comment":"The x-axis label 'p (number of invariants)' is confusing because p is described earlier as a variance threshold; please clarify the relationship between p and the resulting number of invariants K.","section":"Fig. 6(b)"}],"recommendation":"major_revision","confidential_remarks":"The main concern for a journal decision is the test-set hyperparameter selection in Fig. 6 combined with copied baselines without variance; this is fixable but requires rerunning with proper validation. The near-boundary assumption is also under-validated, but I would not reject on that basis because the empirical method may still work even if the interpretative story is incomplete. The paper builds heavily on the authors' own NL-Invs [10], but the adaptation to supervised OOD detection is sufficiently different. Scope is appropriate for a computer vision venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on arXiv:2411.13619 (NCIS). It deserves a serious look. The core idea—embedding images directly in Stable Diffusion's conditioning space via a few gradient steps on the noise-prediction loss, then fitting class-conditional manifolds with a conditional volume-preserving network and sampling low-likelihood points—is a genuine step beyond Dream-OOD and the authors' own NL-Invs. The reported gains are large: FPR95 drops from 40.31 to 10.84 on CIFAR100 and from 38.76 to 33.89 on ImageNet100, with 3-seed variance for the new method. The ablations in Table 3 are convincing: each component (diffusion embedding, cVPN, interpolation augmentation) contributes, and the interpolation finding is a useful practical insight about OOD detectors leaning on low-level statistics.\n\nThe soft spots, in increasing order of concern. First, baseline numbers are copied from [12] without error bars; the margin over Dream-OOD could shift if the comparison were rerun in-place. The authors say they use the same codebase and settings, so this is defensible but leaves some uncertainty. Second, λ and p are tuned on the same test benchmarks used for the headline results (Fig. 6). The curves are fairly flat around the chosen values, so this is not disqualifying, but it weakens the claim of robustness. Third, the 'MAP estimate' of the embedding is really three gradient steps from the label embedding. The paper does not show how performance changes with more steps, so that term does more rhetorical work than the evidence supports.\n\nOn the stress-test concern about off-manifold inverse behavior: I think it is overblown. The paper does not prove that low-likelihood samples land exactly on the ID/OOD boundary, but the qualitative examples and the ablation table show the cVPN representation matters beyond any generic distribution shift. The resizing effect is real, but the full method beats the nearest-neighbor baseline by a wide margin regardless. The mechanism is plausible: the cVPN maps ID embeddings to near-zero invariants, so moving along invariant directions should yield out-of-class but not arbitrary embeddings. Whether that always holds semantically is open, but it is not a load-bearing flaw.\n\nWho should read this: anyone working on synthetic outlier generation, OOD detection, or diffusion-model conditioning. It deserves a serious referee—an expert should verify the setup and push for a re-run of Dream-OOD under identical conditions, plus a gradient-step ablation. I would accept it for peer review.","headline":"Solid empirical outlier-synthesis paper with real gains and honest ablations; the main risks are borrowed baselines and test-set hyperparameter tuning, not the unvalidated geometry.","tokens_in":15860,"tokens_out":2854,"would_cite":true,"duration_ms":26436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NCIS sets new out-of-distribution detection records by generating outliers in a diffusion model's conditioning space and fitting class-conditional manifolds with a volume-preserving network.","keywords":["out-of-distribution detection","diffusion models","synthetic outlier synthesis","conditional volume-preserving network","energy-based OOD scoring","Stable Diffusion","CIFAR-100","ImageNet-100"],"falsifier":"Train the same classifier with outliers generated by the cVPN and with outliers from a plain Gaussian fitted directly in diffusion space, holding all other hyperparameters fixed; if the FPR95 gap is not attributable to the cVPN, the non-linear manifold claim is unsupported. A second decisive check is whether the three-step diffusion embeddings of held-out ID images sit closer to their own class token than to other class tokens; if they do not, the conditioning space is not semantically aligned as assumed.","tokens_in":14822,"feed_emoji":"🎯","tokens_out":11065,"duration_ms":96241,"temperature":0.7,"pith_summary":"NCIS aims to make synthetic outlier images that sit right on the boundary of a classifier's known classes, so training on them teaches the classifier to flag unexpected inputs instead of confidently mislabeling them. The method embeds each training image directly in the diffusion model's conditioning space by a few gradient steps of the denoising objective, then fits class-conditional manifolds with a conditional volume-preserving network (cVPN). Sampling low-likelihood regions of those fitted distributions and decoding with Stable Diffusion produces near-OOD images. The paper reports this lowers the average false-positive rate at 95% true-positive rate (FPR95) by 29.47 on CIFAR-100 and by 4.87 on ImageNet-100 relative to the best previous methods. It also notes that the frozen Stable Diffusion decoder limits direct use in domains such as medical imaging.","feed_headline":"Diffusion-space outliers cut CIFAR-100 OOD errors 29 points","feed_subtitle":"NCIS also lifts ImageNet-100 detection by modeling class manifolds and sampling their edges.","key_machinery":"The load-bearing machinery is the combination of diffusion-space embeddings and a conditional volume-preserving network (cVPN). The embedding step treats each image's condition vector as a learnable token and takes three gradient steps of $\\| \\epsilon - \\epsilon_\\theta(\\hat{x}_t,t,e)\\|^2$ from the class label embedding, so the diffusion model itself defines the in-distribution region. The cVPN is a bijective, volume-preserving map $$f: \\mathbb{R}^D \\times \\mathcal{Y} \\to \\mathbb{R}^D$$ composed of orthogonal and conditional coupling layers; its first $K$ outputs are invariants that stay near zero for in-class embeddings, and the remaining outputs carry within-class variability. Because the Jacobian has unit determinant, fitting a Gaussian to $v_i = f(e_i, y_i)$ and inverting gives a non-linear class-conditional density in diffusion space, and rejection sampling its low-likelihood tail yields embeddings that decode into boundary images.","core_discovery":"The central claim is that outlier synthesis for OOD detection should happen inside the diffusion model's own conditioning space, and that the in-distribution region there is class-conditional and non-linear. NCIS derives an embedding for each training image by minimizing the diffusion noise-prediction loss with respect to a condition vector initialized at the class token, so the collection of embeddings traces where the generator would produce in-distribution images. A conditional volume-preserving network then maps each class's embeddings to an invariant space where in-class points collapse toward zero, and class-conditional Gaussians fitted there induce non-linear densities back in diffusion space. Outliers are obtained by rejection-sampling low-likelihood vectors, inverting the network, and prompting Stable Diffusion; these images are used with an energy regularizer to train the classifier. Used this way, the synthesized near-OOD images are claimed to outperform previous synthetic-outlier pipelines on CIFAR-100 and ImageNet-100, with ablations attributing the improvement to each of the design choices.","pith_inferences":["If the diffusion embedding is what carries the gain, the same recipe should transfer to other text-conditioned diffusion models: swap Stable Diffusion for a domain-specific generator and keep the cVPN module, which would address the medical-imaging limitation the authors state.","The interpolation-mismatch finding suggests a testable extension: perturbing low-level statistics of ID images during training (resizing, JPEG compression) could push the learned boundary toward semantics rather than texture, a direction the paper only begins to explore.","Because only three gradient steps are used per image, the embedding cost scales linearly with dataset size; the paper's subsampling experiment indicates a small fraction of ID embeddings suffices, so the method should scale to much larger training sets.","A stricter test of the boundary claim would be to measure how often cVPN-sampled outliers are semantically related to their conditioning class; if they are mostly unrelated, the gain might come from the energy regularization rather than from precise boundary alignment."],"forward_implications":["On CIFAR-100, NCIS lowers the average FPR95 to 10.84, a 29.47-point improvement over the best previous method, with classification accuracy largely unchanged.","On ImageNet-100, NCIS lowers the average FPR95 to 33.89, a 4.87-point improvement, and beats Dream-OOD on every OOD test set in that benchmark.","The ablation shows each component matters: diffusion-space embeddings improve on Dream-OOD's embeddings, the cVPN improves on Gaussian fitting, and matching the interpolation method between training and test time removes a large spurious drop.","The number of synthetic outliers follows a power law, so generating more outliers (up to 51,200) is a direct and large lever on detection performance.","The method transfers to different classifier backbones (ResNet, ViT, ConvNeXt), with bigger backbones giving better OOD detection as well as higher accuracy."],"supporting_citations":[{"why":"Dream-OOD is the main pixel-space baseline and supplies the training protocol and baseline numbers that NCIS must beat.","marker":"[12]"},{"why":"Introduces the volume-preserving network and non-linear invariants that the cVPN adapts to class-conditional fitting.","marker":"[10]"},{"why":"Defines data invariants, the conceptual basis for modeling in-distribution manifolds by functions that stay near zero for ID samples.","marker":"[9]"},{"why":"Stable Diffusion is the frozen text-conditioned generator whose conditioning space NCIS embeds into and decodes outliers from.","marker":"[37]"},{"why":"Textual inversion provides the prompt-tuning idea of optimizing a token embedding so the diffusion model reconstructs an image, which Section 3.1 adapts.","marker":"[15]"},{"why":"VOS is a feature-space outlier synthesis baseline that NCIS compares against and outperforms.","marker":"[11]"},{"why":"NPOS supplies a feature-space synthesis baseline and the non-parametric sampling method ablated in Table 3.","marker":"[47]"},{"why":"Energy-based OOD scoring defines the energy function and the regularization loss used to train the NCIS classifier.","marker":"[30]"}],"fun_headline_variants":["Diffusion-space outliers cut CIFAR-100 OOD errors 29 points","NCIS: diffusion-embedded outliers improve OOD detection","Class-conditional manifolds in diffusion space sharpen OOD boundaries","NCIS: sampling diffusion-space outliers improves OOD detection","Synthetic outliers from diffusion embeddings improve OOD detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Stable Diffusion's conditioning space is semantically structured, so a few gradient steps put each image's embedding near its class label and low-likelihood points in that space decode to images just outside the class rather than to random corruptions.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion-space outliers cut CIFAR-100 OOD errors 29 points","NCIS: diffusion-embedded outliers improve OOD detection","Class-conditional manifolds in diffusion space sharpen OOD boundaries","NCIS: sampling diffusion-space outliers improves OOD detection","Synthetic outliers from diffusion embeddings improve OOD detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001686,"raw_usage":{"total_tokens":6661,"prompt_tokens":905,"completion_tokens":5756,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":5670}},"tokens_in":521,"tokens_out":5756,"duration_ms":41016,"temperature":1.0,"reasoning_tokens":5670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:45:40.027428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same classifier with outliers generated by the cVPN and with outliers from a plain Gaussian fitted directly in diffusion space, holding all other hyperparameters fixed; if the FPR95 gap is not attributable to the cVPN, the non-linear manifold claim is unsupported. A second decisive check is whether the three-step diffusion embeddings of held-out ID images sit closer to their own class token than to other class tokens; if they do not, the conditioning space is not semantically aligned as assumed.","supporting_citations":[{"cited_title":"Learning non-linear invariants for unsupervised out- of-distribution detection","cited_arxiv_id":null,"evidence_quote":"Introduces the volume-preserving network and non-linear invariants that the cVPN adapts to class-conditional fitting."},{"cited_title":"Data invariants to understand unsupervised out-of- distribution detection","cited_arxiv_id":null,"evidence_quote":"Defines data invariants, the conceptual basis for modeling in-distribution manifolds by functions that stay near zero for ID samples."},{"cited_title":"An image is worth one word: Personalizing text-to-image gen- eration using textual inversion","cited_arxiv_id":null,"evidence_quote":"Textual inversion provides the prompt-tuning idea of optimizing a token embedding so the diffusion model reconstructs an image, which Section 3.1 adapts."},{"cited_title":"V os: Learning what you don’t know by virtual outlier synthesis","cited_arxiv_id":null,"evidence_quote":"VOS is a feature-space outlier synthesis baseline that NCIS compares against and outperforms."},{"cited_title":"Non- parametric outlier synthesis","cited_arxiv_id":null,"evidence_quote":"NPOS supplies a feature-space synthesis baseline and the non-parametric sampling method ablated in Table 3."},{"cited_title":"Energy-based out-of-distribution detection","cited_arxiv_id":null,"evidence_quote":"Energy-based OOD scoring defines the energy function and the regularization loss used to train the NCIS classifier."}],"review_version":1}