{"id":"76203b24-ac90-4e37-9805-b88c6e7fb9be","arxiv_id":"2607.27386","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Diffusion language models are not inherently more robust than autoregressive models to natural input noise, but they are consistently overconfident and resist short gradient-based suffix attacks.","lead":"Two diffusion language models (LLaDA-8B, Dream-7B) were stress-tested against matched autoregressive models (LLaMA-3-8B, Qwen2.5-7B) under typos, deletions, and adversarial suffixes. Everyday noise robustness proved model-specific, while overconfidence and resistance to short gradient-based suffix attacks replicated across both diffusion models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Linear-probe evidence is correlational; abstract overstates it as 'proving... entirely' without any causal intervention","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the mechanistic probe inference in §6.5 is correlational and is inappropriately upgraded to a causal 'proving... entirely' claim in the abstract and conclusion. This is the single point on which the paper's central claim depends. If the probe evidence does not support the encoder/decoder attribution, then the IPM failure is merely one failed pre-generation mitigation strategy, not evidence for a decoding-stage bottleneck. I also flag the T_d inconsistency as a secondary but concrete issue that weakens the IPM interpretation. The paper has genuine empirical value — paired architecture comparison, 32 noise conditions, calibration ablations, a falsifiable IPM test — but the headline mechanistic claim needs causal support (activation patching or equivalent). Since the reader's CONDITIONAL verdict already reflects this need, my stress-test does not change the verdict; hence UNCHANGED.","tokens_in":22649,"tokens_out":3393,"duration_ms":31499,"concrete_test":"Run activation patching on LLaDA-8B for the transposition condition: at the layer reported as peak probe accuracy (Table 4), replace the residual-stream states of the corrupted token positions with the corresponding clean-token states from the clean prompt, then continue the DLM's standard decoding; measure accuracy on the same n=200 sample. If accuracy recovers toward the clean baseline, the encoded corruption is causally upstream of the behavior; if it remains at the noisy-baseline level, the decoder-failure hypothesis is supported. Perform the reverse patch (clean states replaced by corrupted states) to test sufficiency. Both directions are needed to rule out the possibility that the probe feature is epiphenomenal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion — that DLMs 'perfectly encode input corruption' and that behavioral fragility is 'entirely a decoder routing failure' — rests on the linear-probe result in §6.5. That result only demonstrates that a logistic regression can separate hidden states of corrupted tokens from clean tokens with >0.93 accuracy. Since corrupted and clean tokens are different surface forms with different token IDs and embeddings, a linear probe trivially separates them; it does not establish that the model encodes 'corruption' as a causally usable feature, nor that the decoder fails to use it. The paper itself states in §6.5 that the probe is correlational and that 'causally localizing the failure (e.g., via activation patching) is important future work,' yet the abstract and conclusion upgrade this to 'proving... entirely.' This is an internal inconsistency: the evidence cannot bear the weight of the claim. The IPM capstone (§6.6) does not rescue it without the causal link — IPM failure shows only that the five tested prompt-patching variants fail, not that the bottleneck is necessarily in the decoding loop. Additionally, there is a configuration inconsistency: §3.1 specifies T_d=64 for IPM, while Appendices B and E use T_d=16; if IPM was run only with T_d=16, its failure could be due to insufficient denoising rather than to the locus of fragility. This must be resolved before the central claim is accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a paired benchmark study of DLM robustness and calibration against AR models, using LLaDA-8B vs. LLaMA-3-8B and Dream-7B vs. Qwen2.5-7B across 32 natural perturbation conditions, a custom gradient-based adversarial probe (D-GCG), and hidden-state linear probing. It reports that natural-noise robustness is weight-dependent rather than architectural, that DLMs are systematically overconfident, that DLMs are resistant to short gradient-based suffix attacks, and that linear probes can separate clean from corrupted hidden states at >0.93 accuracy in all models. The paper proposes a decoding-stage bottleneck explanation and shows that several input-patching (IPM) variants fail to improve over noisy baselines, concluding that DLM robustness must be integrated into the decoding loop rather than patched onto the input.","tokens_in":23045,"tokens_out":5755,"duration_ms":54358,"significance":"If the central claims hold, the paper would make a valuable contribution by shifting DLM robustness research toward decoding-loop interventions and by providing a two-pair experimental template for isolating architecture from weights. The paired design across two independently trained model families is a genuine strength, as is the calibration analysis, which is shown in Appendix I to survive several reasonable definitions of confidence. The Dream pair serves as a useful negative control for the 'diffusion is inherently robust' hypothesis, and the statistical treatment of the RDI comparisons is solid. The paper is also transparent about its limitations and releases code, which supports reproducibility. However, the headline mechanistic claim — that corruption is 'perfectly encoded' and that behavioral fragility is 'entirely a decoder routing failure' — goes beyond the correlational evidence provided and is not supported by the IPM negative result alone. These issues are load-bearing for the paper's central conclusion and require either additional causal evidence or careful rewording.","major_comments":[{"comment":"The headline causal claim overreaches the evidence. §6.5 explicitly states that the probe is correlational and that causally localizing the failure via activation patching is future work, yet the abstract and conclusion assert that 'all models perfectly encode input corruption' and that fragility is 'entirely' a decoder routing failure. A >0.93 linear-probe accuracy shows that clean versus corrupted token hidden states are linearly separable; because perturbed tokens are different surface forms, the separation may reflect token identity rather than a corruption feature. This distinction is load-bearing because the 'cannot be patched on' conclusion and the interpretation of the IPM negative result both depend on it. Please either add an activation-patching/intervention study or rephrase the abstract and conclusion to say the results are consistent with a decoding-stage bottleneck, not pro","section":"Abstract; §6.5; §7"},{"comment":"The IPM denoising configuration is inconsistent. §3.1 states T_d=64; Algorithm 1 in Appendix B and Table 7 in Appendix E specify T_d=16, and the complexity estimate in Appendix B uses T_d=16. If Table 5 was run with T_d=16, the negative IPM result could be due to an under-powered infilling pass rather than to a decoder bottleneck. In addition, Appendix G reports that the Dream-pair IPM numbers in Table 5 come from an N=20 run, which is too small to support the general claim that none of the five tested IPM variants outperform the noisy baseline. Please state the exact T_d used for each reported result and report the Dream-pair numbers with an adequate sample size or clearly mark them as pilot results.","section":"§3.1; Appendix B; Appendix E; Table 5"},{"comment":"The IPM capstone is a negative result about five specific prompt-editing strategies, not positive evidence for a decoder routing failure. Even if the linear probe did establish that the encoder represents corruption, the failure of IPM to improve accuracy is consistent with several alternatives: the corruption detector may be inaccurate, the re-masking may destroy useful context, or the infilling pass may not be able to reconstruct the original prompt. The 'falsifiable hypothesis' framing in the contributions assumes the very conclusion it is supposed to test. Please either add an intervention that directly tests decoder routing (e.g., patching decoder hidden states) or present IPM only as a null result for the tested pre-generation denoising methods.","section":"§6.6; §7"},{"comment":"The D-GCG probe estimates gradients from a single sampled timestep and mask (Eq. 6). The claim that DLM loss trajectories 'oscillate erratically' and that this reflects an incoherent loss landscape is confounded by the variance of the estimator itself: with one timestep/mask sample per step, the loss and gradient are random, and an AR model would also show noise under the same estimation scheme. The resistance to adversarial suffixes is demonstrated for a narrow regime (suffix length 4–8, 50 steps), and the paper does acknowledge this, but the abstract's phrasing ('strongly resist gradient-based adversarial suffixes') is stronger than the evidence. Please add a control (e.g., multiple timestep samples or a stochastic-loss baseline for the AR models) or soften the claim to 'resistant under the tested short-suffix, single-sample estimator.'","section":"§4; §6.4; Eq. (6)"}],"minor_comments":[{"comment":"The illustrative text contains typos ('What ofthe cpatialis France?') that appear intentional for the noise example, but the figure would be clearer if the clean and corrupted texts were labeled and the artifact of the caption were removed.","section":"Figure 1"},{"comment":"The 'Adv.%' column shows the percentage of conditions where the DLM beats its AR counterpart, but the main text also reports RDI advantage values. Please state whether 'Adv.%' counts ties as wins for the DLM or not.","section":"Table 1 and §6.2"},{"comment":"The DLM confidence definition uses per-token maximum softmax probabilities; however, for AR greedy decoding max equals chosen-token probability, as noted in Appendix I. Please make this explicit in the main text to avoid confusion.","section":"§3.3, Eq. (4)"},{"comment":"The perturbation conditions are evaluated on n=200 samples, but Appendix G reports N=20 for the Dream-pair IPM run and N=100/50 for question-only ablations. Please add a table of sample sizes per experiment in the main text.","section":"§6.1"},{"comment":"The reference list includes 'Zhang et al., 2019' BERTScore in Appendix J; the in-text citation is in the limitations, which is fine, but the reference is not alphabetized correctly (it appears after Zhang et al., 2020).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical effort is substantial and the paired design is a good template for DLM robustness evaluations. The main issue is that the abstract and conclusion state a causal mechanism that the evidence (a correlational linear probe and five failed input-patching variants) cannot support. The T_d inconsistency and the small Dream-pair IPM sample are concrete and fixable. I would be open to accepting a revised version that either adds a causal intervention or carefully downgrades the mechanistic claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real value is the paired empirical benchmark: two matched DLM/AR pairs, 32 natural noise conditions, calibration ablations, and a falsifiable IPM test. The weight-dependent robustness finding is solid — LLaDA beats LLaMA across the board while Dream is no better than Qwen — and the systematic DLM overconfidence result replicates across both pairs and survives the confidence-definition ablation. The D-GCG probe is a reasonable extension, and the claim of resistance to short gradient-based suffixes is honestly scoped. These results will be useful to anyone working on DLM robustness or calibration.\n\nThe soft spot is the mechanistic headline. Section 6.5 reports a linear probe with >0.93 accuracy and explicitly calls it correlational, saying causal localization via activation patching is future work. Yet the abstract and conclusion say the probe \"proving that behavioral fragility stems entirely from the bidirectional decoder.\" That is an internal inconsistency. Linear separability is not causal encoding, and since corrupted and clean tokens are different surface forms, the probe may be separating token identities rather than a corruption feature the decoder could use. The IPM failure does not fix this: it only shows five prompt-patching variants fail, not that the bottleneck is necessarily in the decoding loop. The T_d inconsistency (64 in Section 3.1 and 6.6, 16 in Appendices B and E) needs to be resolved — if IPM ran only at 16 steps, its failure could just as easily be insufficient denoising. Also, the D-GCG and IPM sections lack error bars and no code link is provided despite the promise to release code.\n\nDespite these issues, the empirical core is credible and the paper is honest in its limitations, question-only ablation, and definition stability checks. It deserves a serious referee, not a desk reject. I'd recommend sending it to peer review with the expectation of substantial revision: either add a causal intervention (activation patching or similar) or soften the mechanistic claim to \"consistent with\" a decoding-stage bottleneck, and fix the T_d inconsistency. The benchmark alone justifies citing it.","headline":"Useful paired DLM robustness benchmark, but the abstract and conclusion overclaim a causal decoding-stage bottleneck that the paper's own linear-probe evidence does not support.","tokens_in":23445,"tokens_out":1987,"would_cite":true,"duration_ms":22437,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion language models fail at noisy prompts because their decoder cannot route around corruption, even though their internal states encode the corruption perfectly.","keywords":["diffusion language models","natural noise robustness","calibration","linear probing","decoder routing failure","adversarial suffix attacks","input masking","iterative decoding"],"falsifier":"An activation-patching experiment: take a noisy input where the model errs, replace only the decoder-side hidden states (or attention outputs) with those from the clean-input run, and check whether the error disappears. If behavior stays wrong, the bottleneck is not decoder routing; if it is fixed, the decoding-stage story is causally confirmed. Alternatively, train or fine-tune a DLM with a noise-robust decoding objective and see whether natural-noise RDI actually drops.","tokens_in":22566,"feed_emoji":"🤖","tokens_out":3703,"duration_ms":31361,"temperature":0.7,"pith_summary":"This paper argues that diffusion language models (DLMs) fail on noisy prompts not because they fail to perceive the corruption, but because their iterative decoder cannot route around it. Using two parameter-matched DLM/autoregressive model pairs across 32 natural noise conditions, the authors show that natural-noise robustness is weight-dependent rather than architecture-intrinsic: one DLM is far more robust than its AR counterpart, while the other is not. Two traits do replicate across both pairs: DLMs are systematically overconfident, and they resist short gradient-based suffix attacks because their loss landscape is highly stochastic. The load-bearing mechanistic claim is that linear probes recover corrupted-token information from hidden states with over 0.93 accuracy in every model, so behavioral fragility is consistent with a decoding-stage bottleneck rather than an encoding failure. From this diagnosis, the paper predicts that pre-generation prompt patching cannot help, and confirms that none of five input-masking variants improve over the noisy baseline. The conclusion is that DLM robustness must be trained into the decoding loop, not patched onto the input.","feed_headline":"Noise is encoded perfectly; diffusion models fail at decoding","feed_subtitle":"With two DLM/AR model pairs, the paper traces noisy-prompt errors to a decoding-stage bottleneck, not perception.","key_machinery":"The key machinery is the linear-probe diagnostic combined with the paired model design. A logistic-regression probe on hidden states from 17 layers classifies corrupted vs. clean tokens; its high accuracy (>0.93) is taken as evidence that corruption is linearly separable in the model's representations, so the encoder 'perceives' the noise. This probe, together with denoising-trajectory divergence for DLMs and corruption-attention-ratio analysis for AR models, localizes the behavioral difference to decoding rather than encoding. The D-GCG adversarial probe (a single-timestep gradient estimator) is the tool used to characterize DLM loss-landscape stochasticity, and the Input Denoising via Part","core_discovery":"The central discovery is that when a DLM answers incorrectly under input noise, it is not because the model failed to register the corruption: a simple linear classifier reads corrupted-token status from internal hidden states with above 0.93 accuracy in both DLMs and both AR baselines, with near-identical AUROC across architectures. The paper interprets this as isolating the failure to the decoder's routing—the bidirectional decoder sees the noise but fails to quarantine it during iterative unmasking. This diagnosis is used to explain why input-side fixes (masking suspected tokens and re-infilling) do not help, and why robustness must be integrated into the iterative decoding process itself","pith_inferences":["The 'perfectly encode' claim rests on linear separability; a nonlinear probe or activation-patching experiment could show that the encoder is not as cleanly blame-free. Causally ablating corrupted-token representations would separate encoding and decoding contributions in a way the paper's correlational probe cannot.","The overconfidence result suggests that iterative-commitment decoders need explicit uncertainty-aware training objectives—such as calibrating the commitment threshold against downstream correctness—which the paper does not explore.","Weight-dependent natural robustness hints that training-data composition, not the diffusion objective, determines everyday reliability; a controlled pretraining study varying noise-augmentation would test this directly.","If decoder routing failure is the true bottleneck, then decoding-loop interventions—such as remasking low-confidence response tokens mid-generation or biasing unmasking order away from corrupted positions—become concrete candidates to test next."],"forward_implications":["If fragility is a decoding-stage bottleneck, input-side denoising techniques will not restore DLM accuracy; the five IPM variants tested all fail to beat the noisy baseline.","DLM confidence scores are not trustworthy as reliability signals: ECE gaps persist under every non-degenerate confidence definition, so deployment needs post-hoc calibration before selective prediction.","Natural-noise robustness cannot be assumed from the diffusion architecture; one parameter-matched pair shows a large DLM advantage, the other shows none, so robustness must be audited per model.","Short gradient-based suffix attacks are ineffective against DLMs because the training objective marginalizes over masks and timesteps, yielding large but directionally incoherent gradients—a natural property, not an explicit defense.","The diffusion objective's stochastic loss landscape is the mechanism behind both the adversarial resistance and the overconfidence, tying these robustness properties to training-time noise marginalization."],"fun_headline_variants":["Diffusion models see noise clearly but can't route it away","DLM failures are decoder-side, not perception: noise is encoded","Patching prompts won't fix diffusion models; change the loop","DLM overconfidence: robustness must be built into decoding"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The conclusion that behavioral fragility lies 'entirely' in decoding rests on the inference that a linear probe's >0.93 accuracy proves the encoder perfectly represents the corruption; the paper itself notes this probe is correlational, and causal localization via activation patching remains future work.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion models see noise clearly but can't route it away","DLM failures are decoder-side, not perception: noise is encoded","Patching prompts won't fix diffusion models; change the loop","DLM overconfidence: robustness must be built into decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1204,"prompt_tokens":750,"completion_tokens":454,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":383}},"tokens_in":494,"tokens_out":454,"duration_ms":22289,"temperature":1.0,"reasoning_tokens":383,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T08:18:34.431637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An activation-patching experiment: take a noisy input where the model errs, replace only the decoder-side hidden states (or attention outputs) with those from the clean-input run, and check whether the error disappears. If behavior stays wrong, the bottleneck is not decoder routing; if it is fixed, the decoding-stage story is causally confirmed. Alternatively, train or fine-tune a DLM with a noise-robust decoding objective and see whether natural-noise RDI actually drops.","supporting_citations":[],"review_version":1}