{"id":"b36a4987-c6ba-4b74-8622-2252dd0b3b9c","arxiv_id":"2501.06751","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Padding tokens in text-to-image models can carry semantic information or act as diffusion-time registers, depending on training and attention architecture.","lead":"Padding tokens, the filler words used to make prompts a fixed length, can change what a text-to-image model draws. In some models they carry hidden meaning or act as extra memory, and whether they do depends on the model's training and attention design.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IDP does not control for FLUX's pooled CLIP embedding, so the claimed 'padding-as-registers' effect in Table 3 may be caused by the untouched global text vector rather than by padding tokens.","rationale":"The paper's typology rests on two pillars: trained text encoders storing information in pads (ITE) and frozen MM-DiT models using pads as diffusion-time registers (IDP). The first pillar is supported by LDM and LLaMA-UNet results and is less affected by my concern. The second pillar, which justifies the 'registers' claim and the architecture-level generalization to SD3 and FLUX, hinges on Table 3. FLUX and SD3 inject pooled CLIP embeddings as global conditioning; IDP as described patches token positions only. Unless the released code also ablates or replaces the pooled vector, the Pads-vs-Clean contrast cannot isolate pad-token information. This is more specific than the reader's cross-model confound: it is a possible artifact inside the method itself. The paper deserves credit for releasing code and data and for using multiple convergent metrics, but the ambiguity is central and directly testable. I therefore keep the verdict CONDITIONAL rather than REJECT, with the explicit condition that the pooled-embedding control be reported; if the code already patches the pooled vector, the concern is resolved and the register interpretation would stand.","tokens_in":12705,"tokens_out":12650,"duration_ms":138975,"concrete_test":"Inspect the released IDP implementation to determine whether the pooled CLIP text embedding (the global conditioning vector in FLUX/SD3) is replaced by the clean-pads pooled vector in the 'pads' condition. If it is not, rerun Table 3 for FLUX with prompt-contextual pad tokens but the clean-pads pooled vector; if the prompt-CLIP score drops to the 'clean' level (about 0.10), the register interpretation is unsupported. Report the same check for SD3.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"IDP's causal inference for the MM-DiT 'register' scenario (Section 3.2) is confounded by FLUX's pooled text embedding. The method replaces token vectors before each attention block, but FLUX conditions the transformer on a pooled CLIP text vector (a global conditioning embedding) that is not part of the token sequence and is not mentioned as being patched. In the 'pads' condition, the full-prompt pooled vector is therefore still available to the diffusion model. If so, the higher prompt-CLIP score of 'pads' (0.22) over 'clean' (0.10) in Table 3 is exactly what a retained full-prompt pooled vector would produce; it does not demonstrate that information is written into, or read from, padding tokens. The same omission threatens SD3, which also combines frozen CLIP and T5 encoders. This is an internal-validity problem, not only a cross-model confound: it targets the causal interpretation of IDP itself, not just the attribution to architecture.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies padding tokens in text-to-image (T2I) diffusion models, proposing two activation-patching interventions: ITE, which swaps token segments in the text encoder output, and IDP, which swaps token representations before each diffusion attention block. Across six models and 5,000 images per condition, it reports three regimes: padding tokens are effectively ignored in frozen-CLIP cross-attention models (SD2, SDXL); they become semantically meaningful when the text encoder is trained or fine-tuned (LDM, LLaMA-UNet); and they act as diffusion-stage 'registers' in MM-DiT models (SD3, FLUX). The central claim is that padding tokens are active computational elements rather than a neutral batch-processing convenience.","tokens_in":12901,"tokens_out":7299,"duration_ms":72699,"significance":"If the claims hold, this is a useful mechanistic contribution with practical implications for interpreting, editing, and designing T2I models. The work has real strengths: a large intervention study across six models, 5,000 images per configuration, standard causal-patching methodology, a combination of quantitative metrics (CLIP score, KID) with attention-map and qualitative evidence, and publicly released code. The proposed three-way taxonomy of padding behavior is a valuable organizing framework. However, the mapping from behavior to architecture/training is observational across heterogeneous checkpoints, and the IDP method has a plausible internal-validity confound involving FLUX's pooled CLIP text embedding that must be resolved before the diffusion-level 'register' claim can be accepted.","major_comments":[{"comment":"The IDP experiment does not appear to control for FLUX's pooled CLIP text embedding. Appendix C states that FLUX uses a frozen CLIP encoder in addition to T5, and in current FLUX implementations the CLIP encoder also outputs a global pooled text vector used as a conditioning signal. The paper describes replacing token representations before each attention block at every diffusion step, but it never states whether this pooled vector is replaced, masked, or held constant in the 'Pads' versus 'Clean' conditions. If the full-prompt pooled vector is still fed to the transformer in the 'Pads' condition, then the higher prompt-CLIP score for 'Pads' (0.22) relative to 'Clean' (0.10) in Table 3 can be explained without any information being written into padding tokens. This is an internal-validity problem for the causal interpretation of IDP, not merely a cross-model attribution issue, and the same concern should be addressed for SD3, which also combines frozen CLIP and T5 encoders.","section":"§3.1, §3.2, Table 3"},{"comment":"The paper attributes padding behavior to 'frozen versus trained' text encoders and 'cross-attention versus MM-DiT' architectures by comparing six pre-existing models, but these factors covary with model family, parameter scale, training data, distillation status (FLUX-Schnell is a distilled checkpoint), and release version. This is not a controlled comparison. For example, LDM and SD2 differ not only in whether the text encoder is trained but also in the text encoder architecture, the diffusion backbone, and the training dataset. The claims in Contributions 2 and 3 should therefore be rephrased as correlational observations among the studied checkpoints, or supported by at least one controlled ablation that varies a single factor, such as fine-tuning a frozen CLIP encoder on a T2I objective or removing MM-DiT self-attention.","section":"§2.3, Fig. 5, Table 1"},{"comment":"The clean-pad representation Eclean is used as the counterfactual baseline for both ITE and IDP, but the manuscript does not validate that Eclean is distributionally matched to the padding positions of Efull. In Efull, the padding tokens are encoded in the context of the prompt tokens; in Eclean, all positions are padding tokens. Replacing prompt-contextual pads with clean pads may therefore introduce a distribution shift rather than a pure removal of prompt-related information. The authors should provide a sanity check for this assumption, for example by showing that Eclean and the padding positions of Efull are close in representation space in the frozen-encoder models, or by including a second neutral baseline. This is load-bearing because the causal interpretation of both intervention methods depends on Eclean being an appropriate control.","section":"§2.1, Eqs. (2)–(3)"}],"minor_comments":[{"comment":"Figure 5 presents only point estimates, while the standard deviations are relegated to Table 5 in the appendix; the main figure should display error bars or confidence intervals because the text makes claims such as 'nearly identical' and 'significantly lower' that depend on the spread of the distributions.","section":"Fig. 5"},{"comment":"The Introduction refers to 'FLUX 2' in the list of MM-DiT models, while Appendix C states that the experiments use FLUX-Schnell; please clarify which model release is actually analyzed.","section":"Introduction vs. Appendix C"},{"comment":"In the paragraph on LDM and Lavi-Bridge, the sentence 'images generated from full prompt tokens have lower scores compared to those generated using prompt representations' appears to conflate 'full prompt' and 'prompt representations'; please rephrase this comparison to match the definitions in Equations (2) and (3).","section":"§2.3"},{"comment":"The table defines 'Pads First' and 'Pads Second' only as 'First 20%' and 'Second 20%', but the text and Figure 9 do not specify whether these are 20% of the padding segment by position, and how this segmentation aligns with the ITE segment analysis in Table 2.","section":"Table 3"},{"comment":"The Limitations section should explicitly acknowledge that the architecture and training attributions are based on a small observational sample of six checkpoints with no controlled factor isolation, so that the causal framing in the abstract and conclusion is not over-read by readers.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The topic is well suited to a venue on interpretability and generative models, and the scale of the intervention study is a genuine strength. In my view, the FLUX pooled-embedding confound is the key issue: it is internal to IDP and must be addressed head-on, either by patching the pooled vector as well or by providing evidence that the result is unchanged when it is removed. I would also like the authors to either add a controlled ablation or consistently soften the causal claims about architecture and training. If these points are resolved, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a solid, useful paper on a genuinely overlooked design choice—padding tokens in T2I pipelines—and the ITE results for trained vs frozen text encoders are convincing. The IDP results for MM-DiT “registers” are more fragile than the paper admits.\n\nThe paper does a lot right. The two interventions are standard activation patching applied cleanly, the scale is decent (500 prompts, 10 seeds per configuration), and the code/data are released. The attention-map and no-padding qualitative outputs add convergent evidence. The three-scenario taxonomy is a genuinely useful framing, and the finding that LDM and LLaMA-UNet push real semantic content into pads is new and actionable.\n\nThe main soft spot is the pooled embedding confound in IDP. FLUX conditions on a global CLIP-pooled text vector that is not part of the token sequence and is never mentioned as being swapped. If that vector stays from the full prompt in the “pads” condition, the higher CLIP score versus “clean” could come from it, not from padding tokens. The same risk applies to SD3, which also uses a frozen CLIP encoder. The stress-test note checks out on reading; this is a real internal-validity issue, not a false alarm. It is fixable: patch or ablate the pooled vector as well, and report both conditions.\n\nThe second weakness is the cross-model attribution. Comparing six existing models conflates architecture/training with scale, data, and distillation. That is softer because the paper frames them as “relationships,” but the title’s “mechanistic analysis” slightly overclaims. The missing error bars in Figure 5 are actually handled in Appendix F, so that is minor.\n\nWho is this for: interpretability researchers, T2I model builders, and anyone doing prompt editing or personalization. The ITE finding alone justifies a read. The register claim needs the extra control before I would build on it.\n\nRecommendation: send to peer review. The paper deserves a serious referee, but with a major-revision request: patch the pooled embeddings in IDP, validate the clean-pad baseline, and soften the causal attribution to architecture/training. The central ITE findings will likely survive; the register claim needs the extra control.","headline":"Useful mechanistic study of padding tokens in T2I, but the FLUX register claim rests on an IDP design that may not control for the frozen CLIP pooled embedding.","tokens_in":13417,"tokens_out":3943,"would_cite":true,"duration_ms":37666,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Padding tokens in text-to-image models are not inert: they can encode semantics during text encoding or act as registers during diffusion.","keywords":["padding tokens","text-to-image diffusion","activation patching","causal mediation analysis","multimodal diffusion transformer","registers","text encoder training","CLIP score"],"falsifier":"Run a controlled ablation on one architecture where text-encoder freezing and attention type are varied independently while backbone, data, and prompt set stay fixed; if padding tokens show no causal effect in the trained-encoder condition, or show register behavior in a frozen-encoder cross-attention condition, the paper's two-axis explanation is wrong.","tokens_in":12513,"feed_emoji":"🎨","tokens_out":8371,"duration_ms":72279,"temperature":0.7,"pith_summary":"This paper asks whether the padding tokens that stretch text prompts to a fixed length in text-to-image (T2I) models are inert filler or active participants in generation. It introduces two causal intervention techniques, one aimed at the text encoder and one at the diffusion process, and applies them to six T2I models. The result is a three-way taxonomy: padding tokens can carry semantic information when the text encoder is trained or fine-tuned, they can act as information-carrying registers during diffusion in multi-modal self-attention architectures, or they can be effectively ignored when a frozen CLIP encoder feeds cross-attention. The paper concludes that padding is thus a design choice with real consequences, not a batch-processing convenience.","feed_headline":"Padding tokens actively shape text-to-image outputs","feed_subtitle":"Causal tests across six models show pads carry semantics, act as registers, or stay inert.","key_machinery":"The central objects are the encoded token representations: 'prompt-contextual pads' are padding tokens encoded together with the user's prompt, while 'clean pads' come from encoding an all-padding sequence and serve as a neutral, distribution-matched baseline. The first technique, ITE (intervention in the text encoder), swaps prompt or padding segments in the text encoder's output before image generation; the second, IDP (intervention in the diffusion process), repeats the same swap before every attention block at every diffusion step. This is causal mediation via activation patching: a segment is load-bearing exactly when replacing it with clean pads measurably changes the generated image. The machinery also includes attention-map inspection, which shows whether padding tokens receive attention from image regions.","core_discovery":"The paper's central claim is that padding tokens in T2I pipelines occupy one of three roles depending on how the text encoder was trained and how the diffusion model attends to text. With a text encoder trained or fine-tuned on the image-generation objective, as in LDM and LLaMA-UNet, the pads absorb semantic content: images generated from only the prompt-contextual padding tokens score nearly as well as images from the full prompt, and the information is concentrated in the pads closest to the prompt. With a frozen text encoder feeding a multi-modal diffusion transformer, as in SD3 and FLUX, padding tokens are initially empty but accumulate information during the diffusion process and act as 'registers' that store and pass visual details to image patches. With a frozen CLIP encoder feeding cross-attention, as in SD2 and SDXL, padding tokens are effectively ignored, and replacing them with clean pads does not change the output. The paper's evidence comes from two activation-patching procedures that swap segments of token representations between the full prompt encoding and an all-pad encoding, then measure the resulting images with CLIP score and a distribution-distance metric.","pith_inferences":["A testable extension is to insert extra learnable tokens after the prompt in MM-DiT models with frozen encoders and check whether they improve fidelity the way padding registers appear to.","If trained encoders genuinely store prompt semantics in pads, token-level editing and personalization methods may need to intervene on padding tokens rather than only on prompt tokens.","The finding that information concentrates in early pads suggests variable-length or pad-count-reduced training could change model behavior, and is worth studying as a deliberate design choice.","The LoRA-scaling result implies that pad usage is tied to how strongly a model is adapted, which could serve as a diagnostic for how much of a fine-tuned model's behavior lives outside its prompt tokens."],"forward_implications":["Models with trained text encoders should be expected to change output when padding tokens are removed or replaced, so padding should be treated as part of the prompt representation in those systems.","In MM-DiT models such as SD3 and FLUX, dropping padding tokens should reduce visual detail and coherence, since pads act as registers that store information during diffusion.","In frozen-CLIP cross-attention models such as SD2 and SDXL, padding tokens can be safely ignored or masked, with little effect on the generated image.","Pad position matters: the first padding tokens carry more information than later ones, so pad order is a design parameter rather than an arbitrary convention.","Fine-tuning strength, as reflected in the LoRA scaling factor, controls how many padding tokens a model actually uses, making padding usage a continuous knob during adaptation."],"supporting_citations":[{"why":"Supplies the causal mediation analysis foundation that both ITE and IDP build on.","marker":"Imai et al., 2010"},{"why":"Introduces activation patching for neural NLP, the intervention style the paper adapts to T2I.","marker":"Vig et al., 2020"},{"why":"Provides the registers concept for vision transformers that the paper invokes to explain padding-token behavior in MM-DiT models.","marker":"Darcet et al., 2024"},{"why":"Describes Stable Diffusion 3 and the MM-DiT architecture that defines one of the paper's three padding scenarios.","marker":"Esser et al., 2024"},{"why":"Defines LDM, one of the two trained-text-encoder models in which padding tokens encode semantics.","marker":"Rombach et al., 2022"},{"why":"Defines Lavi-Bridge with LLaMA-UNet, the other trained-text-encoder model in which padding tokens carry semantic content.","marker":"Zhao et al., 2024"},{"why":"Defines SDXL, one of the frozen-CLIP cross-attention models in which padding tokens are effectively ignored.","marker":"Podell et al., 2024"},{"why":"Provides CLIP score, the main metric used to measure how well generated images align with prompts.","marker":"Hessel et al., 2021"},{"why":"Provides KID, the distribution-distance metric used to compare intervention images with full-prompt images.","marker":"Bińkowski et al., 2018"},{"why":"Supplies the Parti benchmark prompts from which the paper derived its 500 test prompts.","marker":"Yu et al., 2022"}],"fun_headline_variants":["Padding tokens are not inert: they shape T2I outputs in three ways","Causal study: padding tokens carry semantics, act as registers, or are ignored","Padding tokens in T2I: determined by architecture and training, not neutral","Padding tokens shape image generation depending on encoder and attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's load-bearing premise is that its six off-the-shelf models differ in the ways that matter, text-encoder freezing and attention architecture, and not in other ways such as scale, training data, distillation, or checkpoint version.","fun_headline_variants_meta":{"raw":{"variants":["Padding tokens are not inert: they shape T2I outputs in three ways","Causal study: padding tokens carry semantics, act as registers, or are ignored","Padding tokens in T2I: determined by architecture and training, not neutral","Padding tokens shape image generation depending on encoder and attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2487,"prompt_tokens":957,"completion_tokens":1530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1451}},"tokens_in":573,"tokens_out":1530,"duration_ms":10656,"temperature":1.0,"reasoning_tokens":1451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:50:15.105123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled ablation on one architecture where text-encoder freezing and attention type are varied independently while backbone, data, and prompt set stay fixed; if padding tokens show no causal effect in the trained-encoder condition, or show register behavior in a frozen-encoder cross-attention condition, the paper's two-axis explanation is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the causal mediation analysis foundation that both ITE and IDP build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the registers concept for vision transformers that the paper invokes to explain padding-token behavior in MM-DiT models."}],"review_version":1}