{"id":"e550fb4a-1b20-48c3-ab63-32297cc1d420","arxiv_id":"2506.07986","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TACA scales cross-modal attention logits by a timestep-dependent temperature to rebalance text and visual tokens, improving T2I-CompBench alignment on FLUX and SD3.5.","lead":"This paper proposes a temperature scaling trick for cross-modal attention in text-to-image diffusion transformers, showing it improves prompt-image alignment on T2I-CompBench. The method adds almost no parameters and works with FLUX and SD3.5 when paired with LoRA fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The suppression mechanism in Eq. 7–9 is unsupported: no evidence that the pretrained model fails to compensate with larger cross-modal logits, so the causal claim behind TACA is not established.","rationale":"The reader's verdict is CONDITIONAL, and the weakest assumption they flag is precisely the logit-scale premise of Eq. 7–9. I agree. If the suppression mechanism is real, the paper's contribution is a principled, parameter-free fix. If it is not real, the method may still improve T2I-CompBench scores, but the paper's central 'rethinking' claim would be overstated, and the improvement would be an empirical heuristic. This is the most load-bearing concern because it undermines the explanatory narrative that the title and abstract emphasize, and because it is testable. The paper does provide some support: Fig. 4 shows visual-text attention is lower than typical cross-attention, Table 4 shows an ablation separating LoRA-only from TACA+LoRA on a subset, and the authors disclose limitations. However, none of this establishes the logit-scale assumption. The lack of error bars and the selection of γ0 and t_thresh on the same benchmark are secondary concerns; they could be addressed by reporting confidence intervals and a held-out split, but they do not go to the mechanism. Thus the paper should remain CONDITIONAL: the mechanism claim needs verification, or the claims should be softened to empirical observations.","tokens_in":18295,"tokens_out":10784,"duration_ms":124972,"concrete_test":"For a set of 100 T2I-CompBench prompts, extract Q and K from several layers/heads of pretrained FLUX.1-Dev (and SD3.5-Medium) at timesteps below and above t_thresh. For each visual query, compute the mean logits s_vt and s_vv, then compute the actual text-attention mass M = Σ_{j∈txt} e^{s_vt_j} / (Σ_{j∈txt} e^{s_vt_j} + Σ_{k∈vis} e^{s_vv_k}). Compare M to the equal-logit baseline N_txt/(N_vis+N_txt) ≈ 0.11. If M is substantially larger (e.g., >0.3), the model has compensated via larger cross-modal logits, and the suppression mechanism in Eq. 7–9 is not the cause of any misalignment. Also verify that applying γ=1.2 increases M in the intended direction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that token imbalance (N_vis ≫ N_txt) suppresses visual-text attention, and that TACA's temperature scaling rebalances it. This claim rests on Eq. 7–9, which approximate the softmax denominator for a visual query by the visual-visual sum and conclude P_vis-txt is much smaller than in typical cross-attention. The approximation is valid only if the unnormalized logits s_vt and s_vv are of comparable scale. The paper never measures these logits. Fig. 4 shows the resulting attention mass averaged over 50 samples, which conflates the effect of token count with the model's learned logit scales. If the pretrained network has compensated by making s_vt larger than s_vv, then the low text-attention mass is an intended allocation, not a structural suppression, and TACA's gains would be a heuristic re-weighting rather than a correction of a diagnosed defect. The comparison to 'typical cross-attention' is also not a controlled baseline: in MM-DiT, visual tokens must attend to visual keys, so a lower text mass is expected and may be optimal. Without a direct logit measurement, the mechanism story is unsupported, and the benchmark improvements (which combine TACA with LoRA) cannot be attributed to the proposed rebalancing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies two alleged defects in the attention mechanism of Multimodal Diffusion Transformers (MM-DiTs): suppression of cross-modal attention caused by the larger number of visual tokens relative to text tokens, and lack of timestep-aware weighting of cross-modal interactions. To address these, it proposes Temperature-Adjusted Cross-modal Attention (TACA), which multiplies visual-text attention logits by a timestep-dependent temperature factor gamma(t) (with gamma(t)=gamma0 for early denoising steps and 1 otherwise), optionally combined with LoRA fine-tuning to suppress artifacts. The method is evaluated on T2I-CompBench for FLUX.1-Dev and SD3.5-Medium, with ablations on gamma0 and tthresh, image-quality metrics, and a user study. The central claim is that TACA rebalances cross-modal attention and significantly improves text-image alignment with minimal computational overhead.","tokens_in":18586,"tokens_out":6475,"duration_ms":72414,"significance":"If the mechanism claims were fully supported, TACA would be an attractive practical contribution: it adds no trainable parameters, requires only a few lines of code around the attention computation, is compatible with two current state-of-the-art MM-DiT families, and the authors provide public code and implementation details (including a speed comparison in the supplementary material). The main tables show consistent gains on several T2I-CompBench attribute categories, and the qualitative examples are often compelling. However, the paper's causal story, which is the basis for its title and framing, rests on an unverified assumption about the scale of unnormalized attention logits, and the benchmark evaluation suffers from selection-on-evaluation and a LoRA confound. The empirical effects are therefore not yet convincingly attributable to the proposed rebalancing mechanism, although the method may still work as a heuristic.","major_comments":[{"comment":"The suppression mechanism is not yet supported by direct evidence. The argument that N_vis >> N_txt suppresses P_vis-txt assumes that the unnormalized logits s_vt and s_vv are of comparable scale, but the paper never measures these logits; Fig. 4 shows only the resulting attention mass averaged over 50 samples, which conflates the token-count effect with any learned rescaling of s_vt. If the pretrained network has compensated by learning larger s_vt values, then the low text-attention mass in the baseline MM-DiT is an intended allocation, and TACA's gains would be a heuristic reweighting rather than a correction of a diagnosed defect. Please report direct statistics of s_vt and s_vv (e.g., mean and variance per layer and per timestep group) and, ideally, a controlled experiment that varies the token counts while holding the semantic content fixed. In addition, the comparison to 'typical cross-attention' in Fig. 4 is not a controlled baseline: in MM-DiT a visual token must attend to all visual keys, so a lower text-attention mass than a pure text-key softmax is expected and may be optimal rather than pathological.","section":"Section 3.2, Eqs. 7-9"},{"comment":"The hyperparameters gamma0=1.2 and tthresh=970 are selected by ablation on T2I-CompBench prompts (Table 4, 100 prompts per attribute) and then evaluated on the same benchmark in Table 1. This selection-on-evaluation setup can inflate the reported gains; a hold-out split or nested evaluation is needed. In addition, Table 1 reports single-seed scores without error bars or significance tests, and several differences are very small (e.g., FLUX.1-Dev Non-Spatial 0.3035 to 0.3041 for r=64; SD3.5-Medium Complex 0.4441 to 0.4470 for r=64). Please provide multiple seeds or bootstrapped confidence intervals for the main quantitative claims.","section":"Section 4.3, Table 4 and Section 4.2, Table 1"},{"comment":"The benchmark improvements are reported for TACA combined with LoRA, not for TACA alone. The abstract and introduction attribute the gains to TACA, but the only training-free evidence is qualitative (Figs. 5 and 8), and the only LoRA-only quantitative result is on the 100-prompt subset in Table 4. Consequently, the specific contribution of the temperature scaling over LoRA fine-tuning is not isolated on the full benchmark; note that the LoRA-only baseline in Table 4 already improves Color and Texture over the frozen model. Please report full-benchmark results for training-free TACA, LoRA-only, and TACA+LoRA under the same evaluation protocol.","section":"Section 4.2, Table 1 and Section 4.3, Table 4"}],"minor_comments":[{"comment":"Please clarify whether the 10K LAION image-text pairs used for LoRA training overlap with the T2I-CompBench evaluation prompts; the fixed seed and default Diffusers settings are helpful but do not rule out distributional leakage.","section":"Section 4.1"},{"comment":"The timestep-insensitivity claim is motivated by a single qualitative example; a quantitative curve such as CLIP similarity or attention mass per denoising step, as in Fig. 8b, would make the claim more convincing.","section":"Fig. 3 and Section 3.2"},{"comment":"The reference to 'Fig. ??' for the long-prompt results is unresolved; please fix the cross-reference.","section":"Supplementary Section B.3"},{"comment":"The LLaVA reference cited in Section 4.1 is missing the year and venue; please update the entry.","section":"References, [24]"},{"comment":"The user study reports percentages but not raw counts or confidence intervals; with 25 prompts and 50 participants, a binomial confidence interval would help assess the strength of the preference.","section":"Table 3, User Study"},{"comment":"Some image-quality differences are at the fourth decimal (e.g., FLUX MUSIQ 0.7186 to 0.7212); please state whether these differences are within the metrics' run-to-run variability.","section":"Table 2, Image Quality"}],"recommendation":"major_revision","confidential_remarks":"The paper proposes a simple, practical, and potentially useful method, and the code release is a plus. The main risk is that the mechanism story in Section 3.2 is not backed by direct logit measurements, and the evaluation has a selection-on-evaluation issue plus a LoRA confound. These are fixable with additional experiments rather than being fundamental, so I recommend major revision over rejection. If the authors can provide logit statistics and a cleaner evaluation protocol, the paper would be significantly stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical upshot: this paper proposes a near-zero-cost modification to MM-DiT attention—multiply the visual-to-text logits by a temperature factor gamma(t) that is active in the first few denoising steps—and shows it improves T2I-CompBench scores by a few points on FLUX and SD3.5, especially when combined with LoRA. If you work on text-to-image alignment, this is worth knowing about; the method is simple, the code is public, and the sensitivity analysis suggests it is not overly finicky.\n\nWhat is actually new: temperature scaling for cross-modal attention in an MM-DiT, with a timestep-dependent schedule, is not in the cited U-Net attention-modification literature. The paper also does a decent job of explaining why token imbalance might dilute text guidance, and Fig. 4 shows the effect on attention mass. The ablations over gamma0, tthresh, and CFG scale are useful, and the authors disclose two limitations up front.\n\nWhere it gets soft: the mechanistic claim in Eq. 7–9 assumes the unnormalized logits for visual-visual and visual-text interactions are comparable in scale. The paper never measures those logits directly. If the pretrained model has simply learned larger cross-modal logits, then the suppression story is wrong, and TACA is a heuristic reweighting that happens to help. The comparison to “typical cross-attention” is also not a controlled baseline, because in MM-DiT visual tokens necessarily attend to many visual keys, so lower text mass is expected and may even be optimal. The stress-test note lands. On the empirical side, the main tables have no error bars or significance tests, and gamma0 is tuned on a subset of the same benchmark used for the headline results. That is a mild selection-on-evaluation issue, though Table 5’s flat sensitivity narrows the concern. The improvements are a few points, not a step change.\n\nAll that said, the central empirical claim—that TACA, with LoRA, improves compositional alignment on two modern models—holds up better than the theory. The mechanism is plausible, and the results are consistent across ablations and a user study. I would want to see logit measurements and bootstrap confidence intervals before fully trusting the causal explanation, but the method itself deserves a serious referee.\n\nFor the reading group, I'd say maybe—good for a discussion of attention modifications, less essential if you're not in text-to-image. I would cite it if I were working on attention-level interventions in diffusion transformers. Send it to peer review; expect the reviewers to push for more rigorous statistics and a direct test of the logit-scale assumption.","headline":"TACA is a simple, plausibly useful temperature-scaling tweak for MM-DiT attention, but the paper's causal story about token imbalance is not backed by direct logit measurements; the empirical gains are real but moderate.","tokens_in":19107,"tokens_out":1740,"would_cite":true,"duration_ms":22258,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temperature scaling of cross-modal attention rebalances text and image tokens to improve prompt following.","keywords":["multimodal diffusion transformer","text-to-image alignment","cross-modal attention","temperature scaling","denoising timestep","LoRA fine-tuning","compositional generation","attention suppression"],"falsifier":"Measure the average visual-text attention probability $P^{(i,j)}_{vis-txt}$ in a frozen MM-DiT model over many prompts and denoising steps, and compare it with the probability a typical cross-attention layer would give for the same text tokens. If early-step cross-modal attention is not substantially lower than the typical cross-attention baseline once logit scales are taken into account, then token imbalance is not suppressing cross-modal attention and TACA's performance would be a heuristic artifact rather than a fix for the diagnosed mechanism.","tokens_in":18135,"feed_emoji":"🌡️","tokens_out":5907,"duration_ms":59908,"temperature":0.7,"pith_summary":"Multimodal diffusion transformers power today's leading text-to-image models, yet these models often omit or misplace objects described in the prompt. The paper traces this failure to two properties of the unified attention mechanism: the large number of image tokens dilutes the attention image tokens pay to text tokens, and the attention weights do not adapt to the phase of denoising. It proposes Temperature-Adjusted Cross-modal Attention (TACA), which multiplies visual-text attention logits by a scalar $\\gamma(t)$ that is active early in denoising and turns off later, so text guidance dominates during layout formation. Paired with a small LoRA fine-tune to remove artifacts, TACA improves compositional alignment on T2I-CompBench for both FLUX.1-Dev and SD3.5-Medium with almost no added computation.","feed_headline":"A single temperature scalar improves text-image alignment in diffusion models","feed_subtitle":"A gamma=1.2 boost during early denoising lifts spatial accuracy by up to 28 percent on FLUX and SD3.5.","key_machinery":"The load-bearing object is a single scalar temperature coefficient $\\gamma(t)$ inserted into the unified softmax of MM-DiT. It multiplies only the visual-text logits, leaving visual-visual and text-text logits untouched, so it acts as a signal booster for text guidance. The threshold $t_{thresh}$ makes the boost conditional on denoising phase, matching the observation that global layout is fixed in the first steps. A lightweight LoRA adapter on the attention layers compensates for the distribution shift the boosted logits introduce; TACA itself adds no learnable parameters.","core_discovery":"The paper's central claim is that the semantic misalignment of MM-DiT models is caused by suppression of visual-to-text attention inside a single softmax over concatenated image and text tokens, together with timestep-insensitive query-key weighting. Because $N_{vis}\\gg N_{txt}$ (e.g., 4096 visual versus 512 text tokens for a 1024$\\times$1024 FLUX image), the denominator of the softmax is dominated by visual-visual terms, so $P^{(i,j)}_{vis-txt}$ is much smaller than it would be in a typical cross-attention layer. TACA counteracts this by replacing $s^{vt}_{ij}$ with $\\gamma(t)\\,s^{vt}_{ij}$ (Eq. 10), where $\\gamma(t)=\\gamma_0>1$ for early denoising steps $t\\ge t_{thresh}$ and $\\gamma(t)=1$ afterwards (Eq. 11). With LoRA fine-tuning to restore the real-image distribution, the paper reports consistent gains on T2I-CompBench: for FLUX.1-Dev, relative gains of 16.4% in spatial relationships and 5.9% in shape, and for SD3.5-Medium, 28.3% in spatial relationships and 2.9% in shape.","pith_inferences":["The same token-imbalance argument should apply more strongly to video MM-DiTs, where the visual token count is much larger relative to text; the paper notes training-free text-to-video experiments showed gains but LoRA training diluted them, so a video-specific schedule for $\\gamma(t)$ is a natural extension.","A per-layer or per-head temperature, instead of one global $\\gamma_0$, could let the model boost exactly the heads whose cross-modal attention is most suppressed, at the cost of a few more hyperparameters.","One can test the mechanism directly: measure the mean of unnormalized logits $s^{vt}$ versus $s^{vv}$ in a frozen pretrained model; if $s^{vt}$ already compensates for the token count, the suppression premise would fail and the success of TACA would be heuristic.","A calibration rule could replace grid search: set $\\gamma_0$ proportional to the observed ratio $N_{vis}/N_{txt}$ or to the measured attention suppression at early steps."],"forward_implications":["If TACA's diagnosis is right, any future MM-DiT model can improve prompt adherence by rebalancing cross-modal logits rather than by adding cross-attention modules or changing the tokenizer.","The gains are largest on spatial relationships and attribute binding, the exact cases where missing or misbound objects appear in baseline samples.","Because $\\gamma(t)$ is active only in the early denoising steps, the method does not slow down the full sampling loop much; the selective recomposition implementation runs a 30-step 1024$\\times$1024 generation in about 16 seconds versus 14 seconds unmodified.","TACA is complementary to classifier-free guidance: the ablations show it improves scores at multiple CFG scales, including the default scales of both tested models."],"supporting_citations":[{"why":"Defines the MM-DiT unified attention mechanism over concatenated text and image tokens, the architecture whose behavior the paper analyzes.","marker":"[10]"},{"why":"Supplies the T2I-CompBench benchmark and its attribute, object-relationship, and complex-prompt metrics used for all quantitative alignment results.","marker":"[16]"},{"why":"Provides the FLUX.1-Dev baseline model and its 4096/512 token ratio used in the suppression analysis.","marker":"[20]"},{"why":"Supplies the LoRA adaptation used to fine-tune attention layers and remove artifacts from temperature scaling.","marker":"[15]"},{"why":"Provides flow matching, the training objective with which the LoRA fine-tuning loss in Eq. 13 is formulated.","marker":"[23]"}],"fun_headline_variants":["One scalar rebalances cross-modal attention in diffusion","Timestep-aware temperature improves text-image alignment","Fix visual-token overwhelm to align diffusion output","TACA: temperature scaling lifts spatial accuracy","Visual-text attention suppression: a simple cure"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the unnormalized visual-text and visual-visual logits have comparable magnitudes, so the larger number of visual tokens mechanically pushes down cross-modal attention; if the pretrained network has learned larger visual-text logits that compensate, the proposed suppression mechanism would not be the true cause of misalignment.","fun_headline_variants_meta":{"raw":{"variants":["One scalar rebalances cross-modal attention in diffusion","Timestep-aware temperature improves text-image alignment","Fix visual-token overwhelm to align diffusion output","TACA: temperature scaling lifts spatial accuracy","Visual-text attention suppression: a simple cure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1782,"prompt_tokens":1048,"completion_tokens":734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":665}},"tokens_in":664,"tokens_out":734,"duration_ms":9602,"temperature":1.0,"reasoning_tokens":665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:20:17.111817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the average visual-text attention probability $P^{(i,j)}_{vis-txt}$ in a frozen MM-DiT model over many prompts and denoising steps, and compare it with the probability a typical cross-attention layer would give for the same text tokens. If early-step cross-modal attention is not substantially lower than the typical cross-attention baseline once logit scales are taken into account, then token imbalance is not suppressing cross-modal attention and TACA's performance would be a heuristic artifact rather than a fix for the diagnosed mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FLUX.1-Dev baseline model and its 4096/512 token ratio used in the suppression analysis."}],"review_version":1}