{"id":"f04b00e4-6cbd-4da1-8cc4-99dbddcd5735","arxiv_id":"2608.08555","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SC-Diff uses SAM3 category masks to bias self-attention inside a latent diffusion model, improving visible-to-infrared translation quality and downstream infrared detection on M3FD, FLIR, and KAIST.","lead":"This paper introduces SC-Diff, a diffusion model that translates visible-light photos into synthetic infrared images while using semantic maps to keep object shapes and locations intact. Specialists in autonomous driving and defense imaging could use it to generate extra infrared training data without collecting more thermal imagery.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SGSC's benefit hinges on unvalidated SAM3 mask labels at low attention resolutions; with λ=10, a wrong same-category label can amplify the very cross-category interference the method claims to suppress.","rationale":"The reader's weakest assumption is exactly the load-bearing point: SAM3 prompt-based masks, after resizing to 64×64 and 32×32, must correctly identify same-category token pairs for SGSC to work as intended. I agree with this and have sharpened it into a concrete, testable concern. The paper is otherwise internally coherent: the SGSC equations are well-specified given the labels, the training objective is standard, and the reported gains are consistent across three datasets and four detectors. The ablations in Table V do show that CCE and SGSC components improve FID, but this only proves the components help with the particular automatic masks used, not that the masks are reliable enough to support the claimed mechanism. A secondary technical concern is the CFG formulation in Eq. 21: the paper does not state whether SGSC is disabled in the condition-dropped forward pass, so ϵ∅ may not be truly unconditional. That issue would affect the interpretation of the guidance scales but is less central than the mask-quality assumption, because it does not directly threaten the existence of the empirical gains. No code is released, so neither concern can be settled from the manuscript alone; both are addressable with the concrete test above. I therefore keep the reader's conditional verdict unchanged.","tokens_in":17322,"tokens_out":13697,"duration_ms":154793,"concrete_test":"On a subset of FLIR and M3FD test images, compute SAM3 masks at full resolution, resize them to 64×64 and 32×32, and compare per-category IoU against available semantic or instance labels or against a stronger segmentation model. Then re-run SC-Diff with two label sources on a small split: (a) the default SAM3 labels and (b) corrected labels that maximize agreement with the reference. Compare FID/LPIPS and per-category detection AP. If per-category IoU is low but the metrics are unchanged, the claim that SGSC 'reduces cross-category interference' is not supported by the reported evidence; if corrected labels change the results materially, the method's robustness to mask quality is unestablished.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is SGSC (Eqs. 12–17), and its correctness depends entirely on token-level semantic labels ℓ^(l) at 64×64 and 32×32 being semantically correct for the generated infrared image. These labels come from SAM3 with 17 text prompts (Eq. 9) on the visible image, but the paper never reports mask quality at either resolution. At 64×64, an 8× downsampling from 512×512, small urban objects such as traffic lights, distant pedestrians, and bicycles occupy only a few tokens; at 32×32 they can disappear entirely. If a query token is mislabeled, Eq. 15 treats it as 'same category' as every other wrongly labeled token of that class, and Eq. 14 can produce β≈1, so the positive bias applied through Eq. 17 is strong (λ=10). In this case SGSC does not reduce cross-category interference; it actively strengthens it. The ablations in Table V compare with and without SGSC using the same automatic labels, so they cannot distinguish 'SGSC helps' from 'SGSC helps with these particular labels.' The paper also does not report per-category detection results, so a degradation on small-object classes would be hidden by the aggregate mAP. This is the most load-bearing concern because, if masks are unreliable, the proposed semantic calibration can be counterproductive, directly contradicting the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces SC-Diff, a latent diffusion framework for visible-to-infrared image translation. The method uses a pretrained SAM3 model with text prompts to produce category masks from the visible image; these masks are fused with the visible latent as a clean condition, and the same masks provide token-level semantic labels that calibrate self-attention in the denoising U-Net through a Semantic-Guided Self-Attention Calibration (SGSC) module (Eqs. 12–17). SGSC adds a query-dependent positive bias to same-category query–key pairs, where the bias is scaled by the entropy of the attention distribution across categories and by the original attention weight. Experiments on M3FD, FLIR, and KAIST report the best LPIPS and FID among compared methods (Table II), and augmenting the training set with translated Cityscapes images improves infrared detector mAP on M3FD and FLIR (Table IV). The paper also reports ablations of the condition-encoding and SGSC components (Table V) and visualizations of attention maps (Fig. 7).","tokens_in":17632,"tokens_out":6164,"duration_ms":57118,"significance":"If the results hold, SC-Diff is a useful contribution to detection-oriented visible-to-infrared translation: the idea of using semantic priors to regulate internal self-attention rather than only external conditioning is interesting, and the reported gains are consistent across three datasets and four detectors. The paper's strengths include explicit formulas for the calibration, clean ablations, and an out-of-distribution evaluation on Cityscapes. However, the significance of the central mechanism is qualified by the unvalidated reliability of the SAM3 masks at the attention resolutions used, the absence of error bars, and the tuning of key hyperparameters on the FLIR test set. These issues are addressable but require additional experiments.","major_comments":[{"comment":"The SGSC mechanism depends entirely on token-level semantic labels ℓ^(l) at 64×64 and 32×32 being correct for the generated infrared image. The paper asserts that these resolutions 'can still provide sufficiently reliable region guidance' (Section IV.A), but no quantitative mask-quality evaluation is provided. If a label is wrong, Eq. (14) will treat the query as well aligned with a category it does not belong to, and with λ=10 the positive bias in Eq. (17) can strengthen the very cross-category interference the method claims to suppress. The ablations in Table V use the same automatic labels for both with-SGSC and without-SGSC conditions, so they do not distinguish 'SGSC helps' from 'SGSC helps when the labels are correct.' I request: (i) report mask quality (e.g., IoU or pixel accuracy of SAM3 masks vs. ground-truth semantic labels on a subset, and after downsampling to 64×64 and 32×32); (ii) add an ablation using ground-truth semantic maps (available, e.g., on Cityscapes) to bound the benefit of SGSC under perfect labels; and (iii) report per-category detection AP, since small-object degradation would be hidden in the aggregate mAP.","section":"Section IV.A, Eq. (17)"},{"comment":"The default values of λ, c_s, and c_v are selected by FID on the FLIR test set (λ is set to 10 before Eq. (17), and c_s=2.0, c_v=1.5 are chosen in Table VI). Consequently, the reported FLIR FID of 57.66 partly reflects tuning, and the paper provides no error bars or multiple-seed runs for any dataset or table. To support the claim that SC-Diff improves perceptual quality, the authors should report means and standard deviations over at least three seeds for the main tables, and should either use a validation split for hyperparameter selection or, at minimum, show the sensitivity of the main comparisons to λ and the CFG scales on M3FD and KAIST.","section":"Section IV.E, Table VI"},{"comment":"Some baseline scores are quoted from prior publications (†) and were not reproduced under the paper's evaluation pipeline. Because the headline claim is that SC-Diff achieves the best LPIPS and FID on all three datasets, the comparison is only meaningful if all numbers are computed with the same preprocessing, resizing, and evaluation code. Please specify exactly which rows are quoted, and either reproduce these baselines with the paper's pipeline or clearly separate quoted from reproduced results in the abstract and tables. The same applies to the detection augmentation table (Table IV), where the 'overall' columns appear to be averages over four detectors; this should be stated explicitly.","section":"Table II"}],"minor_comments":[{"comment":"The definition of n^(l) as 'the number of prompted semantic categories present at layer l' is ambiguous: it should be clarified whether this is the number of categories that actually appear in the resized label map, and how categories with zero attention mass are handled.","section":"Section III.B, Eq. (13)"},{"comment":"The attention-map visualization is qualitative; a quantitative measure of cross-category attention (e.g., the fraction of attention mass assigned to non-same-category keys before and after SGSC, averaged over queries) would support the claim that SGSC reduces cross-category interference.","section":"Fig. 7"},{"comment":"It is not stated how the 'Overall' columns are computed; presumably they are averaged over the four detectors, but this should be specified in the table caption or the main text.","section":"Table IV"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its derivation and the experiments are extensive, but the mask-quality issue is real and should be addressed in revision. I would be inclined to accept after a thorough revision that includes mask-quality evaluation, a ground-truth-label ablation, error bars, and a clearer statement about tuning. The hyperparameter-on-test issue is concerning but not fatal if the method's gains hold on the other datasets with default settings. Also consider adding a limitations paragraph; the paper currently does not mention any limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is SGSC: turning the semantic map into token-level labels and using them to bias same-category attention inside the denoising U-Net, rather than just conditioning on masks externally. That is a real extension of F-ViTA and DiffV2IR, and the equations are self-consistent. The empirical story is also consistent — better LPIPS/FID on M3FD, FLIR, and KAIST, and a few mAP points on downstream detection when Cityscapes images are translated for augmentation. The gains are credible in size and direction. I buy the central claim that soft calibration of attention helps semantic consistency.\n\nThe weakest point is the SAM3 mask reliability at 64×64 and 32×32 attention resolutions. The stress-test worry is legitimate in principle: if a token is mislabeled, Eq. 14 can boost attention to the wrong same-category set, and λ=10 is not small. But the mechanism is softer than that worst case — β is suppressed when attention to the (mislabeled) own category is already high, and the original-attention modulation concentrates bias on already-strong keys. The ablations show SGSC helps with these labels, so empirically the concern does not land as a load-bearing flaw. Still, the paper never reports mask accuracy at those resolutions or per-category detection results, so a small-object failure mode could hide in the aggregate mAP. That is the first thing I would ask for in revision.\n\nThe other soft spots are standard for this area: no code, no error bars or multiple seeds, some baseline numbers quoted from prior papers, and the CFG hyperparameters were tuned on FLIR, which is also a headline test set. That last point is a mild circularity that mainly inflates confidence in the FLIR FID number; the same settings carry over to M3FD and KAIST, so the overall pattern is not an artifact.\n\nWho is this for? Anyone working on cross-modal synthesis or detection-oriented data augmentation will get a solid, citable reference for attention calibration in diffusion models. It is not a big leap, but it is clearly specified and reproducible in principle. I would send it to peer review. The referee should push for code, seeds, SAM mask quality, and per-class detection numbers before acceptance, but the paper deserves that round.","headline":"A genuine but modest mechanism that improves vis-to-IR translation and detection-oriented augmentation; the main risk is the unvalidated SAM3 masks, but empirically SGSC holds up — deserves a real referee.","tokens_in":18156,"tokens_out":2616,"would_cite":true,"duration_ms":31032,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that semantic priors placed inside self-attention, not just at the input, yield structurally faithful visible-to-infrared translation.","keywords":["visible-to-infrared image translation","latent diffusion model","semantic-guided self-attention","self-attention calibration","SAM3","infrared object detection","data augmentation","cross-category interference"],"falsifier":"Take a visible image in which SAM3 misses a small object, such as a distant cyclist, and compare SC-Diff with SGSC disabled. If the missed object's queries do not show reduced cross-category attention after calibration, or if LPIPS/FID do not degrade when SGSC is removed, the claim that SGSC reduces cross-category interference is not supported. A direct check is to visualize $A^{(l)}_t$ and $\\tilde{A}^{(l)}_t$ at the $64\\times64$ and $32\\times32$ layers for that query.","tokens_in":17123,"feed_emoji":"🌡️","tokens_out":8152,"duration_ms":75234,"temperature":0.7,"pith_summary":"This paper is trying to establish that visible-to-infrared translation improves when semantic information is used not only to condition the diffusion model, but also to regulate which tokens attend to which inside the denoising network. It proposes SC-Diff, a latent diffusion model in which a pretrained SAM3 model with text prompts produces category masks that are merged into a semantic map, fused with the visible image as input, and converted into token-level labels for a Semantic-Guided Self-Attention Calibration (SGSC) module. SGSC adds adaptive positive biases to same-category query-key pairs, with the bias scaled by how dispersed the query's attention is across categories and how much it already favors its own category, so cross-category attention is reduced but not eliminated. If this claim is right, synthetic infrared images inherit the visible scene's object locations, shapes, and layouts, making annotations from visible images reusable for infrared object detection. The paper's experiments report the best LPIPS and FID on three paired benchmarks and higher downstream detection mAP when translated images are added to training sets.","feed_headline":"Attention calibration sharpens visible-to-infrared translation","feed_subtitle":"SC-Diff's soft semantic bias preserves object structure and boosts infrared detectors trained on synthetic data.","key_machinery":"The load-bearing object is the Semantic-Guided Self-Attention Calibration (SGSC) module placed inside the denoising U-Net. It converts the SAM3 semantic map into token-level labels $\\ell^{(l)}$ at the attention resolution, then rewrites the attention logits as $\\tilde{A}^{(l)}_{t,ij}=\\mathrm{Softmax}\\bigl(S^{(l)}_{t,ij}+\\lambda\\,\\beta^{(l)}_{t,i}\\,G^{(l)}_{t,ij}\\bigr)$, where $\\beta^{(l)}_{t,i}=H^{(l)}_{t,i}\\bigl(1-\\hat{P}^{(l)}_{t,i,\\ell^{(l)}_i}\\bigr)$ is a query-wise calibration strength combining semantic-category attention entropy with under-attention to the query's own category, and $G^{(l)}_{t,ij}=A^{(l)}_{t,ij}M^{(l)}_{ij}/\\max_r A^{(l)}_{t,ir}M^{(l)}_{ir}$ scales the bias by the original attention to same-category keys. The same-category indicator $M^{(l)}_{ij}$ ensures only tokens with equal nonzero labels are boosted. This mechanism carries the argument because it turns external semantic knowledge into an internal, query-dependent reweighting of token interactions while leaving nonzero attention mass to other categories.","core_discovery":"The central claim is that semantic priors should act twice in visible-to-infrared translation: once as an external condition fused with the visible image, and once as token-level labels that recalibrate self-attention inside the denoising U-Net. The recalibration, called SGSC, adds a positive logit bias to query-key pairs whose tokens carry the same semantic label, with the bias strength governed by the entropy of the query's attention across categories and by the attention the query already assigns to its own category, and the bias is further modulated by the original attention weights so that same-category keys with stronger responses receive larger boosts. This soft scheme is supposed to suppress cross-category leakage without hard masking, preserving global context while protecting object boundaries and small structures. In the reported experiments, SC-Diff achieves the best LPIPS and FID on M3FD, FLIR, and KAIST, the lowest FID on Cityscapes-to-infrared transfer, and the highest detection mAP when Cityscapes images translated by SC-Diff are added to the M3FD and FLIR training sets.","pith_inferences":["Editorial inference: The SGSC operation is not tied to infrared modality; any conditional diffusion task with a semantic map, such as depth- or edge-guided synthesis, could use token-level attention biases to preserve layout, so the method likely generalizes beyond thermal translation.","Editorial inference: The calibration strength depends on SAM3 masks surviving resizing to $64\\times64$ and $32\\times32$, so small-object categories are the natural failure point; deliberately corrupting or dropping masks would map how quickly the reported gains degrade.","Editorial inference: SC-Diff's progressive classifier-free guidance separates visible-layout guidance from semantic-map guidance, which suggests a per-category knob for $c_s$ that could emphasize rare categories; the paper does not explore this."],"forward_implications":["Generated infrared images preserve object locations, shapes, and semantic layouts from the visible input, so detection annotations inherited from visible images remain usable.","Augmenting real infrared training sets with SC-Diff-translated Cityscapes images improves overall mAP on M3FD and FLIR over vanilla training and over PID/F-ViTA augmentation.","SC-Diff reports the best LPIPS and FID on M3FD, FLIR, and KAIST among compared GAN and diffusion translators, at competitive PSNR/SSIM.","For out-of-distribution translation from Cityscapes, SC-Diff achieves lower FID against each target infrared distribution than PID and F-ViTA.","Self-attention visualizations show that SGSC reduces cross-category responses while keeping nonzero attention mass to other categories, which is the mechanism the paper attributes the improvement to."],"supporting_citations":[{"why":"Supplies the pretrained SAM3 masks with text prompts that produce the semantic map and token labels used by SGSC.","marker":"[8]"},{"why":"Provides the latent diffusion backbone and autoencoder that SC-Diff extends with a condition branch and attention calibration.","marker":"[14]"},{"why":"Recent physics-informed diffusion baseline whose over-smoothing and structural loss SC-Diff is compared against and motivated by.","marker":"[1]"},{"why":"Foundation-model-guided baseline that uses semantic priors only as external guidance; primary comparison for both generation and augmentation.","marker":"[22]"},{"why":"Prior diffusion translator using SAM maps and vision-language understanding; also the source of the M3FD split SC-Diff adopts.","marker":"[25]"},{"why":"Classifier-free guidance formulation that SC-Diff's progressive visible-plus-semantic guidance extends.","marker":"[29]"},{"why":"DDIM sampler used for efficient inference in all reported generation and augmentation experiments.","marker":"[28]"}],"fun_headline_variants":["Soft semantic bias preserves IR object shapes","Two-fold semantic priors refine IR translation","Self-attention calibration boosts IR detection training","Same-category attention bias sharpens visible-to-IR","Semantic labels calibrate diffusion for IR translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"SC-Diff assumes the pretrained SAM3 masks label the visible image correctly enough that, after resizing to the $64\\times64$ and $32\\times32$ attention grids, the same-category token pairs it boosts really are same-category in the target infrared image; if masks mislabel or miss objects, the calibration strengthens the wrong attention links instead of reducing cross-category interference.","fun_headline_variants_meta":{"raw":{"variants":["Soft semantic bias preserves IR object shapes","Two-fold semantic priors refine IR translation","Self-attention calibration boosts IR detection training","Same-category attention bias sharpens visible-to-IR","Semantic labels calibrate diffusion for IR translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2949,"prompt_tokens":1028,"completion_tokens":1921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1853}},"tokens_in":644,"tokens_out":1921,"duration_ms":15018,"temperature":1.0,"reasoning_tokens":1853,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:30:58.621669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a visible image in which SAM3 misses a small object, such as a distant cyclist, and compare SC-Diff with SGSC disabled. If the missed object's queries do not show reduced cross-category attention after calibration, or if LPIPS/FID do not degrade when SGSC is removed, the claim that SGSC reduces cross-category interference is not supported. A direct check is to visualize $A^{(l)}_t$ and $\\tilde{A}^{(l)}_t$ at the $64\\times64$ and $32\\times32$ layers for that query.","supporting_citations":[{"cited_title":"SAM3: Segment anything with concepts,","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained SAM3 masks with text prompts that produce the semantic map and token labels used by SGSC."},{"cited_title":"High- resolution image synthesis with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Provides the latent diffusion backbone and autoencoder that SC-Diff extends with a condition branch and attention calibration."},{"cited_title":"PID: Physics-informed diffusion model for infrared image generation,","cited_arxiv_id":null,"evidence_quote":"Recent physics-informed diffusion baseline whose over-smoothing and structural loss SC-Diff is compared against and motivated by."},{"cited_title":"F-vita: Foundation model guided visible to infrared translation,","cited_arxiv_id":null,"evidence_quote":"Foundation-model-guided baseline that uses semantic priors only as external guidance; primary comparison for both generation and augmentation."}],"review_version":1}