{"id":"7cd1dc3e-cb75-4d75-876c-72a820117e1b","arxiv_id":"2505.12606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Diff-MM reuses a pre-trained Stable Diffusion UNet as a tracking feature extractor and obtains state-of-the-art results across RGB-N/D/T/E tracking benchmarks with a single set of parameters.","lead":"This paper builds a single object tracker that reuses Stable Diffusion, a text-to-image model, to track objects using RGB plus depth, thermal, event, or language inputs. The result is a unified tracker that reports large gains over existing multi-modal trackers on several benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SD-prior contribution is not isolated: no same-architecture non-SD baseline, so reported gains may stem from the UNet architecture and two-stage recipe rather than from generative priors.","rationale":"The reader's weakest_assumption focuses on whether depth/thermal/event signals survive VAE encoding. That is a real risk, and the paper's Extra Encoder vs. VAE Encoder ablation (Table 2 right) is some evidence for the chosen path. But the more fundamental, unaddressed control is whether the Stable Diffusion pretraining itself matters. The system's novelty and the abstract's causal story depend on reusing SD's 'multi-modal understanding capability,' yet every ablation compares the full SD-based design to a different interface/backbone pairing; none holds the architecture fixed and removes only the pretrained SD weights. Without that control, the strong benchmark numbers could be explained by the UNet's capacity and the fine-tuning recipe, which would reduce the contribution to an architecture-and-tuning exercise rather than a demonstration of generative-prior transfer. This is not an accusation; it is a missing experiment that is standard for attributing gains to pretraining. The concrete test is expensive but feasible at the scale already reported (4x RTX 3090s), and it would settle the issue. If the random-init control performs comparably, the central claim should be downgraded; if it collapses, Diff-MM's attribution is supported. Since the paper currently reports neither code nor this control, CONDITIONAL remains the appropriate verdict.","tokens_in":13916,"tokens_out":13152,"duration_ms":139605,"concrete_test":"Retrain the full Diff-MM pipeline identically (same PFE, same MST, same data, same schedule, same compute budget) but initialize the UNet and VAE from scratch or with a non-generative pretrained backbone (e.g., MAE ViT-L) instead of Stable Diffusion v1-5, and compare Table 1 numbers on DepthTrack F-score, LasHeR SR/PR, VisEvent SR/PR, and TNL2K AUC. If the control matches Diff-MM within about 1 point, the SD-prior attribution fails; if it drops by more than about 3 points, the generative-prior claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is causal attribution. The paper's central claim is that reusing Stable Diffusion priors is what enables SOTA multi-modal tracking with a single parameter set. Yet no experiment isolates the SD pretraining itself. Table 2 compares PFE against alternative interaction designs (Diff-Tracker prompt tokens; Cross-Attn with MAE/CLIP ViT) and Table 2 right compares MST against Cross-Attn/Early Fusion/T2I-Adapter Extra Encoder; these change the interface and the backbone simultaneously. A same-architecture baseline with the same PFE/MST pipeline but without SD-pretrained weights (random init, or a non-generative pretrained UNet/ViT of comparable capacity) is absent. Therefore the reported gains on TNL2K/DepthTrack/LasHeR/VisEvent could come from the UNet architecture, the large fine-tuned parameter count, or the two-stage training recipe, rather than from 'extensive prior knowledge in the generation model.' The unexplained SD v3-5 (MM-DiT) variant further obscures which components transfer. The reader's modality-signal concern is plausible, but the Extra Encoder ablation partially addresses it; the missing random-init control is the sharper gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Diff-MM, a unified multi-modal object tracker covering RGB-N (natural language), RGB-D, RGB-T, and RGB-E modalities with a single parameter set. It builds on Stable Diffusion v1-5, using the frozen VAE and denoising UNet as a feature extractor. A parallel feature extraction pipeline (PFE) processes template and search frames through the UNet in parallel, concatenating them at each self-attention layer to model their relationship without architectural modification. A multi-modal sub-module tuning method (MST) clones UNet encoder/middle-block weights into a sub-module for auxiliary modalities, injecting features via zero-initialized lateral convolutions while freezing the RGB UNet. A two-stage training recipe first fine-tunes on RGB/RGB-N tracking data, then tunes the sub-module on RGB-D/T/E data. Experiments on TNL2K, OTB99, DepthTrack, VOT-RGBD2022, LasHeR, RGBT234, and VisEvent report consistent improvements over prior dedicated and unified trackers, e.g., 66.3 vs 58.0 AUC on TNL2K and 68.7 vs 60.7 F-score on DepthTrack versus OneTracker (Table 1). Ablations (Table 2) show PFE outperforms prompt-token and cross-attention alternatives, and MST outperforms early fusion and extra-encoder designs.","tokens_in":14181,"tokens_out":6631,"duration_ms":61632,"significance":"If the causal claim holds, the paper makes a strong contribution: it demonstrates that a text-to-image generative model's pretrained representations can be transferred to multi-modal tracking, reducing the need for large multi-modal training data and enabling one parameter set across four modality pairs. The ablations are carefully designed and support the specific design choices of PFE and MST over several alternatives. The reported gains over strong baselines are large and consistent across benchmarks, which makes the method practically interesting. However, the paper's central attribution of these gains to Stable Diffusion's 'extensive prior knowledge' is not isolated experimentally; the missing same-architecture non-SD baseline leaves open alternative explanations based on the UNet architecture, parameter count, and training recipe. The SD v3-5 variant, used to claim generality, is insufficiently specified.","major_comments":[{"comment":"The central claim that gains arise from 'extensive prior knowledge in the generation model' is not supported by a same-architecture control without SD pretraining. Every ablation in Table 2 varies both the interaction design and the backbone simultaneously: PFE uses the SD UNet for both template and search, while Cross-Attn/MAE/CLIP variants introduce a different template encoder; similarly, MST is compared to early fusion and extra encoders that modify the network path. A random-init SD-UNet (or a non-generative pretrained UNet of comparable capacity) trained with the identical PFE+MST pipeline and two-stage recipe would isolate the contribution of SD pretraining. Without such a baseline, the reported improvements on TNL2K, DepthTrack, LasHeR, and VisEvent could stem from the UNet architecture, the large fine-tuned parameter count, or the training recipe rather than from generative priors. Please add this control and report the comparison in Table 2.","section":"§4.3, Table 2"},{"comment":"The SD v3-5 variant is reported as evidence that the design transfers to other diffusion architectures, but no implementation details are given. It is unclear how PFE's self-attention concatenation and MST's sub-module are adapted to the MM-DiT architecture, what text conditioning and VAE/noising scheme are used, whether the same two-stage training recipe and hyperparameters apply, and how the modality inputs are converted. Without these details, the Diff-MM 256† results in Table 1 cannot be reproduced or interpreted, and the generality claim is weakened. Please provide a full description or remove the variant.","section":"§4.1, 'Diff-MM 256†'"},{"comment":"The ablation labeled 'W/ TNL2K vs. W/O TNL2K' claims that the performance improvement is not due to the extra TNL2K training data, but the reported numbers (DepthTrack, LasHeR, VisEvent) do not include any RGB-N tracking result. Since TNL2K is both a first-stage training dataset and a headline test set in Table 1, the claim is unsupported for the RGB-N results. Please report RGB-N results (e.g., on OTB99) obtained without TNL2K in first-stage training, or otherwise clarify the scope of the claim.","section":"§4.3, Table 2 right"}],"minor_comments":[{"comment":"The use of a fixed diffusion timestep t=1 is mentioned but not justified; please state whether the same timestep is used for template and search latents and how it was chosen.","section":"§4.1"},{"comment":"The index N in \\hat{s}_t[N-i] is not defined; clarify that N is the number of lateral-connection injection points or the layer index.","section":"§3.3, Eq. (4)"},{"comment":"The 'Data Flow in UNet Basic Block' inset is too small to read the ConcatL/DeConcatL operations; a larger separate diagram would help.","section":"Figure 2"},{"comment":"The column headers 'Modality-specific Model', 'Unified Model with Modality-specific Parameters', and 'Unified Model with a Single Set of Parameters' are easy to misread because Diff-MM columns appear in multiple groups; consider using separate sub-tables or a clearer grouping.","section":"Table 1"},{"comment":"The statement that Diff-Tracker's poor performance is due to lack of fine-grained detail is plausible but the re-implementation details (training data, epochs, prompt token length) are not given; please add them for reproducibility.","section":"§4.2"},{"comment":"No inference speed or parameter count is reported; since tracking methods are compared also on efficiency, adding FPS or MACs for the 256 and 384 variants would be useful.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically solid in its ablations but the missing random-init control is a real gap. If the authors can add that control and show that SD pretraining matters, the paper would be a strong accept. The TNL2K training-protocol point should also be addressed; the current ablation does not cover the RGB-N case. I do not see evidence of misconduct; the empirical results appear consistent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead Diff-MM. In one line: it is a serious, well-ablated systems paper that gets impressive numbers across RGB-N/D/T/E tracking with a single parameter set, but it stops short of proving the mechanism it claims.\n\nWhat's new: PFE runs template and search through the frozen SD UNet in parallel, swapping in the self-attention for interaction; MST clones encoder/middle blocks for depth/thermal/event and injects via zero-init lateral connections. That is a clean way to adapt a generation backbone to pairwise tracking without modifying its architecture, and it is genuinely different from Diff-Tracker, VPD, and T2I-Adapter. The benchmark results are strong: e.g., 68.7 F-score on DepthTrack vs 60.7 for OneTracker, and 66.3 AUC on TNL2K vs 58.0. The ablations are more honest than most: they test PFE against cross-attention variants, MST against early fusion and extra encoders, show zero-init matters, and include a no-TNL2K variant to show the gains are not from added training data.\n\nThe soft spot is causal attribution. The paper's pitch is that 'extensive prior knowledge in the generation model' is what enables the gains. But there is no same-architecture baseline with randomly initialized UNet, or a non-generative pretrained uni-modal backbone of comparable capacity, using the same PFE/MST pipeline. So the results could plausibly come from the UNet architecture, the large number of tuned parameters, or the two-stage recipe. The stress-test note is right: this is the sharper gap. The SD v3-5 variant is a single line with no implementation detail, which doesn't help. Also, TNL2K appears in first-stage training and in the headline comparison; their own ablation suggests it is not the driver, but the protocol should be stated prominently. No code release, which makes reproduction harder.\n\nNone of this is fatal. The reader's conditional verdict is about right. The paper is a strong engineering contribution and the PFE/MST designs are worth publishing. It just needs a random-init or non-generative pretrained baseline, some SD v3-5 details, and ideally code.\n\nWho it's for: people working on multi-modal or unified tracking, and anyone using diffusion features for perception. It deserves a serious referee. I'd recommend sending it out with a request for the missing baseline.\n\nBest.","headline":"A well-executed, strongly benchmarked unified multi-modal tracker built on a frozen SD UNet; the causal claim that SD priors drive the gains is plausible but not isolated, so the paper needs one more baseline before the conclusion is as strong as the numbers.","tokens_in":14685,"tokens_out":2054,"would_cite":true,"duration_ms":21408,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pre-trained text-to-image generation model's UNet can be repurposed as a unified multi-modal object tracker, with one parameter set covering RGB-N/D/T/E tracking.","keywords":["multi-modal object tracking","diffusion models","Stable Diffusion","unified tracker","RGB-N/D/T/E tracking","transfer learning","parallel feature extraction","modality sub-module tuning"],"falsifier":"Run Diff-MM's full two-stage training with the UNet and VAE re-initialized from random weights, keeping the same architecture, PFE/MST, data, and loss, and compare DepthTrack F-score and TNL2K AUC; if the randomly initialized model matches the reported numbers, the claim that pre-trained generative priors are the source of the gains is refuted. A complementary check is to shuffle the auxiliary modality's pixels at test time: a tracker that truly exploits depth, thermal, or event data should degrade sharply, whereas one that learned benchmark statistics would not.","tokens_in":13683,"feed_emoji":"🎯","tokens_out":6768,"duration_ms":64014,"temperature":0.7,"pith_summary":"The paper tries to establish that a pre-trained text-to-image generation model can be converted into a unified multi-modal object tracker, replacing the usual recipe of training an RGB tracker and then prompt-tuning it on small multi-modal datasets. If true, this matters because the main bottleneck of multi-modal tracking—the scarcity of paired RGB-depth, RGB-thermal, RGB-event, and RGB-language training data—could be bypassed by borrowing priors learned from web-scale image-text pairs. Diff-MM reports benchmark numbers that beat earlier unified and modality-specific trackers, including 66.3 versus 58.0 AUC on TNL2K and 68.7 versus 60.7 F-score on DepthTrack when compared with OneTracker.","feed_headline":"One diffusion model handles four kinds of object tracking","feed_subtitle":"Reusing Stable Diffusion's UNet with parallel inputs and a cloned sub-module, a single parameter set beats dedicated RGB-D/T/E/N trackers.","key_machinery":"The load-bearing machinery is a pair of modifications to a frozen Stable Diffusion UNet. Parallel Feature Extraction (PFE) concatenates the template and search token sequences at the self-attention layer of every Basic Block, runs attention jointly, then de-concatenates, so the UNet handles pairwise inputs while its architecture and weights stay intact. Multi-modal Sub-module Tuning (MST) clones the UNet encoder and middle block into a modality sub-module, feeds the auxiliary modality's noisy VAE latents through it, and adds the result to the RGB branch through zero-initialized convolutions; starting at zero means the auxiliary signal is learned gradually and does not disturb the pretrained RGB features at initialization.","core_discovery":"The central claim is that the UNet of Stable Diffusion—normally a denoiser—can be turned into a tracking feature extractor without altering its architecture. The parallel feature extraction pipeline feeds template and search frames through the same frozen UNet, exchanging information only at the existing self-attention layers; a cloned encoder-and-middle-block sub-module, injected through zero-initialized lateral connections, brings in depth, thermal, or event data; language descriptions enter through the already-present cross-attention text condition. With one parameter set shared across all four modality types, Diff-MM outperforms earlier unified trackers and modality-specific trackers on RGB-N/D/T/E benchmarks, for example 66.3 AUC on TNL2K and 68.7 F-score on DepthTrack in the single-parameter-set configuration. On the paper's own account, the generative model's prior knowledge of depth, thermal, event, and language is what makes small multi-modal training sets sufficient.","pith_inferences":["An inference from the PFE design: any pairwise-input visual task, such as stereo correspondence, frame-to-frame local feature matching, or few-shot segmentation, could reuse the concat/de-concat self-attention trick to inherit generative priors without retraining the backbone.","An inference from the MST design: new sensing modalities could be added to an existing tracker by cloning the encoder and middle block and training only the new sub-module, leaving the RGB branch untouched, which would make a practical recipe for sensor-agnostic trackers.","A testable extension: evaluate the same pipeline with an untrained UNet of identical architecture; if performance stays high, the gains come from scale and architecture rather than the pretrained generative prior, which would reframe the paper's contribution.","A natural question raised by the unified-parameter results is whether mixing all modalities in one second-stage sub-module, rather than one sub-module per modality, is what preserves per-modality accuracy; the paper reports both variants but does not isolate that effect."],"forward_implications":["Training a multi-modal tracker reduces to fine-tuning a frozen generative backbone, so limited multi-modal data no longer limits model capacity.","The same PFE recipe can turn other SD-family diffusion backbones into tracking extractors, as demonstrated by the SD v3-5 variant.","One set of weights handles RGB-N, RGB-D, RGB-T, and RGB-E tracking, and switching modalities amounts to attaching the same sub-module.","Freezing the UNet avoids forgetting, so adding new auxiliary modalities does not degrade RGB or language-guided tracking performance."],"supporting_citations":[{"why":"Supplies the pre-trained Stable Diffusion UNet and VAE whose generative priors Diff-MM reuses as the tracking backbone.","marker":"[37]"},{"why":"Provides the tracking head, focal/GIoU/L1 losses, and the template–search tracking formulation that Diff-MM builds on.","marker":"[52]"},{"why":"OneTracker is the unified foundation-model tracker that Diff-MM compares against and outperforms on TNL2K and DepthTrack.","marker":"[14]"},{"why":"ViPT is the prompt-tuned multi-modal tracker baseline that motivates the claim that frozen diffusion priors beat small parameter-efficient adapters.","marker":"[58]"},{"why":"Un-Track is the any-modality single-model baseline whose LoRA-based approach Diff-MM contrasts with its frozen-UNet design.","marker":"[47]"},{"why":"VPD establishes the precedent of using the pre-trained SD UNet as a perception feature extractor after removing noise prediction.","marker":"[56]"},{"why":"DIFT shows the same UNet yields emergent visual correspondence features, supporting the claim that generative priors transfer to tracking.","marker":"[41]"},{"why":"ControlNet's zero-initialized convolution injection is the conditional-control mechanism that MST adapts for auxiliary modality fusion.","marker":"[53]"},{"why":"T2I-Adapter serves as the ablation baseline for an extra modality encoder, against which reusing the VAE encoder is shown effective.","marker":"[31]"},{"why":"Diff-Tracker is the prior diffusion-based tracker that PFE outperforms in ablation, motivating the parallel-input design.","marker":"[55]"}],"fun_headline_variants":["Diff-MM: Stable Diffusion UNet as a universal tracker","Diff-MM: One diffusion model, four tracking modalities","Diff-MM: Reusing text-to-image model for multi-modal tracking","Diff-MM: Diffusion UNet unifies RGB, depth, thermal, event","Diff-MM: Single parameter set beats dedicated multi-modal trackers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that depth, thermal, and event frames, once converted to RGB-like images and passed through the frozen VAE and diffusion noising pipeline, still carry the modality-specific information a tracker needs; if that premise fails, the reported gains would come from the cloned sub-module and lateral connections rather than from Stable Diffusion's priors.","fun_headline_variants_meta":{"raw":{"variants":["Diff-MM: Stable Diffusion UNet as a universal tracker","Diff-MM: One diffusion model, four tracking modalities","Diff-MM: Reusing text-to-image model for multi-modal tracking","Diff-MM: Diffusion UNet unifies RGB, depth, thermal, event","Diff-MM: Single parameter set beats dedicated multi-modal trackers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2930,"prompt_tokens":946,"completion_tokens":1984,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":1897}},"tokens_in":562,"tokens_out":1984,"duration_ms":16445,"temperature":1.0,"reasoning_tokens":1897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:30:34.940903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Diff-MM's full two-stage training with the UNet and VAE re-initialized from random weights, keeping the same architecture, PFE/MST, data, and loss, and compare DepthTrack F-score and TNL2K AUC; if the randomly initialized model matches the reported numbers, the claim that pre-trained generative priors are the source of the gains is refuted. A complementary check is to shuffle the auxiliary modality's pixels at test time: a tracker that truly exploits depth, thermal, or event data should degrade sharply, whereas one that learned benchmark statistics would not.","supporting_citations":[{"cited_title":"Joint feature learning and relation modeling for tracking: A one-stream framework","cited_arxiv_id":null,"evidence_quote":"Provides the tracking head, focal/GIoU/L1 losses, and the template–search tracking formulation that Diff-MM builds on."},{"cited_title":"Onetracker: Unifying visual object tracking with foundation models and efficient tuning","cited_arxiv_id":null,"evidence_quote":"OneTracker is the unified foundation-model tracker that Diff-MM compares against and outperforms on TNL2K and DepthTrack."},{"cited_title":"Visual prompt multi-modal tracking","cited_arxiv_id":null,"evidence_quote":"ViPT is the prompt-tuned multi-modal tracker baseline that motivates the claim that frozen diffusion priors beat small parameter-efficient adapters."},{"cited_title":"Single-model and any-modality for video object tracking","cited_arxiv_id":null,"evidence_quote":"Un-Track is the any-modality single-model baseline whose LoRA-based approach Diff-MM contrasts with its frozen-UNet design."},{"cited_title":"Unleashing text-to-image diffusion models for visual perception","cited_arxiv_id":null,"evidence_quote":"VPD establishes the precedent of using the pre-trained SD UNet as a perception feature extractor after removing noise prediction."},{"cited_title":"Emergent correspondence from image diffusion","cited_arxiv_id":null,"evidence_quote":"DIFT shows the same UNet yields emergent visual correspondence features, supporting the claim that generative priors transfer to tracking."},{"cited_title":"T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"T2I-Adapter serves as the ablation baseline for an extra modality encoder, against which reusing the VAE encoder is shown effective."},{"cited_title":"Diff-tracker: Text-to-image diffusion models are unsupervised trackers","cited_arxiv_id":null,"evidence_quote":"Diff-Tracker is the prior diffusion-based tracker that PFE outperforms in ablation, motivating the parallel-input design."}],"review_version":1}