{"id":"467c83be-81c8-4c3d-b365-a94d4a3e4608","arxiv_id":"2411.11505","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single diffusion transformer with a spatial-temporal VAE and in-context example pairs unifies over 20 vision tasks, with strong scores on several image benchmarks and uneven evidence on video tasks.","lead":"LaVin-DiT is a 3.4-billion-parameter diffusion transformer that tackles more than 20 image and video tasks by compressing visuals into a latent space and denoising outputs guided by example input-output pairs. It reports faster inference and stronger benchmark numbers than the autoregressive LVM baseline, but quantitative evidence covers only about half of the claimed tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NYUv2 depth/normal gains may be inherited from the very pseudo-label teachers used to create training data; undisclosed teacher provenance is the main risk to the SOTA claim.","rationale":"The paper's strongest quantitative evidence is Table 2: on NYUv2, LaVin-DiT improves depth AbsRel from LVM's 30.2 to 6.2 (abstract: '24 lower AbsRel') and normal MAE from 23.4 to 15.9, surpassing StableNormal. Because the training labels for depth and normal are generated by Depth-Anything V2 and StableNormal themselves, this evaluation is not independent of the strongest competing expert models. The load-bearing question is whether those teachers have seen NYUv2 during their own training. For the DPT-based Depth-Anything pipeline, which uses a heterogeneous multi-dataset teacher including common depth benchmarks, this is highly plausible; for StableNormal it is unaddressed. If the teachers were NYUv2-exposed, the model is rewarded for distilling a benchmark-exposed expert, not for achieving a new form of in-context generalization. This is an experimental-design/correctness risk, not a consensus disagreement, and it directly targets the paper's boldest claim. Other issues identified by the reader, such as missing video metrics, the colorization inconsistency, and the 'unseen task' framing, are real but secondary; they affect completeness and internal consistency rather than the central comparison against LVM. The paper does have genuine strengths: the ST-VAE is a concrete architectural contribution, flow-matching training is specified, scalability curves are provided, and the measured 1.7-2.3x inference speed advantage over LVM is architectural and therefore robust. The limitation paragraph is candid about out-of-distribution generalization. If the teacher-provenance check comes back clean and the off-teacher benchmark gap persists, the conditional verdict stands and the SOTA claim is credible. If not, the headline claims must be heavily qualified. The reader's weakest assumption is exactly this provenance issue, so I agree and recommend no change to the conditional verdict at this time.","tokens_in":20187,"tokens_out":7064,"duration_ms":73383,"concrete_test":"Check the released training-data documentation for Depth-Anything V2, StableNormal (turbo), and their teachers (e.g., DPT), and determine whether NYUv2's train split or NYUv2-derived data appears. Independently of that, run LaVin-DiT (3.4B) on a depth/normal benchmark that is absent from all teachers' training sets (e.g., KITTI depth or ETH3D for depth, and a held-out normal benchmark) using the same affine-invariant protocol. Also compute the prediction agreement between LaVin-DiT and the teachers on NYUv2; if LaVin-DiT reproduces teacher outputs almost exactly, the reported gains are distillation artifacts. Only if the margin over LVM and near-expert accuracy persist on off-teacher benchmarks should the SOTA claim be considered established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that pseudo depth and normal labels are generated with Depth-Anything V2 and Stable-Normal (turbo), and Table 4 lists these teachers as supervision sources for ImageNet-1K, COCO, ADE20K, and Cityscapes. Table 2 then compares LaVin-DiT against StableNormal, Marigold, DPT, and LVM on NYUv2. If either teacher, or the DPT-style teacher used to train Depth-Anything V2, was trained on NYUv2 or NYUv2-derived data, then LaVin-DiT's AbsRel of 6.2 and normal MAE of 15.901 are partly inherited: the model learns to imitate an expert that already saw the test distribution. The headline comparison against LVM (AbsRel 30.2 to 6.2) and the claim of surpassing StableNormal would then demonstrate teacher-to-student distillation, not independent in-context generalization by a unified diffusion transformer. The paper gives no provenance statement for either teacher's training data and reports no depth/normal evaluation on benchmarks outside the likely training corpora of these teachers. Supporting caution: colorization is reported as MSE 0.24 in Table 1 but 0.273 in Figure 5 for the same 3.4B model, indicating protocol drift across reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LaVin-DiT, a unified diffusion-transformer foundation model that handles over 20 image and video tasks via a spatial-temporal VAE (ST-VAE), a joint diffusion transformer (J-DiT) with full-sequence joint attention and 3D RoPE, and in-context conditioning through input-target pairs. Training data combines about 3.2M images and 0.6M videos, with pseudo-labels for depth and normals from Depth-Anything V2 and Stable-Normal (turbo). Quantitative results are reported for foreground segmentation, single object detection, colorization, NYUv2 depth/normal estimation, and ImageNet inpainting, plus qualitative results for many tasks and a latency comparison against LVM. The paper claims state-of-the-art performance and 1.7–2.3× faster inference relative to LVM.","tokens_in":20480,"tokens_out":5401,"duration_ms":48326,"significance":"If the reported results are reproducible and the issues below are resolved, this is a potentially influential contribution: it shows that a diffusion-based generative framework can unify diverse vision tasks and outperform the autoregressive LVM baseline on several benchmarks. The paper provides useful implementation details, including algorithm pseudocode, architecture configuration tables, and a plan to release code and models, which supports reproducibility. The strongest quantitative gains are on foreground segmentation (e.g., 67.87 mIoU vs. 48.94 for LVM on Split 1), single object detection, colorization, and inpainting (FID 1.65 vs. 4.05). However, several load-bearing claims currently lack sufficient evidence, particularly the depth/normal results, the 'unseen task' claim, and all video-task claims.","major_comments":[{"comment":"The claim that foreground segmentation and single object detection are 'unseen tasks during our training' (Table 1 note and main text) is contradicted by Table 4, which lists single object detection on COCO/Object365 and instance segmentation on COCO/ADE20K/Cityscapes as training tasks. Even if the Pascal-5i classes are held out, the task families are trained; this overstates the generalization claim and should be reworded with a precise statement of what is unseen (e.g., dataset splits or classes).","section":"§4.2, Table 1, Table 4"},{"comment":"The NYUv2 depth and surface normal results in Table 2 may be inherited from the pseudo-label teachers. Training depth labels are generated with Depth-Anything V2 and normal labels with Stable-Normal (turbo) on ImageNet-1K/COCO/ADE20K/Cityscapes. The paper does not report whether these teachers or their training data include NYUv2 or NYUv2-derived benchmarks, nor does it provide depth/normal evaluation on a dataset outside the teachers' likely training corpora. Without this provenance, the AbsRel 6.2 and MAE 15.901 gains over LVM could reflect teacher-to-student distillation rather than in-context generalization, so the comparison in Table 2 is not decisive.","section":"§4.1, Table 4, Table 2"},{"comment":"The colorization results are internally inconsistent: Table 1 and the text in §4.2 report MSE 0.24 for LaVin-DiT (3.4B), while §4.3 and Figure 5 report MSE 0.273 for the same model. The evaluation subsample sizes also differ from the protocol in §B.2 (1,000 images vs. possibly different for Figure 5). The authors should state one protocol and report all numbers from it, or explain the discrepancy.","section":"§4.2, §4.3, Table 1, Figure 5"},{"comment":"No quantitative results are reported for any video task, despite the abstract and introduction claiming state-of-the-art performance across 'diverse vision tasks' including video. Frame prediction, video depth/normal/optical flow/instance segmentation, and depth/normal-to-video are shown only qualitatively. Since the spatial-temporal VAE and video evaluation are a core part of the claimed contribution, the paper should either provide quantitative video evaluations or explicitly limit the SOTA claim to the evaluated image tasks.","section":"§4.2, §B.1, Figures 3, 23-29"},{"comment":"The latency comparison in §4.4 states that LaVin-DiT and LVM are 'both 7B models', but LaVin-DiT is 3.4B by default (Table 3). Comparing a 3.4B diffusion model to a 7B autoregressive model confounds architecture with capacity. A matched-capacity or per-parameter latency/quality comparison is needed to support the inference-efficiency claim.","section":"§4.4, Figure 6"}],"minor_comments":[{"comment":"'LaVin-Dit' should be 'LaVin-DiT'.","section":"§4.2, first paragraph"},{"comment":"The phrase 'from the respective of image-resembling generation' should be 'from the perspective'; there are several other grammatical issues throughout the text that should be corrected.","section":"§2, Related Work"},{"comment":"The final line 'Decode sample: ˆy ← Dec(z0)' is ambiguous; it should refer to the integrated latent z(0) at t=0, not the initial latent z0 defined in Algorithm 1.","section":"Algorithm 2, line 9"},{"comment":"The column header 'K.V . groups' has a formatting issue; also clarify that grouped-query attention is only used for the 3.4B model (the 0.1B and 1.0B rows have no KV groups), and what value is used for the 3.4B model.","section":"Table 3"},{"comment":"The depth evaluation protocol mentions least-squares alignment but does not define the threshold accuracy δ1; please specify the standard thresholds (e.g., δ1 < 1.25) and whether the metrics are computed after alignment.","section":"§B.2, Depth Estimation"},{"comment":"The caption of Figure 7 does not define the plotted metric or the ten tasks included; add a description so the figure is self-contained.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the undisclosed relation between the pseudo-label teachers (Depth-Anything V2, Stable-Normal turbo) and the NYUv2 test set; this is checkable and should be resolved with a cross-dataset evaluation. The 'unseen tasks' wording directly contradicts the paper's own training table and should be revised. The complete absence of quantitative video results also weakens the headline claim. These issues are fixable with additional experiments and text revision, so I recommend major revision rather than rejection. If the depth/normal provenance check fails, the paper would need to be substantially reframed, but I would not make that decision based on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious piece of systems work, but the paper as written overstates what it shows. The architecture is a legitimate new combination – ST-VAE latent compression, joint diffusion transformer with full-sequence attention, 3D RoPE, flow matching, and in-context conditioning for 20+ tasks. The foreground segmentation numbers (67–76 mIoU vs LVM's 48–51) are eye-catching, and the scalability and context-length experiments are clean. The speed advantage over an autoregressive LVM is plausible in principle.\n\nThe soft spots are real. First, every video task is only shown qualitatively; there are no quantitative video results anywhere. You can't claim to unify video and image tasks on that evidence. Second, the paper says foreground segmentation and single object detection are 'unseen tasks' but Table 4 lists single object detection as training data (with COCO and Object365). That's an internal contradiction that needs a fix. Third – the big one – depth and normal labels are generated by Depth-Anything V2 and StableNormal, and the model is then evaluated on NYUv2. The paper never discloses whether those teachers were trained on NYUv2. If they were, the AbsRel of 6.2 and normal MAE of 15.9 are partly inherited knowledge, not evidence of in-context generalization. The authors need to either state teacher provenance or evaluate on benchmarks outside the teachers' training corpora.\n\nThere are smaller inconsistencies that erode confidence: colorization MSE is 0.24 in Table 1 but 0.273 in Figure 5 for the same 3.4B model; and the latency section says 'both 7B models' when LaVin-DiT is 3.4B, which muddles the efficiency comparison. The 'state-of-the-art' claim in the abstract is also too strong given the sparse baselines.\n\nWho's this for? People building unified vision foundation models, and anyone who cares about pseudo-label leakage in evaluation. It deserves a serious referee, but it needs major revision before the claims can be trusted. I'd recommend sending to peer review with a request for the authors to fix the contradictions, add video metrics, and clarify the pseudo-label provenance.","headline":"A genuinely interesting unified vision diffusion transformer whose headline claims are undercut by missing video numbers, an 'unseen tasks' contradiction, and unresolved pseudo-label provenance.","tokens_in":21058,"tokens_out":3521,"would_cite":false,"duration_ms":32801,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single diffusion transformer, LaVin-DiT, can unify more than 20 image and video tasks and outperform the strongest autoregressive large vision model baseline.","keywords":["diffusion transformer","large vision model","in-context learning","spatial-temporal variational autoencoder","flow matching","multi-task vision"],"falsifier":"Check the training-data documentation of Depth-Anything V2 and Stable-Normal (turbo) for NYUv2; if either model was trained on NYUv2 or its derivatives, the Table 2 depth and normal evaluations measure label inheritance rather than in-context generalization. A cleaner experiment is to retrain LaVin-DiT with pseudo labels from a teacher provably never exposed to NYUv2 and re-measure AbsRel and MAE on the same test set.","tokens_in":19986,"feed_emoji":"🖼️","tokens_out":7194,"duration_ms":63560,"temperature":0.7,"pith_summary":"LaVin-DiT sets out to show that a diffusion-based transformer, rather than an autoregressive language-model-style architecture, is the right backbone for a unified large vision model. The paper argues that by compressing images and videos into a continuous latent space with a spatial-temporal variational autoencoder and training one joint diffusion transformer with in-context input-target pairs as task definitions, a single 3.4B model can handle more than 20 image and video tasks without fine-tuning. Across benchmarks it reports consistent gains over the strongest autoregressive baseline, including large margins on NYU-v2 depth estimation and surface normals, and 1.7–2.3× faster inference. The payoff, if true, is a scalable vision foundation model that preserves spatial structure and adapts to tasks by example rather than by task-specific heads.","feed_headline":"One diffusion model beats autoregressive LVM on 20+ vision tasks","feed_subtitle":"LaVin-DiT's parallel denoising runs 1.7-2.3x faster while topping depth, normals, and segmentation.","key_machinery":"The load-bearing machinery is the combination of a spatial-temporal variational autoencoder (ST-VAE) and a joint diffusion transformer (J-DiT). ST-VAE compresses both images and videos into a continuous $4\\times 8\\times 8$ latent representation using causal 3D convolutions, with two stages of spatial-temporal downsampling followed by spatial-only downsampling, which keeps diffusion computation tractable. J-DiT extends MM-DiT by giving the clean condition latents and the noisy target latents separate patch embeddings and separate adaptive RMS normalization, then lets them interact through full-sequence joint attention with 3D rotary position embeddings so the model knows where each token sits in space and time. Training uses the conditional flow-matching loss to regress the velocity field, and inference integrates the learned ODE backward with Euler steps, so the same network can denoise any task's target given the right context.","core_discovery":"The paper's central claim is that the conditional distribution $p(y|x,s)$ of a vision output $y$ given a query $x$ and a task context $s$ can be modeled as a flow-matching denoising problem in a shared latent space. The task context is a set of input-target pairs that define the task, the query is encoded with the same spatial-temporal variational autoencoder, and the joint diffusion transformer denoises the target latent in parallel across the full sequence. The paper reports that at 3.4B parameters LaVin-DiT outperforms LVM on foreground segmentation (mIoU 67.87–75.80 versus 47.66–51.29 across splits), single-object detection (mIoU 67.85–69.32 versus 48.25–50.08), NYU-v2 depth (AbsRel 6.2 versus 30.2), surface normals (MAE 15.901 versus 23.433), and ImageNet inpainting (FID 1.65 versus 4.05), while running 1.7–2.3× faster at inference. These numbers are the concrete evidence for the thesis that diffusion modeling is a better fit than sequential token prediction for unified vision.","pith_inferences":["Because the depth and normal training labels come from teacher models, LaVin-DiT's NYU-v2 results likely inherit the teachers' biases; a direct test would be to compare the student's errors on out-of-distribution images with the teachers' errors, and the paper does not report such a comparison.","If pseudo-label contamination is ruled out, the strong depth and normal numbers would suggest that in-context conditioning can transfer metric knowledge from a teacher to a student; a clean way to check this is to hold out a teacher's label set and see whether the student still reproduces the teacher's distinctive failure modes.","A natural extension is automatic context selection: since longer contexts help, picking the most informative input-target pairs could deliver large gains without any architectural change, a direction the paper names as future work.","The limitation stated in the paper implies that in-context generalization should be read as generalization within the neighborhood of the training task distribution; tasks far outside that neighborhood would require either more data or a different mechanism."],"forward_implications":["Diffusion-based vision foundation models can match or beat autoregressive models on dense prediction while being faster, so future large vision models need not borrow LLM-style token-by-token decoding.","Longer task-context sets monotonically improve downstream accuracy, making the selection of good input-target pairs a practical lever for performance.","Scaling LaVin-DiT from 0.1B to 3.4B parameters produces monotone gains in both training loss and downstream metrics, supporting further scaling efforts.","A single checkpoint can serve both image and video tasks without task-specific heads or fine-tuning, simplifying deployment of generalist vision models."],"supporting_citations":[{"why":"The autoregressive large vision model (LVM) is the strongest baseline and the main comparison point for accuracy and latency.","marker":"[5]"},{"why":"Visual prompting via image inpainting supplies the in-context baseline that LaVin-DiT extends and outperforms.","marker":"[8]"},{"why":"Depth-Anything V2 generates the pseudo depth labels that supervise ImageNet-1K depth and depth-to-image training.","marker":"[72]"},{"why":"Stable-Normal (turbo) generates the pseudo surface normal labels and also serves as an expert baseline in Table 2.","marker":"[74]"},{"why":"The diffusion transformer (DiT) is the backbone that J-DiT builds on.","marker":"[43]"},{"why":"MM-DiT provides the condition- and target-specific normalization design that J-DiT adapts with AdaRN.","marker":"[20]"},{"why":"Flow matching supplies the training objective and the conditional flow-matching loss used for J-DiT.","marker":"[37]"},{"why":"NYU-v2 is the benchmark used to evaluate depth and surface normal estimation.","marker":"[54]"}],"fun_headline_variants":["Diffusion beats autoregression on 20+ vision tasks, runs 2x faster","LaVin-DiT: diffusion transformer outperforms LVM, 1.7-2.3x speedup","Parallel denoising wins: LVM beaten on depth, normals, segmentation","Vision diffusion model sweeps 20+ tasks, beats LVM in speed and accuracy","One diffusion transformer: 20+ vision tasks, faster than autoregressive"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pseudo depth and surface-normal labels generated by Depth-Anything V2 and Stable-Normal (turbo) are accurate and have not already been exposed to the NYUv2 evaluation distribution, since those labels supervise training and the paper's headline geometric numbers are measured on NYUv2.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion beats autoregression on 20+ vision tasks, runs 2x faster","LaVin-DiT: diffusion transformer outperforms LVM, 1.7-2.3x speedup","Parallel denoising wins: LVM beaten on depth, normals, segmentation","Vision diffusion model sweeps 20+ tasks, beats LVM in speed and accuracy","One diffusion transformer: 20+ vision tasks, faster than autoregressive"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000382,"raw_usage":{"total_tokens":2072,"prompt_tokens":1042,"completion_tokens":1030,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":916}},"tokens_in":658,"tokens_out":1030,"duration_ms":9322,"temperature":1.0,"reasoning_tokens":916,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:26:44.778871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the training-data documentation of Depth-Anything V2 and Stable-Normal (turbo) for NYUv2; if either model was trained on NYUv2 or its derivatives, the Table 2 depth and normal evaluations measure label inheritance rather than in-context generalization. A cleaner experiment is to retrain LaVin-DiT with pseudo labels from a teacher provably never exposed to NYUv2 and re-measure AbsRel and MAE on the same test set.","supporting_citations":[{"cited_title":"Visual prompting via image inpaint- ing","cited_arxiv_id":null,"evidence_quote":"Visual prompting via image inpainting supplies the in-context baseline that LaVin-DiT extends and outperforms."},{"cited_title":"Scalable diffusion models with transformers","cited_arxiv_id":null,"evidence_quote":"The diffusion transformer (DiT) is the backbone that J-DiT builds on."},{"cited_title":"Scaling recti- fied flow transformers for high-resolution image synthesis","cited_arxiv_id":null,"evidence_quote":"MM-DiT provides the condition- and target-specific normalization design that J-DiT adapts with AdaRN."},{"cited_title":"Flow matching for generative modeling","cited_arxiv_id":null,"evidence_quote":"Flow matching supplies the training objective and the conditional flow-matching loss used for J-DiT."},{"cited_title":"Indoor segmentation and support inference from rgbd images","cited_arxiv_id":null,"evidence_quote":"NYU-v2 is the benchmark used to evaluate depth and surface normal estimation."}],"review_version":1}