{"id":"b2e2a72a-cc2b-48a9-938f-34742e85a71e","arxiv_id":"2607.25948","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A decoder-only transformer with two internal experts achieves competitive any-to-any translation across text and 14 vision modalities in one model, using a staged curriculum and uniform timestep sampling.","lead":"MODUS is a single decoder-only AI model that converts between fifteen kinds of visual and text data — photos, depth, edges, segmentation, object boxes, captions, and learned features — with no task-specific output heads. It also chains transformations (text → edges → photo) and can grade its own generated images using another of its own modalities.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot benchmarks rest on a self-reported contamination check that is likely flawed (e.g., DINOv2 training data), so the 'competitive with specialists' claim may be invalidated by train/eval overlap.","rationale":"The reader's weakest assumption is that the training corpus is machine-annotated by teachers and that the zero-shot results are not contaminated by teacher training or source-image overlap. I agree this is the most load-bearing concern. The central claim of competitive zero-shot performance depends entirely on the validity of the zero-shot evaluations. The paper's contamination check is a self-reported assertion without sufficient detail to verify. Moreover, one part of the check is likely false (DINOv2 has seen ImageNet images during pretraining), which casts doubt on the reliability of the other parts. A concrete test is to check overlap between the released MODUS-DATASET and the evaluation sets. If overlap exists, the zero-shot numbers are invalid and the 'competitive' claim weakens; if no overlap, the concern is resolved. I do not see a reason to change the reader's CONDITIONAL verdict: the concern is real but addressable, and the paper's contributions (decoder-only any-to-any recipe, dataset release, ablations) are substantial. Thus UNCHANGED is appropriate.","tokens_in":31483,"tokens_out":12168,"duration_ms":114257,"concrete_test":"Run a duplicate/near-duplicate detection between the released MODUS-DATASET source images (or their URLs/identifiers) and the evaluation sets NYUv2, DIODE, RefCOCO (COCO), and ImageNet, using perceptual hashing (e.g., pHash) and URL matching. If any overlap is found, re-evaluate MODUS on a non-overlapping subset and compare. Also, independently confirm DINOv2's training data composition (e.g., from the DINOv2 paper) to see whether ImageNet-1k images were used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim — that MODUS is competitive with specialists in zero-shot settings (Table 1) — depends on MODUS-DATASET having no overlap with the evaluation benchmarks (NYUv2, DIODE, RefCOCO, ImageNet). The only evidence for this is Appendix C.5, a brief paragraph asserting that the source images (SA-1B, JourneyDB, CC12M) do not overlap. This is not a quantitative check; no hashes, duplicate-detection method, or overlap statistics are provided. The assertion is also contradicted by a known fact: DINOv2's LVD-142M training set includes ImageNet-22k, of which ImageNet-1k is a subset. If DINOv2 was trained on the exact images used for the retrieval evaluation, then the claim 'DINOv2 is not trained on the ImageNet splits used for our retrieval evaluation' is false. This undermines confidence in the entire contamination check. If any evaluation image appears in MODUS-DATASET — e.g., CC12M can contain COCO images (which are the basis of RefCOCO) — then the model has seen test images at train time, making the zero-shot results invalid. Without a verifiable contamination check, the headline claim of competitive zero-shot performance is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MODUS, a decoder-only any-to-any multimodal model built on BAGEL-7B. It supports 15 modalities by mapping all inputs and outputs into a unified token sequence, with a 1D expert for discrete tokens (next-token prediction) and a 2D expert for continuous spatial latents (flow matching). Training uses uniform timestep sampling and a three-stage curriculum on MODUS-DATASET, a 29M-sample corpus with pseudo-labels from pretrained teachers. The paper reports competitive zero-shot results on depth, surface normals, grounding, and retrieval, and demonstrates chained generation, cross-modal self-verification, and visual representation composition. The central claim is that any-to-any generation across diverse modalities can be realized in a single, fully decoder-only model without modality-specific heads, losses, or task pipelines.","tokens_in":31749,"tokens_out":7295,"duration_ms":67015,"significance":"If the central claim holds, this is a significant result: it shows that strong pretrained decoder-only priors can be extended to arbitrary modality-to-modality mappings with a relatively cheap training recipe (roughly 5.7k GH200 hours), rather than training any-to-any models from scratch. The paper's strengths include controlled ablations of timestep sampling, staged training, and unified vs decoupled heads; a generalization experiment to a different base model (Janus-Flow); a self-verification demonstration scored on an external benchmark; and open release of the dataset and checkpoints. The main risk is the contamination check in Appendix C.5, which is self-reported and contains a likely incorrect assertion about DINOv2's training data. If evaluation images overlap with MODUS-DATASET, the zero-shot claims in Table 1 would be invalid. The ablation evidence otherwise supports the proposed recipe, though the 'competitive' claim is not uniformly true across all benchmarks.","major_comments":[{"comment":"The contamination check is a brief assertion with no quantitative evidence. It states 'DINOv2 is not trained on the ImageNet splits used for our retrieval evaluation,' but DINOv2 was pretrained on LVD-142M, which includes ImageNet-22k; ImageNet-1k is a subset, so this claim is almost certainly false. The check also asserts that SA-1B, JourneyDB, and CC12M do not overlap evaluation benchmarks without reporting any hashing, duplicate-detection method, or overlap statistics. This is load-bearing: RefCOCO is built on COCO images, and CC12M is a web-scraped corpus that can plausibly contain COCO images. If any evaluation image appears in MODUS-DATASET, the zero-shot results in Table 1 are invalid. The authors must provide a verifiable contamination analysis (exact and near-duplicate overlap statistics) and correct the teacher-training-data claims.","section":"Appendix C.5 / Table 1"},{"comment":"The claim 'without modality-specific heads, losses, or task pipelines' is overstated as written. The architecture uses two experts with independent parameters and two different training objectives: cross-entropy for 1D tokens (Eq. 3) and flow-matching MSE for 2D latents (Eq. 4). It also relies on modality-specific encoders and tokenizers (SigLIP-2 ViT, FLUX VAE, DINOv2/CLIP/ImageBind tokenizers). This does not invalidate the unified-decoder idea, but the contribution statement should be qualified, for example as 'no task-specific output heads' or 'no per-task pipelines, but with modality-specific input encoders and two training objectives.'","section":"Abstract / Sec. 3.2 / Fig. 2"},{"comment":"The claim of being 'competitive with specialist and multitask baselines' is only partially supported. On DIODE, MODUS (0.285) is worse than DepthAnything2 (0.249); on NYUv2 surface normals, MODUS (19.92) is worse than Marigold (16.40); on ImageNet retrieval, MODUS (77.9/92.5) is worse than DINOv2 (82.1/93.9). Only on RefCOCO grounding does MODUS clearly exceed the specialist (54.5 vs 50.4). The authors should either provide error bars/statistical tests or temper the claim to avoid overstating the comparison.","section":"Table 1 / Sec. 4.2"}],"minor_comments":[{"comment":"'Mode' timestep sampling is not defined. Please define it or cite the source.","section":"Sec. 3.3 / Table 5"},{"comment":"The VQA-based verifier is not described: what prompt is used, and how are VQA answer likelihoods aggregated into a score? Provide details.","section":"Sec. 4.4 / Table 3"},{"comment":"The 'modality instructions' in Figure 2(b) are mentioned but not exemplified. Provide concrete examples of the instruction format used to specify the target modality.","section":"Sec. 3.1 / Fig. 2"},{"comment":"The claim that the GLaMM-based grounding pipeline 'remains zero-shot on COCO' should clarify whether GLaMM's own training data includes COCO images. If it does, the grounding pseudo-labels are not zero-shot with respect to COCO-based benchmarks.","section":"Appendix C.5"},{"comment":"The arrow notation in Eqs. (1) and (2) is informal; define it as denoting the training or inference mapping from conditioning modalities to the target.","section":"Eqs. (1) and (2)"}],"recommendation":"major_revision","confidential_remarks":"The contamination issue is the main risk to the paper's central claim. The authors should be asked to provide a rigorous quantitative overlap analysis between MODUS-DATASET and all evaluation benchmarks, and to correct or justify the DINOv2 training-data claim. Given that other teacher-training claims (DepthAnything, Marigold, GLaMM) are also asserted without verifiable evidence, a careful revision is needed before the zero-shot comparisons can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"For you, the short version: this is a real systems contribution, and the best part is the recipe — uniform flow-matching timestep sampling to stop modality mixing, plus a staged curriculum that extends a pretrained decoder-only MoT model (BAGEL-7B) to 15 modalities without per-modality heads. That finding is concrete, ablated under controlled budgets, and they even show it transfers to Janus-Flow-1.3B. The released dataset and checkpoints are a legit community asset.\n\nWhat isn't as solid is the 'competitive with specialists' headline. The zero-shot numbers depend on a contamination check (Appendix C.5) that is one paragraph of assertions. The stress-test is right: DINOv2 was trained on LVD-142M, which includes ImageNet-22k, which includes the ImageNet-1k images used for retrieval. So 'DINOv2 is not trained on the ImageNet splits used for our retrieval evaluation' is very likely false. CC12M can also contain COCO images, which is what RefCOCO is built on. That doesn't sink the architecture claim, but it means the retrieval and grounding numbers should be treated as unverified until they release overlap statistics or hashes.\n\nThere's a second, less dramatic soft spot: the pseudo-label teachers (DepthAnything, Marigold, GLaMM) are the very specialists MODUS is compared against. So zero-shot depth and normals partly measure imitation of those teachers. That's not a flaw in the recipe — it's the point — but it tempers the 'out-of-the-box' language.\n\nThe ablation suite is genuinely good: from-scratch vs init, decoupled vs unified I/O, timestep sampling, staged curriculum, capability preservation, and the Janus-Flow transfer. The abstract overstates a bit — MODUS is not competitive with DepthAnything2 on DIODE (0.285 vs 0.249), and it loses on RefCOCOg. Single-run numbers without error bars also make the comparisons noisier.\n\nNet: the core recipe and the uniform-timestep insight are worth taking seriously. The benchmark claims need a verifiable contamination check before being cited as evidence. This deserves a serious referee — send it out, but ask for either hashes/overlap stats or softened claims. I'd bring it to reading group; it would generate good discussion about pseudo-label contamination and what 'zero-shot' means in this setup.","headline":"Real recipe-level finding; zero-shot numbers need a verifiable contamination check before the 'competitive with specialists' claim holds.","tokens_in":32345,"tokens_out":2636,"would_cite":true,"duration_ms":25590,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that any-to-any generation across diverse modalities can be realized in a single decoder-only model without modality-specific heads, losses, or task pipelines, and reports competitive zero-shot results across depth, normals","keywords":["any-to-any modeling","decoder-only multimodal generation","flow matching","unified tokenization","modality confusion","chained generation","self-verification","pseudo-label dataset"],"falsifier":"Compare the teacher models' training sets against the evaluation benchmarks used for zero-shot depth, normals, grounding, and retrieval; if any overlap exists, or if a fresh run with provably uncontaminated teachers reproduces the reported numbers, that would settle whether the results reflect genuine cross-modal modeling or imitation of the teachers.","tokens_in":31318,"feed_emoji":"🔀","tokens_out":6440,"duration_ms":52443,"temperature":0.7,"pith_summary":"MODUS sets out to show that any-to-any multimodal generation does not require encoder–decoder or diffusion backbones trained from scratch. The paper argues that one decoder-only transformer, initialized from a pretrained image–text model, can be extended to 15 modalities—depth, surface normals, edges, segmentation, grounding, learned visual features, and more—using a single unified token sequence and two training objectives inside one decoder. If correct, this means strong pretrained decoder-only priors can be reused for arbitrary modality-to-modality mappings with a cheap recipe, rather than building any-to-any models from scratch. The paper reports zero-shot results competitive with specialists on depth, normals, grounding, and retrieval, and it demonstrates chained generation, cross-modal self-verification, and representation composition as consequences of the unified design. The paper also flags two scope limits: it does not support iterative editing within the same modality, and it is pretraining-only without dedicated reasoning post-training (Appendix F).","feed_headline":"One decoder maps any modality into any other","feed_subtitle":"One decoder-only model handles depth, normals, edges, text, grounding, and more—with no task-specific heads or losses.","key_machinery":"The load-bearing mechanism is the unified token sequence with two experts inside one decoder: a 1D Expert (causal attention, next-token prediction) for text, grounding boxes, detection, and feature tokens, and a 2D Expert (bidirectional attention within each modality, flow matching in a VAE latent space) for RGB, depth, normals, segmentation, and edges. Cross-modality attention remains causal so any token can condition later tokens. A second mechanism is uniform timestep sampling for flow matching, which keeps early denoising steps—where the target modality is decided—well represented and prevents modality mixing. A third is a staged curriculum that first learns 1D modalities, then adds 2D m","core_discovery":"MODUS claims that any-to-any multimodal generation can be achieved in a single decoder-only transformer: all modalities are tokenized into one shared sequence, discrete modalities are generated by next-token prediction, continuous spatial modalities are generated by flow matching in latent space, and both share one causal attention context. No modality-specific heads, losses, or task pipelines are used. Starting from a pretrained image–text decoder and training on a 29M-sample pseudo-labelled corpus with uniform timestep sampling and a three-stage curriculum, MODUS reports zero-shot performance competitive with specialist models on depth, surface normal, grounding, and retrieval, while addit","pith_inferences":["If the pseudo-label supervision is clean and the contamination check holds, the results suggest that any-to-any modeling can be obtained by fine-tuning a pretrained vision-language decoder rather than training a new architecture from scratch, which could lower the cost of entering any-to-any domains like genomics or astronomy.","The modality-confusion story—early timesteps decide the target modality—implies that any multimodal flow-matching system sharing a single noise source may need early-timestep weighting; this is testable by reproducing the ablation on other unified diffusion or flow models.","The self-verification result suggests a general design principle: a model that generates all modalities can grade its own samples, which may extend to other modalities such as audio and to reasoning-style evaluation.","Because the released dataset is built from pseudo-labels produced by fixed teacher models, downstream users of the open-source release may inherit biases encoded by those teachers; this is a caution about the resource, not a claim the paper makes."],"forward_implications":["A single MODUS model performs zero-shot depth estimation, surface-normal prediction, grounding, and retrieval at levels near single-task specialists (Table 1).","Chaining through a spatially aligned intermediate (e.g., RGB→Canny→Surface Normal) improves the final target, while redundant or non-aligned intermediates do not (Table 2).","The same model can score its own outputs: best-of-4 self-verification lifts GenEval from 0.81 to 0.84 without an external verifier (Table 3).","The training recipe transfers: applying the same pipeline to a different 1.3B base model adds depth and surface-normal abilities at roughly 13× less compute than the base model's original pretraining (Appendix B.5).","Multi-conditioning (e.g., RGB+Depth→Surface Normal) beats single-condition RGB→Surface Normal, and Edge+Depth→Surface Normal nearly matches the RGB-based baseline without RGB input (Appendix C.4)."],"fun_headline_variants":["One decoder-only transformer for all any-to-any modality pairs","Single decoder maps any input modality to any output, no heads","Any-to-any generation with one decoder, no task-specific piping","Decoder-only MODUS: all modalities, one network, zero extra heads","From text to depth to grounding: one decoder unifies it all"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The training corpus is machine-annotated by a fixed set of pretrained teachers, and the zero-shot results assume those teachers were not trained on the evaluation benchmarks and that their pseudo-labels are accurate enough to teach true cross-modal structure; the contamination check in Appendix C.5 is self-reported and cannot be independently verified from the text.","fun_headline_variants_meta":{"raw":{"variants":["One decoder-only transformer for all any-to-any modality pairs","Single decoder maps any input modality to any output, no heads","Any-to-any generation with one decoder, no task-specific piping","Decoder-only MODUS: all modalities, one network, zero extra heads","From text to depth to grounding: one decoder unifies it all"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000402,"raw_usage":{"total_tokens":1916,"prompt_tokens":713,"completion_tokens":1203,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1113}},"tokens_in":457,"tokens_out":1203,"duration_ms":9238,"temperature":1.0,"reasoning_tokens":1113,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:01:03.383945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the teacher models' training sets against the evaluation benchmarks used for zero-shot depth, normals, grounding, and retrieval; if any overlap exists, or if a fresh run with provably uncontaminated teachers reproduces the reported numbers, that would settle whether the results reflect genuine cross-modal modeling or imitation of the teachers.","supporting_citations":[],"review_version":1}