{"id":"e9d5aaeb-f704-4025-8997-68f625150d56","arxiv_id":"2412.10758","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"An encoder-free vision-language model that claims state-of-the-art scores; the paper lacks any reproducible evidence.","lead":"This paper proposes MUDAIF, a decoder-only vision-language model that turns images into pseudo-text tokens and fuses them with text via a co-attention module. The authors say it beats existing models on several benchmarks, but they provide no code, data, or enough detail to check the results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) is ill-typed unless the pseudo-token count N equals the text token count L; the paper never states or justifies this, so the core fusion mechanism and all reported results are not attached to a well-defined model.","rationale":"I read the paper as making an empirical claim: an encoder-free decoder-only vision-language model with a Vision-Token Adapter and adaptive co-attention achieves state-of-the-art results on VQA, captioning, and reasoning benchmarks. For that claim to be true, the described architecture must be realizable and the training/evaluation protocol must be reproducible. The weakest point is not the absence of code or error bars, though those matter; it is that the central fusion equation is mathematically ill-defined for the usual case where visual token count differs from text token count. The reader's verdict identified exactly this as the weakest assumption, and my analysis agrees. The dimensional mismatch in Eq. (5) means that either the equation is wrong, or the model silently requires N = L without ever stating it. Even if N = L were intended, variable-length text generation at inference time would break the sum unless text is padded or truncated to exactly N tokens, which is not mentioned. The additional self-referential notation in Eq. (3) further undermines the specification of the VTA. Since the method cannot be instantiated as written, the reported benchmark numbers lack a coherent referent. No amount of additional hyperparameter detail or baseline curation would fix this without a substantive revision of the fusion mechanism or a clear and justified constraint on token lengths. Therefore, I do not change the reader's reject verdict.","tokens_in":7465,"tokens_out":3775,"duration_ms":34989,"concrete_test":"Ask the authors to provide a minimal executable implementation of Eqs. (2)-(5) with explicit tensor shapes, e.g., a PyTorch function that takes an image I and a token list T and returns Z. Run it with N = 256 visual tokens and text length L = 32. If the addition in Eq. (5) fails, or if the implementation avoids Eq. (5) by transposing, padding, concatenating, or otherwise changing the stated operation, then the architecture is not as written. Alternatively, independently derive the tensor ranks: confirm that Avt T is [N, d] and Atv V is [L, d], so the sum is undefined for N != L.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MUDAIF, a decoder-only model with a Vision-Token Adapter and adaptive co-attention, outperforms state-of-the-art baselines. For that claim to be meaningful, the architecture must be implementable. The load-bearing weakness is the fusion step in Eq. (5). In Eq. (4), Qv is naturally the query from the N visual pseudo-tokens and Kt is the key from the L text tokens, so Avt = Softmax(Qv Kt^T / sqrt(d)) has shape [N, L]. Symmetrically, Atv has shape [L, N]. If T is the text embedding matrix of shape [L, d] and V is the visual pseudo-token matrix of shape [N, d], then Avt T has shape [N, d] and Atv V has shape [L, d]. Adding them in Eq. (5) is well-defined only when N = L. The paper specifies V in R^{N x d} and T as a sequence of L tokens, but gives no relation between N and L. During inference, text length varies, so even if N and L were equal at training time through padding, the sum would become ill-defined for inputs whose text length differs from N. A second notational blocker compounds this: Eq. (3) defines V as the output of self-attention while also using V as the value matrix on the right-hand side, making the VTA itself a circular definition. These are not cosmetic issues; they prevent the model from being built, so the accuracy numbers in Tables I and II cannot be attributed to the system as described.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MUDAIF, a decoder-only vision-language model that removes the separate visual encoder and instead converts image patches into pseudo-text tokens via a Vision-Token Adapter (VTA), then fuses visual and textual tokens with an adaptive co-attention mechanism. The paper reports evaluations on VQA-v2, GQA, VizWiz, SEED, MM-Vet, and a BLEU-based captioning measurement against InstructBLIP, LLaVA-1.5, and EVE-7B, and it claims that MUDAIF consistently outperforms these baselines. Ablation studies, human evaluations, robustness tests, and generalization experiments are also reported. The central contribution is the claim that an encoder-free decoder-only model with a simple adapter and co-attention can match or beat established encoder-based VLMs.","tokens_in":7790,"tokens_out":4096,"duration_ms":38272,"significance":"Encoder-free vision-language models are a timely direction, and the paper identifies a real efficiency and scalability motivation for avoiding a dedicated visual encoder. The proposed VTA and adaptive co-attention mechanisms could in principle be a useful contribution if they were well-defined and rigorously validated. However, the manuscript does not provide an implementable architecture: the key fusion equation is dimensionally ill-typed, and the empirical section consists of bare tables without training details, error bars, code, or evaluation protocols. The central claim of the abstract is therefore not supported by the presented evidence. Credit is due for including an ablation that separates the VTA and co-attention contributions, but this does not compensate for the missing definitions and verifiability.","major_comments":[{"comment":"The fusion equation Z = A_vt T + A_tv V is not well-defined for token sequences of different lengths. With A_vt in R^{N x L}, A_tv in R^{L x N}, T in R^{L x d}, and V in R^{N x d}, the products A_vt T and A_tv V have shapes N x d and L x d, respectively, and can be summed only when N = L. The paper never states or justifies this equality, and during autoregressive inference the text length L varies, so the sum cannot be evaluated even if training sequences were padded to a common length. Because this fusion step is the central mechanism of the model, the accuracy numbers in Tables I and II cannot be attributed to the system as described.","section":"III-C, Eq. (5)"},{"comment":"Equation (3) defines V as the output of self-attention while using V as the input value matrix on the right-hand side, making the definition circular. A separate value matrix, e.g., V' = Linear(V), or distinct notation is needed. As written, the VTA cannot be instantiated, and this compounds the shape problem in Eq. (5) because V is a core operand in both equations.","section":"III-B, Eq. (3)"},{"comment":"The quantitative claims are presented as bare numbers with no training configuration (model size, optimizer, learning rate, batch size, learning-rate schedule, number of steps), no image preprocessing or prompt-template details, no decoding settings, no error bars or significance tests, and no code or data release. The human evaluation in Table III and the user study in Table VI do not report the number of raters, inter-annotator agreement, or statistical intervals. Without this information, the abstract's claim that MUDAIF 'consistently outperforms state-of-the-art methods' is not verifiable.","section":"IV, Tables I-VI"},{"comment":"Only three baselines are compared, and the paper does not state whether their numbers were reimplemented or taken from the literature, nor does it include standard recent encoder-free baselines used in the EVE paper. The evaluation protocol is also incomplete: captioning is measured by BLEU, but no captioning benchmark is named, and the SEED and MM-Vet composite scores are not defined. This makes the head-to-head comparisons in Tables I, II, IV, and V difficult to interpret.","section":"IV-A"}],"minor_comments":[{"comment":"There are typographical inconsistencies in benchmark and model names, such as 'MM-V et' for MM-Vet and 'LLaV A-1.5' for LLaVA-1.5.","section":"Throughout"},{"comment":"The losses are defined only over text tokens t_i, but no masking or attention structure is specified for how V interacts with the decoder's causal masking; a precise description of the token layout would clarify the training objective.","section":"III-D, Eqs. (6)-(7)"},{"comment":"The term 'pseudo-text tokens' is introduced without specifying token lengths, embedding initialization, or how these tokens are positioned relative to text tokens in the unified sequence; a concrete example would help.","section":"III-A"},{"comment":"The introduction mentions COCO, LAION, and Visual Genome as training data, but no references or data splits are provided, and the 45M-pair dataset is not described in terms of filtering, deduplication, or train/validation separation.","section":"I and IV-A"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an early draft with a load-bearing architectural definition that is not mathematically valid, and the experimental section lacks the detail required to support the claimed state-of-the-art results. Even with a generous reading, the paper does not currently support its abstract. I recommend rejection rather than major revision because the core fusion mechanism and the evaluation would need to be redefined and re-run; that is beyond a normal revision cycle. If the authors can supply a corrected fusion definition, complete implementation details, and reproducible experiments, a fresh submission would be the appropriate venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this one should be desk-rejected, and not just for missing details. The central fusion equation, Eq. (5), is ill-defined for unequal token counts, so the method as written cannot be implemented. The stress-test note is correct: Avt T is [N,d], Atv V is [L,d], and adding them requires N=L, which is never stated or justified. And Eq. (3) is circular, defining V in terms of V.\n\nWhat's genuinely here: the paper correctly motivates encoder-free VLMs and points to EVE as the main predecessor. The VTA is a conv plus linear projection, which is a standard projector; the adaptive co-attention is a bidirectional cross-attention sum. The only architectural delta over EVE is that sum, which is routine. So novelty is modest.\n\nThe experiments are bare tables. No training hyperparameters, no evaluation protocol, no error bars, no code, no model release. ViLT is misdescribed as encoder-free and decoder-only when it is neither, and it's cited with the wrong reference. Ablation numbers come without variance. The human evaluation is a Likert table with no rater details or instructions. None of this supports the abstract's \"new standard\" claim.\n\nThe technical blocker is not cosmetic. Eq. (5) is undefined unless N is tied to L, and the paper never says how that tie is handled during inference. A referee could ask for clarification, but the paper gives too little to even start a useful review.\n\nWho gets value from this? Someone skimming the related-work framing might find the encoder-free motivation worth a minute. No serious reader should build on the method as stated, and I wouldn't cite it.\n\nRecommendation: desk reject. If the authors return with a well-defined fusion rule, an actual training setup, and reproducible evaluation, it could be worth another look.","headline":"Eq. (5) doesn't type-check unless N equals L, so the reported SOTA numbers are unattached to a buildable model.","tokens_in":8318,"tokens_out":2240,"would_cite":false,"duration_ms":21668,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MUDAIF, a decoder-only model that turns images into pseudo-text tokens with a Vision-Token Adapter, claims to outperform encoder-based VLMs across VQA, captioning, and reasoning benchmarks.","keywords":["vision-language model","decoder-only architecture","encoder-free VLM","vision-token adapter","adaptive co-attention","visual question answering","image captioning","multimodal reasoning"],"falsifier":"Implement the published equations with an image that produces N pseudo-tokens and a text prompt of length L, with N ≠ L: the addition A_vt T + A_tv V is undefined because the two terms have shapes N×d and L×d. Observing that failure, or running the model with the fusion step replaced by a simple concatenation, would directly test whether the reported VQA-v2 80.3 result depends on an actually implemented alignment not described in the paper.","tokens_in":7243,"feed_emoji":"🖼️","tokens_out":5960,"duration_ms":48627,"temperature":0.7,"pith_summary":"MUDAIF is a vision-language model built entirely from a decoder Transformer, with no separate image encoder. The paper's central claim is that converting images into pseudo-text tokens through a Vision-Token Adapter, combined with a bidirectional adaptive co-attention mechanism, aligns visual and textual representations better than encoder-based designs. On the benchmarks reported, MUDAIF scores 80.3 on VQA-v2, 65.5 on GQA, and 53.7 on VizWiz, and the authors report 20-35 percent lower training time and 1.5x faster inference at high resolution. If these results hold, encoder-free decoder-only architectures would become a credible alternative to the dominant encoder-decoder paradigm for multimodal tasks.","feed_headline":"Decoder-only model beats encoder VLMs on VQA and reasoning","feed_subtitle":"Its vision-token adapter reports higher VQA scores while cutting training time by up to 35 percent.","key_machinery":"Vision-Token Adapter (VTA): turns the image I into a sequence V of N pseudo-text tokens via Conv, Linear, and self-attention, so the decoder can consume vision like language. Adaptive Co-Attention: two softmax attention maps, A_vt from visual to text and A_tv from text to visual, combine into Z = A_vt T + A_tv V; this is the mechanism claimed to enable bidirectional cross-modal interaction. The shared decoder then autoregressively generates text conditioned on Z.","core_discovery":"The paper argues that the visual encoder is not necessary for state-of-the-art vision-language performance. Instead, a lightweight Vision-Token Adapter (a convolutional embedding, a linear projection, and self-attention) maps the raw image into N pseudo-text tokens, which a shared decoder processes together with text tokens. An adaptive co-attention mechanism computes attention from visual to text and from text to visual, fusing the two sequences into a joint representation Z = A_vt T + A_tv V. Trained on 45M image-text pairs and instruction-tuned, MUDAIF is reported to beat InstructBLIP, LLaVA-1.5, and EVE-7B on VQA-v2, GQA, VizWiz, SEED, and MM-Vet, as well as in human ratings of relevance, coherence, and informativeness.","pith_inferences":["Editorial inference: the fusion equation as printed, Z = A_vt T + A_tv V, requires N and L to match or an unstated projection; a reader cannot reproduce the architecture from the paper alone, and the performance claim depends on that missing piece.","Editorial inference: if the VTA is truly task-agnostic, it should transfer to other decoder-only language models without retraining; testing that transfer would separate the adapter's contribution from the decoder's scale.","Editorial inference: the reported efficiency gains are relative to encoder-based baselines; a direct comparison against a same-scale encoder-free model like EVE-7B would clarify how much of the gain comes from removing the encoder versus from the co-attention design.","Editorial inference: extending the pseudo-token mechanism to video frames or audio spectrograms is a natural next step that the paper lists as future work; the same adapter logic could apply with minor modifications."],"forward_implications":["If the reported gains are reproducible, encoder-free models can match or exceed encoder-based VLMs without inheriting resolution and aspect-ratio constraints.","The 20-35 percent training-time reduction and 1.5x inference speedup would make decoder-only VLMs more practical for resource-constrained deployment.","The VTA's pseudo-token design suggests that the same pre-trained decoder can absorb new modalities by adding lightweight adapters rather than retraining a full encoder.","Instruction tuning on top of the 45M-pair pretraining would let a single model serve VQA, captioning, and reasoning tasks with one shared parameter set.","If the high cosine similarity (0.82) between visual and textual embeddings is confirmed, the co-attention mechanism directly improves cross-modal alignment."],"supporting_citations":[{"why":"EVE-7B, the encoder-free baseline MUDAIF must beat; it motivates the decoder-only design and appears in all comparison tables.","marker":"[1]"},{"why":"VisionLLM's 'image as a foreign language' idea underlies the Vision-Token Adapter's translation of images into tokens.","marker":"[6]"},{"why":"VisionLLM v2's unified decoder for hundreds of vision-language tasks provides the architectural precedent for a shared multimodal decoder.","marker":"[7]"},{"why":"MoE-LLaVA's mixture-of-experts efficiency discussion frames the computational-efficiency claims.","marker":"[8]"},{"why":"MMStar's analysis of evaluation pitfalls supports the benchmark choices and the concern about data leakage.","marker":"[13]"},{"why":"InternVL's scaling study grounds the decision to pretrain on 45M image-text pairs.","marker":"[14]"},{"why":"ViLT, the convolution-free encoder-free baseline, supports the claim that visual data can be processed like text.","marker":"[18]"}],"fun_headline_variants":["Decoder-only model ditches visual encoder, tops VQA","Vision-token adapter replaces encoder, improves multimodal fusion","MUDAIF: no encoder needed, beats InstructBLIP and LLaVA","Decoder-only model with adaptive co-attention outperforms encoder baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the visual and textual token sequences can be fused by the vector addition in Eq. (5); the paper never specifies how token sequences of different lengths are aligned or projected, so if that sum cannot be defined the architecture as written cannot be built.","fun_headline_variants_meta":{"raw":{"variants":["Decoder-only model ditches visual encoder, tops VQA","Vision-token adapter replaces encoder, improves multimodal fusion","MUDAIF: no encoder needed, beats InstructBLIP and LLaVA","Decoder-only model with adaptive co-attention outperforms encoder baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001526,"raw_usage":{"total_tokens":6076,"prompt_tokens":875,"completion_tokens":5201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":5128}},"tokens_in":491,"tokens_out":5201,"duration_ms":34406,"temperature":1.0,"reasoning_tokens":5128,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:37:32.645959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Implement the published equations with an image that produces N pseudo-tokens and a text prompt of length L, with N ≠ L: the addition A_vt T + A_tv V is undefined because the two terms have shapes N×d and L×d. Observing that failure, or running the model with the fusion step replaced by a simple concatenation, would directly test whether the reported VQA-v2 80.3 result depends on an actually implemented alignment not described in the paper.","supporting_citations":[{"cited_title":"Visionllm: Large language mode l is also an open-ended decoder for vision-centric tasks,","cited_arxiv_id":null,"evidence_quote":"VisionLLM's 'image as a foreign language' idea underlies the Vision-Token Adapter's translation of images into tokens."},{"cited_title":"Vilt: Vision-and-language t ransformer without convolution or region supervision,","cited_arxiv_id":null,"evidence_quote":"ViLT, the convolution-free encoder-free baseline, supports the claim that visual data can be processed like text."}],"review_version":1}