{"id":"cb60ed22-6998-4f70-b1c2-fa217d4729a4","arxiv_id":"2507.21741","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multimodal LLM with a learnable alignment projector and a combined cross-entropy and mean-squared-error loss reports state-of-the-art scores on several vision-language benchmarks using only 144 visual tokens.","lead":"Researchers built MAGE, a multimodal AI model that uses a new visual alignment module and a dual loss (text generation plus embedding distance) to report top scores on benchmarks such as MME, MMBench, and SEED while using fewer visual tokens. The same model plans and calls external tools to produce images, audio, and video from complex instructions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's benchmark gains may come from MAGE's full fine-tuning of CLIP+LLM and 980K data, not from IAN/dual loss; Tables 3-4 never state whether controlled rows match that recipe.","rationale":"The paper's central contribution is an architecture and a loss function, not a training recipe. Section 4.1 announces full fine-tuning of both CLIP and the LLM plus a 980K-sample mixture, which are known to move VLM benchmark scores substantially. Table 1's baselines were trained under different recipes, so the headline deltas cannot be assigned to IAN or the dual loss without further control. Table 4 is the natural controlled comparison, but the text omits whether B1-B7 share the same data, freeze flags, and loss weights; Table 3 similarly omits these details for the 'w/o IAN proj.' row. The reader's condition is exactly right: release code and configs, and show same-recipe comparisons. If those configs confirm that Table 4 used the same full-fine-tuning protocol throughout, the central claim is substantially supported; until then the strongest claim is unverified. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":12953,"tokens_out":8653,"duration_ms":98251,"concrete_test":"Ask the authors to release the exact training configuration, including freeze flags, data lists, steps, learning rates, and loss weights, for Table 3's rows and Table 4's B1-B7 rows. Then rerun the decisive comparison, IAN versus C-Abstractor at 144 tokens under an identical full-fine-tuning, 980K-sample, 336-resolution protocol, on MMBench, SEED-Bench, and MME. If IAN's margin persists, the attribution is supported; if it collapses, Table 1's claim is confounded by training recipe and data volume.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the IAN projector and ITG+ITDM dual loss deliver SOTA benchmark results with 144 visual tokens. For that attribution to hold, Table 1 must compare like with like. Section 4.1 states that MAGE fine-tunes all parameters, including CLIP and the LLM, across Stages 1-3, using about 980K samples in Stage 2. Among the listed baselines, LLaVA-1.5 keeps the CLIP vision encoder frozen and trains on 665K samples; Honeybee's C/D-Abstractor rows are published with a frozen vision encoder and frozen LLM. The 6.9-point MMBench gap over LLaVA-1.5 (71.2 vs 64.3) and the 2.3-point gap over Honeybee D-Abstractor (71.2 vs 70.8 at 7B) are therefore confounded by training recipe and data volume. The paper's internal comparisons could resolve this: Table 4 compares Linear, Resampler, C-Abstractor, and IAN under what appears to be the same MAGE pipeline, but the manuscript never states that B1-B7 share the same data, freeze flags, learning rates, and loss weights. Likewise, Table 3's 'w/o IAN proj.' row does not say whether the vision encoder and LLM remain fully fine-tuned. Without that specification, the strongest benchmark claim is not attributable to the proposed components.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MAGE, a vision-language model architecture with three components: IAN (Vector Alignment Block plus Semantic Enhancement Block), a dual-loss training objective (ITG and ITDM), and an HMDSet tool-use dataset. It reports MME, MMBench, SEED-Bench, and POPE results for Vicuna-7B/13B with 64-256 visual tokens, claiming state-of-the-art performance with 144 tokens, and shows example tool-calling workflows for text-to-image, text-to-video, and image-to-audio/video generation.","tokens_in":13241,"tokens_out":6591,"duration_ms":67944,"significance":"If the architecture and losses are responsible for the reported gains, the result would be practically useful: with 144 tokens, MAGE matches or exceeds models using 256 or 576 tokens, and the structured JSON tool-use interface extends the model toward compound multimodal generation. The paper has potentially informative internal comparisons in Table 4, where IAN is ranked against Linear, Resampler, and C-Abstractor at matched token counts, and the ablations in Table 3 are directionally consistent. The authors also promise complete code and an appendix, which would support reproducibility. However, the headline benchmark comparison is confounded by full fine-tuning and a different data schedule, and several load-bearing implementation details are underspecified, so the significance is currently conditional on a controlled evaluation.","major_comments":[{"comment":"The central claim that IAN and the dual loss deliver the reported state-of-the-art accuracy is not supported by Table 1 as presented. Section 4.1 states that MAGE fine-tunes all parameters, including CLIP and the LLM, and uses roughly 980K samples in Stage 2, while the published baselines cited in Table 1 (e.g., LLaVA-1.5 and Honeybee C/D-Abstractor) are typically trained with a frozen vision encoder and/or a frozen LLM and with smaller or different data mixtures. For example, LLaVA-1.5 keeps CLIP frozen and uses 665K visual instruction samples, and Honeybee's published rows use a frozen vision encoder and frozen LLM. The 6.9-point MMBench gap over LLaVA-1.5 and even the smaller gap over D-Abstractor can therefore reflect full fine-tuning and extra data rather than the proposed IAN module. To support the attribution, the authors should either train the baselines under the same full-fine-tuning recipe and data schedule or clearly report the freeze flags, data, and schedule for every row in Table 1.","section":"§4.1, Table 1"},{"comment":"The dimensions in the SEB description are inconsistent. The text says the CNN produces b ∈ R^(N×D_b) as the query, while a ∈ R^(N×D_l) is used as key and value, and the output V'' is declared to be in R^(N×D_l). In standard attention, the output follows the query dimension, so if D_b differs from D_l the attention output would not be D_l-dimensional unless an additional projection exists. Please specify D_b, state whether a projection maps the query to D_l, and give the exact ordering of operations, since IAN is the core architectural contribution.","section":"§3.1, SEB Module"},{"comment":"The ITDM loss is not well defined. Equation (2) compares d_ian and d_llm, but the manuscript does not state how d_ian is derived from the IAN output, how d_llm is derived from the LLM, which LLM layer is used, or how per-token embeddings are aggregated into a single vector per sample. Without this specification, the dual-loss training strategy cannot be reproduced, and it is unclear whether the two vectors are even in the same dimensional space. Please give the exact computation for both quantities.","section":"§3.2, Eq. (2)"},{"comment":"The internal comparisons meant to isolate the effect of IAN do not state whether the rows are controlled. Table 3 removes the IAN projector and the alignment strategy, and Table 4 compares Linear, Resampler, C-Abstractor, and IAN, but neither table reports whether all rows share the same data mixture, freeze flags, learning rate, number of steps, and loss weights. Since Section 4.1 specifies full fine-tuning of all parameters, the 'w/o IAN proj.' and 'w/o IAN & align.' rows must state whether they also fine-tune CLIP and the LLM, or the ablation does not isolate the proposed components. Please add a sentence for Tables 3 and 4 specifying the controlled recipe.","section":"§4.4-4.5, Tables 3-4"},{"comment":"The tool-use/tool-calling contribution is only demonstrated with three hand-picked examples in Figure 4 and is not evaluated quantitatively. There is no reported metric for task-planning accuracy, tool-call validity, end-to-end task success, or human preference, so the claim that HMDSet 'extends the model's output capability boundaries' is not verified. A small evaluation with task success rates or human ratings would be needed to make this contribution testable.","section":"§4.6"}],"minor_comments":[{"comment":"Stage 2 says the 665K mixture is combined with ALLaVA-Instruct-VFLAN-4V to reach about 980K samples, but Table 2 does not list ALLaVA or its sample count; please correct the table-text mismatch.","section":"§4.1, Table 2"},{"comment":"Y is defined as the aligned visual embedding, but the loss is computed over text tokens y_i; the notation should clearly distinguish visual tokens from text tokens.","section":"§3.2, Eq. (1)"},{"comment":"The paper reports no variance or repeated runs; please state whether these are single-run results and, where feasible, include seed information or standard deviation.","section":"Tables 1, 3-5"},{"comment":"Avg N is introduced as a normalized average but its formula and normalization constants are not given; please provide the definition or cite the exact source.","section":"§4.2"},{"comment":"The abbreviation 'align.' is not defined in the table caption or the surrounding text; it should be expanded to 'dual-loss alignment strategy' at first use.","section":"§4.4, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's publication value depends almost entirely on whether the authors can rule out the recipe and data confound in Table 1. If the GitHub repository contains the controlled runs or if the authors re-run the baselines under the same full-fine-tuning schedule, the central claim would be much stronger. I would also verify that the baseline numbers are obtained with the same evaluation scripts rather than copied from prior papers. The tool-use section is currently more of a demonstration than an evaluated contribution and would not by itself support the 'Any-to-Any' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid systems paper, but the central claim—that IAN and the dual loss drive the benchmark gains—is not actually supported by the evidence as presented, because the Table 1 comparisons are not controlled for training recipe or data volume.\n\nWhat's genuinely new: the IAN design that decouples dimensional alignment (a linear/MLP block) from semantic enrichment (a cross-attention block where CNN-based local patch features query global CLIP features) is a clean, reasonable idea. The dual CE+MSE loss is a small twist on existing alignment losses, and the HMDSet dataset plus the hierarchical JSON tool-calling protocol are concrete artifacts. The framework is described clearly enough to reproduce, and the internal numbers are self-consistent: Table 3 ablations drop monotonically, Table 5 scales sensibly with tokens and model size. So I believe the model works as a system.\n\nThe soft spot is attribution. Section 4.1 says MAGE fine-tunes all parameters on roughly 980K Stage-2 samples, while LLaVA-1.5 and the Honeybee rows in Table 1 came from different pipelines—typically a frozen CLIP, less data, and different schedules. The 0.4-point gap over Honeybee at 7B on MMBench and the 6.9-point gap over LLaVA-1.5 are well within what recipe differences can produce. The paper never states whether the 'w/o IAN proj.' row in Table 3 or the projector rows in Table 4 share the exact same freeze flags, data, and loss weights. If they do, those ablations are meaningful; if not, the central claim is unsupported.\n\nTwo smaller issues: no error bars or repeated runs, which matters for 0.5–2 point differences, and the ITDM loss is underspecified—'text vector encoded by the LLM' doesn't say which layer or which text gets encoded, or how the MSE is masked. Fixable, but needed for reproducibility.\n\nWho this is for: people working on efficient visual token compression or VLM tool-calling agents. It deserves a serious referee because the architecture is plausible and the internal comparisons support the design choices, but the authors should be required to run a controlled experiment or at least state the training recipe for every row in Tables 3–4, and pin down the ITDM target. I'd accept it for review with the expectation of heavy revision.","headline":"A competent systems paper with an incrementally new projector and a useful tool-calling dataset, but the headline benchmark claim is confounded by training-recipe differences.","tokens_in":13785,"tokens_out":3843,"would_cite":false,"duration_ms":41574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new projection module, IAN, paired with a generation-and-distance dual loss, lets MAGE beat larger multimodal models while using only 144 visual tokens.","keywords":["MAGE","multimodal large language model","visual projector","cross-modal alignment","visual token compression","dual-loss training","tool calling","Any-to-Any generation"],"falsifier":"Re-run the Stage 2 recipe with IAN replaced by a plain linear projector, keeping the same 980K samples, the same full fine-tuning, and the same dual loss. If MMBench and SEED stay near 71.2 and 65.3, then the projector is not the cause; if removing the ITDM loss alone reproduces the 'w/o align.' numbers, then the MSE term is not doing the claimed work.","tokens_in":12758,"feed_emoji":"🖼️","tokens_out":8643,"duration_ms":86940,"temperature":0.7,"pith_summary":"Multimodal models lose information when visual features pass through the projector into the language model, and the paper proposes MAGE as a fix. MAGE's Intelligent Alignment Network (IAN) separates the job into two blocks: one aligns the vector dimension of CLIP features to the LLM, and the other injects local patch detail through cross-modal self-attention. Training combines an image-to-text generation loss with an image-text distance minimization loss, so the projector learns to place images near their captions in embedding space. The paper reports that with only 144 visual tokens, MAGE beats comparable 7B and 13B systems on MME, MMBench, and SEED, and that a new tool-scheduling dataset, HMDSet, lets the model call external generators for images, audio, and video. A sympathetic reader would care because the result points to a cheaper alignment path: fewer tokens, better scores, and a clear separation of alignment from semantic enrichment.","feed_headline":"MAGE uses 144 visual tokens to outscore larger-token rivals","feed_subtitle":"New projector plus dual loss lifts MME, MMBench, and SEED scores while cutting visual token count.","key_machinery":"The load-bearing object is the Intelligent Alignment Network (IAN), made of the Vector Alignment Block (VAB) and the Semantic Enhancement Block (SEB). VAB is a nonlinear MLP plus learnable normalization that converts $N \\times D_v$ encoder embeddings into $N \\times D_l$ LLM-dimensional embeddings; SEB uses a CNN patch embedding as query and the VAB output as key/value in self-attention, so local detail is added without changing the aligned dimension. The training mechanism is the composite loss $L = L_{\\text{ITG}} + L_{\\text{ITDM}}$, where the generation term is autoregressive cross-entropy and the distance term is mean squared error between the IAN-mapped image vector and the LLM-encoded caption vector. The paper also relies on the observation that 2D pooling at the patch level can reduce visual tokens without a Q-Former, which is what lets IAN operate at 144 tokens.","core_discovery":"The central claim is that dimensional mismatch and semantic loss are separate problems and should be solved by separate modules. The Vector Alignment Block (VAB) is an MLP with learnable normalization that maps visual embeddings from the encoder dimension to the LLM dimension while retaining context; the Semantic Enhancement Block (SEB) then runs cross-modal self-attention where a CNN patch embedding acts as query and the VAB-aligned global embedding acts as key and value, fusing local and global information. The dual-loss strategy adds an image-text distance minimization (ITDM) MSE term to the usual image-text generation (ITG) cross-entropy loss, so the aligned image vector is pushed toward the LLM's encoding of the caption. The authors demonstrate the claim with ablations; removing IAN, the alignment loss, or both progressively lowers scores on MMBench, SEED, MME, and POPE, and a matched-token projector comparison places IAN above linear, Resampler, and C-Abstractor on spatial reasoning tasks.","pith_inferences":["Inference: VAB/SEB decoupling should transfer to other visual encoders and LLMs; the design principle is that dimension matching and semantic enrichment can be trained separately, which is not restricted to CLIP and Vicuna.","Inference: The ITDM MSE target is a single LLM-encoded caption; a testable extension would use learned anchors or multiple caption embeddings to make the distance signal more stable and less sensitive to the caption.","Inference: HMDSet, released with the code, could serve as a benchmark for multimodal tool scheduling; its manual review of GPT-4 dialogues gives it a quality bar that future datasets can be measured against.","Inference: The 144-token results suggest a practical latency/accuracy tradeoff; measuring throughput and memory at 144 vs 576 tokens on the same hardware would quantify the efficiency benefit the paper claims."],"forward_implications":["As reported, MAGE with 144 tokens reaches MMBench 71.2 (7B) and 73.9 (13B), surpassing LLaVA-1.5, LLaVA-NeXT, and TokenPacker at 256 or 576 tokens.","On six spatial tasks in MMBench and SEED-Bench, the IAN projector wins the normalized average over Linear, Resampler, and C-Abstractor at matched token counts.","The ablations show each component matters: removing IAN drops SEED from 65.3 to 62.9, removing the alignment loss drops it to 61.4, and removing both drops it to 60.9.","The token-scaling table shows monotonic gains from 64 to 144 to 256 tokens and from 7B to 13B, which is consistent with the efficiency claim.","With HMDSet training, the model outputs structured JSON tool plans and can chain image-to-text, text-to-audio, and text-to-video tools for composite generations."],"supporting_citations":[{"why":"Honeybee supplies the C-Abstractor and D-Abstractor baselines at 144 tokens that MAGE is directly compared with in Table 1 and Table 4.","marker":"[Cha et al., 2024]"},{"why":"LLaVA-1.5 is the 576-token linear-projector baseline whose recipe and scores anchor the main comparison.","marker":"[Liu et al., 2024a]"},{"why":"BLIP-2 introduces the Q-Former projector that IAN is positioned against, and supplies the BlipCapFilt pretraining data used in Stage 1.","marker":"[Li et al., 2023b]"},{"why":"TokenPacker provides token-efficient projector baselines at 64 tokens whose performance MAGE claims to surpass.","marker":"[Li et al., 2024b]"},{"why":"CLIP ViT-L/14 is the visual encoder whose penultimate-layer features IAN aligns; the whole alignment pipeline starts from these embeddings.","marker":"[Radford et al., 2021a]"},{"why":"MME is one of the three main benchmark families reporting the perception and cognition numbers in Table 1.","marker":"[Fu et al., 2024]"},{"why":"MMBench supplies the overall and Chinese scores plus the spatial-relationship subtasks used for the projector analysis in Table 4.","marker":"[Liu et al., 2025b]"},{"why":"ALLaVA-Instruct-VFLAN-4V composes 90% of the Stage 2 instruction-tuning mixture, so the data recipe depends on it.","marker":"[Chen et al., 2024]"}],"fun_headline_variants":["MAGE's dual-loss alignment beats standard projectors on spatial","MAGE's IAN: separate fixes for dim and semantic gaps","MAGE's any-to-any tool tuning expands output capabilities","MAGE's IAN and dual loss lift MME, MMBench, SEED scores","MAGE's projector beats linear, Resampler on spatial tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that IAN and the dual loss cause the benchmark gains assumes the comparisons are fair, even though MAGE fully fine-tunes CLIP and the LLM on a different 980K-sample mix and the 'w/o IAN' ablation does not state whether it uses those same settings.","fun_headline_variants_meta":{"raw":{"variants":["MAGE's dual-loss alignment beats standard projectors on spatial","MAGE's IAN: separate fixes for dim and semantic gaps","MAGE's any-to-any tool tuning expands output capabilities","MAGE's IAN and dual loss lift MME, MMBench, SEED scores","MAGE's projector beats linear, Resampler on spatial tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002632,"raw_usage":{"total_tokens":10077,"prompt_tokens":976,"completion_tokens":9101,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":9006}},"tokens_in":592,"tokens_out":9101,"duration_ms":77913,"temperature":1.0,"reasoning_tokens":9006,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:24:41.458065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Stage 2 recipe with IAN replaced by a plain linear projector, keeping the same 980K samples, the same full fine-tuning, and the same dual loss. If MMBench and SEED stay near 71.2 and 65.3, then the projector is not the cause; if removing the ITDM loss alone reproduces the 'w/o align.' numbers, then the MSE term is not doing the claimed work.","supporting_citations":[{"cited_title":"Honeybee: Locality-enhanced projector for multimodal llm","cited_arxiv_id":null,"evidence_quote":"Honeybee supplies the C-Abstractor and D-Abstractor baselines at 144 tokens that MAGE is directly compared with in Table 1 and Table 4."}],"review_version":1}