{"id":"7b5ebe68-efc8-416d-a0f0-e58ef1d09226","arxiv_id":"2607.09029","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"MOSAIC uses multi-objective MIP search over linear/sparse/low-rank operators plus two-stage distillation to convert a homogeneous VLM into a hardware-aware heterogeneous model that matches teacher performance at 2.5× decoding speed.","lead":"MOSAIC turns a standard homogeneous vision-language model into a faster heterogeneous one by automatically choosing efficient attention and FFN variants for each layer under a hardware latency budget. The resulting 4B model matches the teacher’s accuracy on image, video and text benchmarks while delivering 1.76× prefilling and 2.54× decoding speedups at under 2 % of the original training cost.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified that overturns the central claim at the reported operating point.","rationale":"The paper's strongest claim is an engineering result at a concrete operating point, not a general optimality theorem. The isolated-block scoring assumption is acknowledged by the authors (they introduce global recovery precisely because BLD lacks inter-block compatibility) and is stress-tested by the progressive ablations: multi-objective search beats fixed ratios (Tab. 4), full search space beats ablated spaces (Tab. 5), and dual-teacher recovery is necessary (Tab. 7). Because the final model is released and the latency numbers are measured end-to-end in an extended vLLM, the empirical claim stands even if the MIP solution is only approximately optimal. The concrete test above would quantify how much of the gain is truly architectural versus distillation; a negative result would not invalidate the released model, only re-attribute credit. Hence the reader's ACCEPT / high-confidence verdict needs no adjustment.","tokens_in":24029,"tokens_out":493,"duration_ms":5311,"concrete_test":"Re-run the full two-stage recovery on one of the hand-designed 1:3 baselines from Table 4 (e.g., GQA:KDA) using identical data and hyper-parameters; if that recovered model closes the gap to MOSAIC-4B within 1 % overall, the MIP search itself contributes little beyond the recovery pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (predictivity of isolated block scores S^k_{l,i} from Eq. 2 for the joint heterogeneous stack under additive runtime Eq. 6) is real but does not undermine the paper's actual claim. The claim is empirical: after MIP search + two-stage recovery, MOSAIC-4B matches Qwen3-VL-4B-Instruct averages on 19 benchmarks at the measured 1.76\times/2.54\times speed-ups and <2 % training cost. Tables 1–3, Fig. 3, and the ablations (Tabs. 4–7) already show that the recovered model works at the chosen 1.5\times budget; the MIP is only a search heuristic, not a proof of global optimality. No internal contradiction or circularity appears, and the public model release supplies independent verification.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces MOSAIC, a hardware-aware method that converts a homogeneous VLM (Qwen3-VL-4B-Instruct) into a heterogeneous architecture by searching a per-layer space of linear (KDA/GDN), sparse (SWA), low-rank (MLA), and full (GQA) attention operators plus FFN width scalings. Selection is cast as a multi-objective MIP (Tchebycheff scalarization of KL, PPL, LLM and VLM scores subject to an additive profiled latency budget). After blockwise local distillation for scoring, a two-stage recovery (global off-policy KD from the 4B teacher, then dual-teacher on-policy distillation with a 235B oracle) yields MOSAIC-4B. Empirically, MOSAIC-4B matches the teacher’s average performance on 19 image/video/text benchmarks while delivering 1.76× prefilling and 2.54× decoding speed-ups at <2 % of the original training cost; systematic ablations of fixed-ratio baselines, search-space components, objectives and recovery stages are provided, and the model is released publicly.","tokens_in":24331,"tokens_out":1143,"duration_ms":22687,"significance":"If the empirical results hold under independent verification, the work is significant for efficient multimodal modeling. It is the first automated, hardware-constrained search over heterogeneous attention mechanisms for VLMs (rather than hand-crafted periodic interleaving or unimodal size-only NAS). The public MOSAIC-4B checkpoint, the open MIP formulation, the two-stage recovery recipe, and the extensive ablations (Tables 4–7, Figs. 3–4,7) supply a concrete, reproducible path from a strong homogeneous teacher to a faster student at modest extra cost. The discovered layer-wise patterns (SWA early, linear mid-network, MLA/GQA late) also offer transferable design intuitions. These contributions are of clear practical value for long-context and resource-constrained VLM deployment.","major_comments":[{"comment":"Section 3.2 (Eqs. 2, 4–6) and the BLD procedure: capability scores S^k_{l,i} are obtained by inserting a single candidate into an otherwise teacher residual stream. The MIP then treats these scores and the profiled runtimes as additive. While the final recovered MOSAIC-4B succeeds (Tables 1–3), the paper never directly measures how well the isolated scores rank full heterogeneous stacks after global recovery. Because the central efficiency claim rests on the MIP finding a near-optimal configuration under the latency budget, a short correlation study or a small set of random/full-stack re-evaluations of high- versus low-scoring architectures would strengthen (or bound) the search heuristic.","section":"Section 3.2, Eqs. (2),(4)–(6)"},{"comment":"Section 4.2 / Appendix B.4 and Fig. 3: all latency numbers (TTFT, TPOT, block profiles) are measured on a single platform (H800) with a custom-extended vLLM backend that the authors themselves implemented for the heterogeneous operators. The hardware-aware claim is therefore demonstrated only for this stack. Reporting at least one additional backend or device class (or releasing the profiling harness) would make the “strict hardware latency constraints” claim more robust and would let readers assess portability of the discovered architectures.","section":"Section 4.2, Appendix B.4, Fig. 3"}],"minor_comments":[{"comment":"Tables 1–3 report only point estimates; even a single-run standard deviation or bootstrap interval on the average Δ would help readers judge whether the −0.6 % / −0.8 % / −2.3 % gaps are stable.","section":"Tables 1–3"},{"comment":"Figure 4 and the architecture diagrams in Appendix C annotate FFN pruning with an asterisk, but the precise intermediate dimensions chosen by the MIP for each layer are never listed; a short table or supplementary file would aid reproducibility.","section":"Figure 4, Appendix C"},{"comment":"The combinatorial size is written “35 36” (missing exponentiation symbol) in Section 3.1; likewise a few minor typos (“w.r.t.”, “prefilling” vs. “prefill”) appear throughout.","section":"Section 3.1"},{"comment":"The dual-teacher stage uses a 235B oracle; while total training cost is claimed <2 %, a one-sentence breakdown of GPU-hours spent on oracle generation versus student updates would clarify the practical resource envelope for readers who lack a 235B teacher.","section":"Section 3.3, Appendix B.2"}],"recommendation":"accept","confidential_remarks":"The work is a solid systems/ML contribution that fits a top CV or ML systems venue. The public model release and the systematic ablations are genuine strengths. The two major points I raised are real but do not overturn the empirical claim; they can be addressed in a short revision or even left as acknowledged limitations. I see no novelty or citation-pattern concerns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is practical and clean. MOSAIC is the first method that treats attention mechanism type (GQA, KDA, GDN, MLA, SWA) plus FFN width as a searchable variable per layer, scores candidates with multi-capability metrics, and solves a multi-objective MIP under measured latency. The resulting MOSAIC-4B matches Qwen3-VL-4B-Instruct averages across 19 image/video/text benchmarks while posting 1.76\times prefilling and 2.54× decoding speed-ups at under 2% of the original training cost, and the weights are public.\n\nWhat is new is the combination: blockwise local distillation to warm-start a library of operators, relative capability scores (PPL, KL, LLM and VLM benches), Tchebycheff MIP under an additive runtime budget, and a two-stage recovery that first does global off-policy KD then dual-teacher on-policy distillation with a 235B oracle plus the original 4B teacher. The ablations are thorough—fixed-ratio hand designs lose to the searched assignment (Tab. 4), linear attention is essential for video (Tab. 5), each objective adds non-redundant signal (Tab. 6), and both recovery stages matter (Tab. 7). Latency curves on an extended vLLM backend and the consistent shallow-SWA / mid-linear / deep-MLA pattern across speedup targets are useful engineering evidence.\n\nThe soft spots are real but secondary. Isolated block scores (Eq. 2) are only a heuristic for the joint stack; the paper never claims global optimality, only that the recovered model works at the chosen 1.5\times budget. Runtime is additive and hardware-specific (H800 + custom vLLM). No error bars, and full search code is not released. None of these overturn the empirical claim at the reported operating point.\n\nThis is for people building real-time multimodal systems who need a concrete recipe rather than another hand-crafted hybrid. The math is standard MIP, the data are open, the citations are fair. I would send it to peer review; the contribution is clear and the evidence is sufficient for a serious referee.","headline":"Solid engineering paper: first automated MIP search over per-layer attention mechanisms for VLMs, delivers a public 4B model that matches its teacher at 2.5\times decode with <2% training cost.","tokens_in":24944,"tokens_out":559,"would_cite":true,"duration_ms":7043,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Automated multi-objective search converts a homogeneous vision-language model into a heterogeneous one that matches teacher accuracy while more than doubling decoding speed.","keywords":["vision-language models","heterogeneous architectures","neural architecture search","mixed integer programming","knowledge distillation","efficient attention","linear attention","hardware-aware search"],"falsifier":"Run the full MOSAIC pipeline on Qwen3-VL-4B-Instruct under the stated 1.5× latency budget; if the recovered model falls more than a few points behind the teacher’s average on the same 19 image/video/text benchmarks, or fails to realize the claimed TTFT/TPOT speedups in the same vLLM-style measurement, the central claim is false.","tokens_in":24960,"feed_emoji":"⚡","tokens_out":697,"duration_ms":9816,"temperature":0.7,"pith_summary":"Homogeneous vision-language models pay a quadratic cost for full attention and are hard to reshape for specific hardware. Prior hybrid designs mix efficient operators by hand with fixed ratios, which is suboptimal and not hardware-adaptive. MOSAIC treats every layer as a choice among linear, sparse, low-rank, and full attention plus FFN width options, scores candidates cheaply by local distillation, and solves a multi-objective mixed-integer program that maximizes language and vision capability under a measured latency budget. After the structure is chosen, a two-stage recovery—global off-policy distillation then dual-teacher on-policy distillation with a large oracle and the original teacher—restores performance. The resulting MOSAIC-4B matches its 4B teacher across image, video, and text benchmarks, delivers roughly 1.8× prefilling and 2.5× decoding speedups, and costs less than 2% of the original training budget. A sympathetic reader cares because the method turns expensive hand redesign into a constrained optimization problem that can be re-run for new latency targets.","feed_headline":"Search builds a VLM that matches accuracy at 2.5× decode speed","feed_subtitle":"MOSAIC-4B keeps teacher-level scores across 19 benchmarks while using under 2% of the original training cost.","key_machinery":"Multi-objective Mixed Integer Programming (MIP) over binary per-layer selections: each candidate is scored by relative capability (KL, PPL, LLM and VLM benchmarks) after blockwise local distillation, then the solver maximizes a Tchebycheff scalarization subject to an additive measured runtime budget.","core_discovery":"A homogeneous VLM can be automatically rewritten as an optimized heterogeneous stack by solving a multi-objective MIP over a unified per-layer search space of efficient attention and FFN variants; after two-stage dual-teacher distillation, the derived model matches the original teacher’s average performance while meeting strict hardware speedup targets at a tiny fraction of the original training cost.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["MIP search rewrites VLM into heterogeneous stack at 2.5× decode","MOSAIC finds efficient VLM layers matching teacher scores under 2% cost","Hardware-aware MIP turns homogeneous VLM into 1.76× prefill model","Adaptive composition yields VLM matching accuracy with 2.54× decode","Two-stage distillation recovers MOSAIC-4B after efficiency search"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Scores obtained by swapping one candidate block into an otherwise teacher-driven residual stream remain predictive of how a fully heterogeneous stack will perform after global recovery, so the MIP solution is near-optimal for the finished model.","fun_headline_variants_meta":{"raw":{"variants":["MIP search rewrites VLM into heterogeneous stack at 2.5× decode","MOSAIC finds efficient VLM layers matching teacher scores under 2% cost","Hardware-aware MIP turns homogeneous VLM into 1.76× prefill model","Adaptive composition yields VLM matching accuracy with 2.54× decode","Two-stage distillation recovers MOSAIC-4B after efficiency search"]},"model":"grok-4.5","effort":"low","cost_usd":0.00468,"raw_usage":{"total_tokens":1400,"prompt_tokens":834,"num_sources_used":0,"completion_tokens":84,"cost_in_usd_ticks":46800000,"prompt_tokens_details":{"text_tokens":834,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":482,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":834,"tokens_out":84,"duration_ms":4710,"temperature":1.0,"reasoning_tokens":482,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T00:52:20.354934+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the full MOSAIC pipeline on Qwen3-VL-4B-Instruct under the stated 1.5× latency budget; if the recovered model falls more than a few points behind the teacher’s average on the same 19 image/video/text benchmarks, or fails to realize the claimed TTFT/TPOT speedups in the same vLLM-style measurement, the central claim is false.","supporting_citations":[],"review_version":1}