{"id":"bae6b630-065b-4e42-9688-4232ad5db531","arxiv_id":"2607.08029","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"MoE structure, not parameter count, governs INT4 robustness in sub-3B VLMs; SigLIP INT8 latency spikes on Jetson Ampere are a BitsAndBytes-Ampere interaction, and INT4 VRAM savings come with TPOT and energy penalties.","lead":"Component-wise quantization of small vision-language models on Jetson Orin shows MoE backbones resist INT4 noise better than dense ones, while SigLIP INT8 incurs large latency from kernel-hardware mismatch. The work gives practical mixed-precision rules for edge multimodal deployment under tight VRAM and bandwidth.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified that would overturn the MoE-vs-dense claim under the paper's stated scope.","rationale":"The reader's strongest claim matches the paper's revised H1rev and is directly supported by the signed ΔMME values in Table 5 and the architecture labels in Table 4. The reader's weakest assumption (BitsAndBytes representativeness) is real for the latency/energy claims (H2/H3/H5) and is already partially stress-tested by the authors' own AWQ appendix, but it does not undercut the accuracy-direction claim that is the load-bearing result for H1rev. No internal contradiction, missing control that would reverse the MoE/dense split, or unacknowledged confound appears strong enough to move the workshop-level ACCEPT. The concrete test above is a low-cost extension that would further harden (or qualify) the paradigm claim without requiring a full re-implementation of the stack.","tokens_in":20936,"tokens_out":564,"duration_ms":5588,"concrete_test":"Re-run cfg1 (LLM INT4 only) on one additional dense sub-3B VLM of intermediate scale (e.g., a ~1.5–2B dense backbone such as a non-MoE Qwen2-VL or Phi-3-Vision variant) under the same VLMEvalKit/MME protocol on NX; if that dense model also degrades by tens of MME points while the existing MoE pair continues to hold or improve, the paradigm claim is reinforced; if it holds or improves, the MoE-vs-dense attribution weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (H1rev) rests on a clean 2\times2 contrast in Table 5: MoE backbones (Qwen3-VL 1.72B, DeepSeek-VL2 2.93B) gain MME under BitsAndBytes LLM INT4 while dense backbones (PaliGemma2 2.61B, LLaVA-OV 0.49B) lose substantially, falsifying pure scale dependence. Within the paper's own framing this contrast is internally consistent, the residual for the projector+LLM composite (cfg3) is near-additive, and the authors already flag the BitsAndBytes-centric limitation and supply an AWQ sanity check that removes the TPOT penalty without reversing the accuracy direction for Qwen3. The remaining soft spot is sample size (only two MoE and two dense families, one of which is ultra-small), but that is a scope limitation already visible in the tables rather than a hidden inconsistency that collapses the reported pattern.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"This paper presents a hypothesis-driven, component-wise quantization study of five sub-3B vision-language models (Qwen3-VL-2B, DeepSeek-VL2-Tiny, PaliGemma2-3B, LLaVA-OV-0.5B, Kosmos-2.5) on Jetson Orin NX and AGX. Six configurations isolate or combine INT4 on the LLM backbone and INT8 on the vision encoder or projector (Table 2). Using MME accuracy, peak VRAM, component latencies (including TPOT), energy, and an intelligence-per-joule (IPJ) metric, the authors revise five hypotheses: (H1rev) quantization sensitivity is governed by MoE vs. dense structure rather than scale alone (Table 5); (H2) SigLIP INT8 incurs a large, accuracy-independent latency penalty on Jetson Ampere via BitsAndBytes kernel fragmentation (Tables 6, 16–17); (H3) BitsAndBytes LLM INT4 saves VRAM but raises TPOT and often energy (Tables 7–8); (H4) projector+LLM errors are near-additive while vision+LLM residuals are architecture-dependent (Tables 9–10); (H5) accuracy rankings are platform-invariant while IPJ is bandwidth-sensitive (Tables 11–12). An AWQ sanity check (Appendix F) and isolated Nsight vision-path profiling (Appendix E) support the claims. Code is released.","tokens_in":21164,"tokens_out":1424,"duration_ms":25614,"significance":"If the results hold, the work supplies concrete, hardware-aware guidance for mixed-precision allocation of small VLMs on edge SoCs—an area where most prior quantization literature remains end-to-end and server-centric. The clean MoE-vs-dense contrast under LLM INT4, the documented SigLIP–BitsAndBytes–Ampere latency anomaly with kernel-level evidence, and the platform-invariant accuracy ranking versus platform-specific IPJ are immediately useful to practitioners. Strengths include explicit hypothesis revision when data contradict the original statements, multi-run averages (n≥3), component-isolated latency measurement (Algorithm 1), Nsight kernel breakdowns, an AWQ control that removes the TPOT penalty without reversing accuracy direction, and a public profiling toolkit. These elements raise the paper above a pure empirical dump and make the findings falsifiable and reusable within the stated BitsAndBytes/Jetson scope.","major_comments":[{"comment":"§4.1 / Table 5 (H1rev): The MoE-vs-dense claim is the paper’s strongest result, but the architectural sample is thin—only two MoE families (Qwen3, DeepSeek-VL2) and two dense families (Gemma-2, Qwen2-0.5B), one of which is ultra-small. The pattern is internally consistent and falsifies pure scale dependence, yet a single additional dense backbone near 1.5–2B (or a non-MoE sparse control) would substantially reduce the risk that the contrast is family-specific rather than paradigm-level. The authors already note the limitation; a short discussion of how far H1rev should be extrapolated is still needed for the claim to carry the weight given in the abstract and conclusion.","section":null},{"comment":"§4.3 / Appendix F (H3 and generalization): The TPOT and energy penalties under LLM INT4 are clearly tied to the BitsAndBytes dequantization path on Ampere; the AWQ W4A16 sanity check on Qwen3-VL removes the TPOT penalty while preserving accuracy. This is correctly flagged in Limitations, but the abstract and contribution list still present the VRAM–TPOT trade-off as a general property of “INT4 quantization of LLMs.” The manuscript should state more prominently (abstract or §5) that the latency/energy conclusions are backend- and platform-specific, so readers do not over-generalize beyond BitsAndBytes on Jetson Orin.","section":null},{"comment":"§4.4 / Tables 9–10 (H4): cfg3 residuals are convincingly near-additive (±4 points). cfg5 residuals, however, are large and opposite in sign for PaliGemma2 (−16.33) versus DeepSeek-VL2 (+8.03). The interpretation that modality-alignment pathways differ is plausible but remains post-hoc; without an intermediate diagnostic (e.g., cosine similarity of projected visual tokens or layer-wise activation error under joint quantization), the architecture-dependent non-additivity claim is under-supported relative to the strength with which it is stated.","section":null}],"minor_comments":[{"comment":"Table 1 and Appendix D: Perception/cognition breakdowns are valuable, yet the main text rarely discusses why cognition sometimes improves under INT4 (e.g., Qwen3, LLaVA-OV). A sentence or two linking this to possible regularization or noise effects would help.","section":null},{"comment":"Algorithm 1 and §3.5: TPOT excludes the first token (prefill). This is appropriate for generation-focused analysis, but the text should note that end-to-end latency (Table 15) still includes prefill, so readers do not conflate the two.","section":null},{"comment":"Table 4 / Kosmos-2.5: The model is a useful non-SigLIP control, but its absolute vision latency is an order of magnitude higher; a brief remark that relative (not absolute) ratios are the relevant comparison for H2 would prevent misreading.","section":null},{"comment":"IPJ definition (§4.5): The formula is clear, but the normalization (Score/2800) and the exact tegrastats sampling window should be restated once in the main text for self-containment.","section":null},{"comment":"Minor typos and consistency: “LLaV A” spacing, “cf gN” vs. “cfgN”, and occasional “BitsAndBytes” vs. “Bitsandbytes”. Unify notation for residual (cfg3−Exp vs. cfg5−Exp).","section":null},{"comment":"Figure 1 (Appendix C): Independent y-axis scales are fine, but a shared zero baseline or a relative-Δ panel would make cross-model magnitude easier to judge.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is a solid empirical systems contribution well suited to an ICML workshop or a specialized edge-AI / efficient-ML venue. The BitsAndBytes-centric scope and small architectural sample are real but already largely acknowledged; they do not invalidate the reported contrasts. I see no integrity or novelty-disclosure issues. Fit for a top general ML journal would be tighter because of the hardware-specific framing, but for the stated workshop/edge context the work is appropriate after the minor clarifications above."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful part of this paper is not another end-to-end BitsAndBytes run. It is the controlled separation of vision encoder, projector, and LLM on two real Jetson platforms, plus the willingness to rewrite H1 when the data kill pure scale dependence. Table 5 is the core: MoE backbones (Qwen3-VL, DeepSeek-VL2) hold or gain MME under LLM INT4 while dense ones (PaliGemma2, LLaVA-OV) drop hard, including a larger dense model. That falsifies the original H1 cleanly and supports the revised structural claim. The SigLIP INT8 latency spike is also real and well-isolated: isolated vision-path Nsight shows the quant/dequant + launch fragmentation, and the non-SigLIP Kosmos control stays near 1.2×. They further show INT4 VRAM savings without TPOT wins under BnB, near-additivity for projector+LLM, architecture-dependent residuals for vision+LLM, and platform-specific IPJ driven by bandwidth. Hypotheses get narrowed when evidence requires it; that is good practice.\n\nWhat is new is the edge-specific, component-resolved measurement set and the MoE-vs-dense contrast at this scale on NX/AGX, not a new quant algorithm. The AWQ sanity check is the right control: it removes the TPOT penalty without reversing the accuracy direction for Qwen3, so they already flag that the latency story is stack-dependent. Soft spots are real but proportional: only two MoE and two dense families (one ultra-small), MME-only, and BnB-centric kernels. Those are scope limits already visible in the tables and Limitations, not hidden contradictions. Citation pattern is standard and appropriate; no circularity; IPJ is just measured accuracy over measured energy. Code is released.\n\nThis is for people shipping multimodal models on bandwidth-constrained SoCs and for anyone who needs a reusable ablation template. It will not redefine quantization theory, but the measurements are solid enough that I would cite the MoE robustness and SigLIP-Ampere interaction notes. Send it to referees; workshop acceptance is the right bar.","headline":"Clean component-wise edge ablation that actually revises its hypotheses: MoE vs dense INT4 robustness at sub-3B is the real finding, with honest Nsight and AWQ controls.","tokens_in":21814,"tokens_out":539,"would_cite":true,"duration_ms":5920,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"For small vision-language models on edge chips, MoE structure—not parameter count—decides whether INT4 quantization preserves or wrecks accuracy.","keywords":["small VLM","component-wise quantization","edge deployment","MoE vs dense","Jetson Orin","INT4 TPOT","SigLIP latency","intelligence-per-joule"],"falsifier":"Repeat the identical MoE-versus-dense LLM INT4 ablations with a native hardware INT4 or AWQ path that removes the dequantization tax; if dense models then stop collapsing in accuracy or the token-speed penalty disappears while MoE gains vanish, the architecture-over-scale claim and the reported efficiency trade-offs would need revision.","tokens_in":21825,"feed_emoji":"📱","tokens_out":1072,"duration_ms":17305,"temperature":0.7,"pith_summary":"Small vision-language models (under about 3 billion parameters) are the practical route to multimodal AI on phones, robots, and other edge devices, but engineers still lack clear rules for which parts to quantize and how many bits to use. This paper isolates the vision encoder, the projector that aligns modalities, and the language-model backbone, then runs six controlled precision mixes on two Jetson Orin boards. The central finding is that architecture type, not mere size, governs INT4 sensitivity: mixture-of-experts backbones absorb the noise and can even gain accuracy, while dense backbones of similar or larger size drop sharply. Along the way the authors map hardware-specific traps—SigLIP vision encoders that balloon in latency under INT8 on Ampere kernels, INT4 that saves VRAM yet slows tokens because of dequantization, mostly additive composite errors except on the vision-to-language path, and energy efficiency that flips with memory bandwidth. The practical payoff is a set of modality- and platform-aware precision rules rather than one-size-fits-all compression.","feed_headline":"MoE, not size, rules INT4 accuracy for small edge VLMs","feed_subtitle":"Component ablations on Jetson show architecture and memory bandwidth decide what survives quantization","key_machinery":"A component-wise ablation framework that freezes two of the three VLM parts (vision encoder, projector, LLM backbone) at full precision while quantizing the third, plus two joint configurations, run on identical MME prompts across Jetson Orin NX and AGX so marginal accuracy, VRAM, latency, and intelligence-per-joule effects can be attributed to each piece.","core_discovery":"Quantization sensitivity of sub-3B vision-language models is governed by the structural paradigm of the language backbone—mixture-of-experts versus dense—rather than by scale alone. Under LLM INT4, MoE models improve or hold multimodal accuracy while dense models of comparable or larger size degrade substantially; parameter count acts only as an aggravating factor inside a homogeneous architecture family.","pith_inferences":["If MoE sparsity truly localizes INT4 noise, the same resilience should appear in other sparse multimodal designs (not only the two MoE models tested here).","Hardware vendors and runtime authors who supply native low-bit vision kernels for SigLIP-like transformers would remove a deployment-specific tax that pure model compression cannot fix.","Automatic mixed-precision search that includes both component identity and target SoC bandwidth could turn the paper’s manual guidelines into a deploy-time policy.","The VRAM-versus-TPOT paradox implies that future edge VLMs may need separate “memory mode” and “latency mode” quantization profiles rather than a single bit-width setting."],"forward_implications":["Prefer MoE language backbones when INT4 is required for VRAM-limited edge VLMs; dense ultra-small models are far more fragile.","Treat BitsAndBytes LLM INT4 as a pure memory-saving tool, not a latency win, on current Jetson Ampere.","Avoid vision INT8 for SigLIP-style encoders on these platforms unless a different kernel path is available; the latency cost can exceed 4× with little accuracy change.","Composite projector+LLM quantization can be budgeted additively; vision+LLM mixes need architecture-specific checks.","Accuracy rankings transfer across NX and AGX, but intelligence-per-joule does not—energy planning must be platform-specific."],"fun_headline_variants":["MoE not size rules INT4 accuracy in sub-3B edge VLMs","Dense backbones degrade under INT4 while MoE hold steady","Structural paradigm not scale decides VLM quantization sensitivity","MoE backbones mitigate INT4 noise where dense models fail","Architecture family not parameter count governs edge VLM INT4"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The measured speed, energy, and latency penalties are taken as representative of practical edge quantization, even though they come mainly from one software stack (BitsAndBytes) on Jetson Ampere kernels.","fun_headline_variants_meta":{"raw":{"variants":["MoE not size rules INT4 accuracy in sub-3B edge VLMs","Dense backbones degrade under INT4 while MoE hold steady","Structural paradigm not scale decides VLM quantization sensitivity","MoE backbones mitigate INT4 noise where dense models fail","Architecture family not parameter count governs edge VLM INT4"]},"model":"grok-4.5","effort":"low","cost_usd":0.006524,"raw_usage":{"total_tokens":1628,"prompt_tokens":763,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":65240000,"prompt_tokens_details":{"text_tokens":763,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":792,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":763,"tokens_out":73,"duration_ms":7010,"temperature":1.0,"reasoning_tokens":792,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T13:25:30.669440+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Repeat the identical MoE-versus-dense LLM INT4 ablations with a native hardware INT4 or AWQ path that removes the dequantization tax; if dense models then stop collapsing in accuracy or the token-speed penalty disappears while MoE gains vanish, the architecture-over-scale claim and the reported efficiency trade-offs would need revision.","supporting_citations":[],"review_version":1}