{"id":"e855d3e8-e3dc-4628-ae53-8550d1bf47db","arxiv_id":"2505.22664","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training a vision encoder against a small surrogate made from a target LLM's early layers lets the encoder be grafted into the full LLM with no fine-tuning, matching some full-training results at roughly half the cost.","lead":"The authors build small surrogate language models that copy the early layers of a big LLM like Llama-70B, train a vision encoder against the surrogate, and then plug that encoder straight into the big LLM without extra training. This zero-shot grafting matches full decoder training on some benchmarks and cuts the reported training cost for Llama-70B VLMs by about 45%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ~45% cost-reduction claim depends on unstated construction of the 10-30% data subsets in Table 8 and on single runs; if those subsets are not random/stratified, '10% data reaches baseline 100%' could be an ordering artifact.","rationale":"The paper presents a genuine method with controlled ablations (Tables 2 and 4), cross-model generalization (Section A.6), and a code release, so it is not rejectable on its face and should not be. The central cost claim is load-bearing because the abstract's '~45%' is the practical benefit advertised. That claim rests on single-run comparisons over unspecified data subsets, and the non-monotonic baseline jump in Table 8 is a concrete red flag: without knowing how the 10-30% slices were drawn, the finding that ours@10% matches baseline@100% could be a data-ordering artifact. This is exactly the reader's weakest assumption, and our stress-test confirms it as the most load-bearing concern. The zero-shot grafting phenomenon remains supported by independent ablation evidence, so the conditional verdict stays appropriate; no verdict change is needed, but the quantitative cost-reduction headline should be softened or verified with repeated stratified sampling.","tokens_in":22286,"tokens_out":13056,"duration_ms":158332,"concrete_test":"Re-run the Llama-70B experiments in Table 8 with at least 3 independent stratified random 10%/20%/30% subsets of LLaVA-1.5-665K, stratified by source/task and using the same seeds for baseline and ours; report mean +/- std of the average score and decoder-training hours. If ours@10% does not consistently match or exceed baseline@100% within roughly 0.5 std, the ~45% cost claim should be stated as preliminary rather than as a headline result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest headline claim is the ~45% cost reduction. Table 9 reports 34.79h baseline vs 19.17h ours (44.9% saving), but this saving is only valid if the 20%-data decoder run in Table 8 is representative. The paper never states how the X% subsets are sampled from LLaVA-1.5-665K. The baseline average score jumps non-monotonically (10%: 0.5127, 20%: 0.5153, 30%: 0.6277, 100%: 0.6538), and ours also fluctuates (10%: 0.6612, 20%: 0.6701, 30%: 0.6704), consistent with high variance or ordering effects rather than a smooth learning curve. If the 20% slice happened to be easier (e.g., a prefix or an unbalanced slice), the 5.56h decoder-training figure and the associated 45% saving would not be method effects. The zero-shot grafting mechanism itself has supporting controlled ablations, so this concern targets the quantitative headline rather than the core mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for training vision encoders for VLMs without repeatedly training against a large target LLM. It first identifies a 'transition point' in the layer-wise prediction trajectory of an LLM, then constructs a small 'surrogate' decoder by preserving the early layers before that point and replacing later layers with a single translator layer initialized from the preserved block. A vision encoder is fine-tuned on this surrogate decoder, and the resulting encoder is then either plugged directly into the full-size target LLM ('zero-shot grafting') or used as a warm start for full decoder training. Experiments on Llama-3B, Llama-8B, Llama-70B, and Qwen3-4B show that grafting often improves on the surrogate's own VLM performance, sometimes matches or exceeds full decoder training on individual benchmarks, and that the warm-start approach reduces total Llama-70B training time by approximately 45%.","tokens_in":22541,"tokens_out":5461,"duration_ms":68766,"significance":"If the claims hold, the paper offers a practical and conceptually interesting recipe for decoupling vision-encoder training from the cost of a giant LLM decoder. The strongest support comes from controlled ablations: the T(16,26)* experiment in Section 1.2 shows that preserving the early-phase parameters of the target is necessary for zero-shot grafting, and the Llama-8B ablations T(6,30), T(12,30), and T(17,30) in Tables 3-4 demonstrate a monotonic relationship between early-layer preservation and grafting quality. The evaluation spans multiple model families and scales, includes external VLM benchmarks, and the code is released. The main caveats are experimental: all numbers come from single runs, the construction of the X% training subsets in Table 8 is not stated, and the MME/POPE prompt protocol for zero-shot grafting is not fully specified. These issues directly affect the quantitative headline claim of ~45% cost reduction.","major_comments":[{"comment":"The headline ~45% cost reduction rests on the decoder-training time at 20% data in Table 9 (5.56 h) and on the claim that 10% of surrogate-trained data already matches 100% baseline performance. However, the paper does not state how the 10%, 20%, 30%, and 60% subsets of LLaVA-1.5-665K were sampled, and all numbers are single-run with no error bars. The baseline curve is non-monotonic (average score 0.5127 at 10%, 0.5153 at 20%, 0.6277 at 30%, 0.6538 at 100%), so an unrepresentative 20% slice (for example, a prefix or an unbalanced split) could explain the '20% of data suffices' result. Please specify the subsetting procedure, report repeated runs or variance, and, if possible, use a random stratified sampling scheme.","section":"Section 2.2, Tables 8-9"},{"comment":"The evaluation protocol for MME and POPE appears to differ between zero-shot grafting and full decoder training. The text states that for grafting, because the target LLM has not been instruction-tuned on VLM data, 'we prompt the model with binary prompts' (Section 1.2), while the full decoder-training baseline is presumably evaluated under the standard LLaVA 'single word or phrase' protocol. The exact prompt template used for each row is not given, so it is unclear whether the comparisons labeled 'MMEbinary' and 'POPEbinary' are apples-to-apples across all rows. Since a binary yes/no prompt can change scores substantially, the claim that zero-shot grafting 'surpasses the full-size Llama-70B decoder training on some benchmarks' requires the same prompt protocol for every method compared. Please specify the prompt used for each row, and ideally report both protocols for both methods.","section":"Section 1.2 and Section 2.1.1, Tables 2, 4, and 7"},{"comment":"All reported VLM scores are from single runs without error bars or significance measures. Some differences that support the central narrative, such as the grafted encoder outperforming the surrogate decoder on LLaVA-Wild or MMBench, may be within run-to-run variability of VLM evaluation. It would strengthen the paper to report at least two or three seeds for the main Llama-70B comparisons in Tables 7-8, or to provide evaluation with multiple prompt variants, before claiming performance parity or superiority.","section":"Tables 2, 4, 6, and 7"}],"minor_comments":[{"comment":"The word 'cabability' is a typo and should be 'capability'.","section":"Figure 5 caption"},{"comment":"The dynamic loss weight normalization in Section A.11 appears notational: the update w_i ← w_i / ((Σ_j w_j)/(Σ_j L_j)) is dimensionally inconsistent. Please clarify whether the intent is to rescale the average weight to 1, and rewrite the equation accordingly.","section":"Section A.11"},{"comment":"The transition point is identified visually from KL-trajectory plots (Figure 3). A quantitative criterion, such as a threshold on the KL distance or a change-point detector, would make the method easier to reproduce for a new target LLM.","section":"Section 1.1"},{"comment":"The column headers in Tables 7 and 8 are extremely compact (e.g., 'MMEbinary MME POPE binary POPE'), and the reader cannot tell at a glance which columns correspond to which benchmark subset and protocol. Expanding the headers and adding a short caption explaining the binary vs. original protocol would improve readability.","section":"Table 7 and Table 8"}],"recommendation":"major_revision","confidential_remarks":"The core mechanism is interesting and the controlled ablations are convincing evidence for the early-phase preservation hypothesis. The main risk is the quantitative cost-saving claim in Table 8, which depends on the unstated construction of the data subsets and single-run evaluations. If the authors can provide the subset sampling details, repeated runs, and exact prompt protocols, the paper could become acceptable. I would also encourage the editor to ask for the evaluation scripts so that the MME/POPE protocol ambiguity can be resolved at review time."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a real method with a clear mechanism, and the central claim about zero-shot grafting survives a fairly careful reading. The headline ~45% cost reduction is more fragile and needs another pass before I'd trust the number.\n\nWhat's new: building a small surrogate by inheriting the target LLM's shallow layers, inserting a translator, training the vision encoder on that, then grafting directly into the full-size model. The layer-trajectory analysis with the transition point is a clean way to decide where to cut, and the control experiment T(16,26)* — unfreezing early layers to show grafting breaks — is exactly the kind of ablation that earns trust. The results across 3B/8B/70B Llama plus Qwen/Gemma family plots in the appendix are more evidence than most papers of this type ship.\n\nThe soft spots are real but localized. First, Table 8's X% training-data subsets: the paper never says how the 10/20/30% slices are drawn from LLaVA-1.5-665K. If they're a prefix or an unbalanced slice, the \"10% beats baseline 100%\" claim becomes an ordering artifact. The non-monotonic baseline curve makes this more than pedantry. That's the load-bearing pillar under the 45% number. Second, all comparisons are single-run with no error bars or seeds; for a training-cost claim that's a bigger issue than for a mechanism demonstration. Third, the zero-shot grafting evaluation changes the MME/POPE prompt protocol to binary yes/no, which is defensible but should be stated louder and checked against the original protocol. None of these sink the core idea — the grafting mechanism is independently supported by the layer ablations — but they cap how much of the quantitative headline a careful reader should repeat.\n\nThe citation pattern looks fair; the related work discussion (Tuned Lens, layer pruning, InternVL) is honest about overlap and difference. The appendix's own admission that surrogate training is less useful for small models is a sign of balanced reporting.\n\nWho this is for: anyone training large VLMs or working on efficient encoder training. It deserves a serious referee. I'd send it to review and ask for one revision round focused on the data-subsampling methodology and repeated-run numbers; the mechanism itself can stand.\n\nYes, worth a reading group slot too.","headline":"Solid method with a genuine grafting mechanism; the ~45% cost savings headline needs data-subsampling and repeated-run details before I'd quote it.","tokens_in":23097,"tokens_out":1736,"would_cite":true,"duration_ms":19362,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision encoders trained on small LLM surrogates can be grafted directly into the full-size model, matching full decoder training on several benchmarks and cutting Llama-70B VLM training cost by roughly 45%.","keywords":["vision-language models","vision encoder","surrogate model","zero-shot grafting","layer transition analysis","training cost reduction","Llama-70B"],"falsifier":"Re-run the decoder-training comparison with the 10% and 20% subsets drawn as random stratified samples of all 665K vision-language instructions, repeated over several seeds, and report the full benchmark table; if the surrogate-trained encoder no longer matches the 100%-data baseline at 10% data, the convergence and ~45% cost-reduction claims are artifacts of how the subsets were chosen.","tokens_in":22086,"feed_emoji":"🖼️","tokens_out":8646,"duration_ms":91106,"temperature":0.7,"pith_summary":"Training a vision-language model normally means fine-tuning a large language decoder, and with decoders the size of Llama-70B that step dominates the compute bill. This paper argues the decoder does not need to sit in the loop for most of encoder training: you can first train the vision encoder against a small 'surrogate' language model built from the target LLM's own shallow layers, then plug the finished encoder straight into the full-size model. The authors call this zero-shot grafting and report that the grafted pair surpasses the encoder-with-surrogate pair, matches full decoder training on several benchmarks, and cuts overall VLM training cost by about 45% when the target is Llama-70B. If true, the result separates vision-encoder alignment from expensive decoder-scale training and makes VLM development substantially cheaper.","feed_headline":"Surrogate-trained encoders cut 70B VLM training cost ~45%","feed_subtitle":"The encoder plugs into Llama-70B and matches full decoder training on several benchmarks.","key_machinery":"The load-bearing object is the surrogate model: a small transformer that keeps the target LLM's early-phase layers unchanged, keeps its last layer, and inserts a single 'translator' transformer layer at the transition point where the late phase begins. The translator is initialized from the shallowest replaced layer and fine-tuned on text-only instructions, so the surrogate speaks the same embedding language as the target. The transition point itself is located by computing, for each layer, the Kullback-Leibler divergence between the layer's softmax next-token distribution and the final distribution, and finding where 300 random text samples abruptly converge; the paper verifies with layer-replacement ablations and a control that perturbs early layers, showing that preserving the inherited early parameters is what makes zero-shot grafting work.","core_discovery":"The central discovery is that a vision encoder can be aligned to a much larger LLM by training it against a small stand-in that shares that LLM's early layers. Tracing the Kullback-Leibler divergence between each layer's next-token distribution and the model's final output reveals a sharp transition point: layers before it form an early phase that fixes the model's representation space and carries the ability to transfer an encoder, while layers after it mostly settle on a concrete prediction. Replacing those late layers with a single translator layer yields a surrogate that keeps the target's embedding space intact. An encoder trained on this surrogate, when grafted into the target LLM with no further training, produces image features the target can read directly; on several benchmarks the grafted model exceeds the encoder-surrogate pair and approaches or matches full decoder fine-tuning. The same encoder also serves as a warm start, letting the full decoder reach baseline-level performance with roughly ten to twenty percent of the training data and cutting total training hours from about 35 to 19 for a Llama-70B VLM.","pith_inferences":["A natural extension the paper does not pursue is applying the same surrogate construction to align encoders for other modalities, such as audio or depth, whenever a large autoregressive decoder is the training bottleneck.","The saturation seen at 20% data suggests the alignment signal is concentrated and that better data selection or curricula could shrink the required fraction below the paper's 10-20% range.","If the transition point can be predicted from model size or architecture, surrogate construction could be automated without recomputing the layer-trajectory analysis for each new LLM."],"forward_implications":["A newly released LLM can be turned into a vision-language model by building a surrogate from its early layers and training only a vision encoder, avoiding full decoder fine-tuning.","The surrogate-trained encoder can be grafted into the target LLM without any further training, giving an immediate VLM that on several benchmarks matches a model trained end-to-end with the full decoder.","If further decoder fine-tuning is desired, the grafted encoder acts as a warm start and reaches baseline-level performance after training on only about 10-20% of the instruction set.","The approach appears to transfer across model families, since the layer-trajectory transition point is observed in Llama, Gemma, and Qwen models of different sizes.","Full decoder fine-tuning on top of the surrogate-trained encoder preserves the LLM's text-benchmark performance, whereas baseline VLM training degrades it."],"supporting_citations":[{"why":"Supplies the 665K vision-language instruction set used for encoder, translator, and decoder training in all stages.","marker":"[27]"},{"why":"Provides the 500K text-only instructions used to fine-tune the translator and the random samples used for the layer-trajectory analysis.","marker":"[5]"},{"why":"Defines the Llama family whose early layers are inherited by the surrogates and whose 70B model is the main grafting target.","marker":"[9]"},{"why":"Provides the CLIP vision encoder that is fine-tuned on the surrogate and then grafted into the target LLM.","marker":"[37]"},{"why":"Source of the translator idea and tuned-lens probing that informs the layer-replacement construction.","marker":"[3]"},{"why":"The LoRA baseline compared against, showing that parameter-efficient tuning does not accelerate convergence the way surrogate-trained encoders do.","marker":"[16]"}],"fun_headline_variants":["Surrogate-trained encoder cuts 70B VLM cost 45%, matches full training","Zero-shot encoder grafting into 70B LLM from surrogate, cost down 45%","Train encoder with LLM's early layers, graft into 70B, save 45%","Reuse LLM's early layers to train encoder, graft into 70B, cut cost 45%","Small surrogate trains encoder that grafts into 70B LLM with no fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cost and convergence claims assume that the 10% and 20% slices of the instruction set used for decoder fine-tuning are representative of the full 665K-example set; if those slices are an easier or unbalanced portion, the apparent speed-up could come from data ordering rather than from the surrogate-trained encoder.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate-trained encoder cuts 70B VLM cost 45%, matches full training","Zero-shot encoder grafting into 70B LLM from surrogate, cost down 45%","Train encoder with LLM's early layers, graft into 70B, save 45%","Reuse LLM's early layers to train encoder, graft into 70B, cut cost 45%","Small surrogate trains encoder that grafts into 70B LLM with no fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001646,"raw_usage":{"total_tokens":6537,"prompt_tokens":942,"completion_tokens":5595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":5477}},"tokens_in":558,"tokens_out":5595,"duration_ms":40524,"temperature":1.0,"reasoning_tokens":5477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:02:57.338711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the decoder-training comparison with the 10% and 20% subsets drawn as random stratified samples of all 665K vision-language instructions, repeated over several seeds, and report the full benchmark table; if the surrogate-trained encoder no longer matches the 100%-data baseline at 10% data, the convergence and ~45% cost-reduction claims are artifacts of how the subsets were chosen.","supporting_citations":[{"cited_title":"Improved Baselines with Visual Instruction Tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the 665K vision-language instruction set used for encoder, translator, and decoder training in all stages."},{"cited_title":"Learning Transferable Visual Models From Natural Language Super- vision","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP vision encoder that is fine-tuned on the surrogate and then grafted into the target LLM."},{"cited_title":"LoRA: Low-Rank Adaptation of Large Language Models","cited_arxiv_id":null,"evidence_quote":"The LoRA baseline compared against, showing that parameter-efficient tuning does not accelerate convergence the way surrogate-trained encoders do."}],"review_version":1}