{"id":"17234c1c-183b-4bad-9e31-7e07d44b5388","arxiv_id":"2607.16305","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"LookME extends per-layer embedding lookup from text to continuous image and cross-modal embeddings and reports improved VLM scores on most benchmarks.","lead":"LookME adds large external lookup tables stored in ROM to a vision-language model, letting it retrieve 'memory' embeddings for image and cross-modal tokens rather than text only. The authors report gains of 1–4 points over a fine-tuned Qwen2.5-VL-3B baseline on most of 11 visual benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparative claim rests on an unverifiable fair-comparison assumption: training recipe, data mixture, and per-method budgets are deferred to a missing supplement, so LookME's gains over text-only PLE could reflect tuning or data rather than the lookup mechanism.","rationale":"LookME's architecture is internally coherent: the hierarchical two-level table factorization (Eq. 4–9), SKA gating (Eq. 10–11), and NetProp (Eq. 12) are specified with equations, and the ablations show monotonic gains from adding modalities and from the default 512D/[2/10/18] configuration. That internal consistency does not, however, establish the central comparative claim. The claim is only meaningful if data, tuning effort, and compute budgets are matched across methods. The manuscript asserts this but defers the substantive details to a missing supplementary and reports no variance. This is the weakest link because if the data mixture or hyperparameters favor LookME, every benchmark delta in Table 1 is confounded, and the method's value as a lookup mechanism is not demonstrated. This is more load-bearing than the BLINK inconsistency, which is a fixable wording/table error, whereas an uncontrolled comparison undermines the entire evaluation. A replication with released code, exact data mixture, matched budgets, and multiple seeds would settle it. The reader's CONDITIONAL verdict already captures this, so no change is needed.","tokens_in":14033,"tokens_out":5197,"duration_ms":55200,"concrete_test":"Obtain the missing supplementary material and run a controlled replication: re-train Engram and MeKi on the exact same 60M-sample stream and data-sampling ratios as LookME, with the same optimizer schedule, token budget, and a matched total external-table parameter budget (using LookME's 0.82B external parameters), across at least 3 seeds, reporting per-benchmark means and variances. If LookME no longer wins a majority of the 11 benchmarks under these matched conditions, the central empirical claim is not established; if the gains persist, the fair-comparison concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that multimodal lookup 'consistently outperforms baselines and text-only PLE methods across diverse visual benchmarks'—depends on the assumption that all methods were compared under matched training conditions. The Experimental Setup states that all models are 'pre-trained on the same unified dataset' and that Engram/MeKi use 'the same hyperparameter settings as LookME,' but the corpus composition, sampling ratios, and hyperparameters are deferred to a supplementary that is not present. No repeated-seed variance or error bars are reported. The only budget information is the external-parameter column in Table 1 (Engram 2.35B, MeKi 0.26B, LookME 0.82B), which does not specify per-pathway table sizes, ROM storage, latency, or active-parameter overhead. If the 60M-sample mixture was curated for the evaluated tasks, or if LookME's full-parameter fine-tuning benefited from more effective tuning than the baselines, the reported +0.3–3.9 point gains could reflect the training recipe rather than the lookup mechanism. The direct BLINK inconsistency (Table 1: MeKi 49.0 vs LookME 48.7) further shows that the 'consistently best' phrasing overstates the evidence, but the deeper risk is that the empirical advantage itself is not yet established. This is addressable by releasing artifacts; it does not by itself invalidate the architectural idea, so a conditional verdict is appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LookME, a framework that extends per-layer embedding (PLE) lookup from text-only operation to multimodal embeddings in vision-language models. It proposes a hierarchical two-level lookup: continuous image and cross-modal embeddings are first routed to a coarse 'scene' and then softly combined over 'scene-primitives', while text tokens retain ID-based lookup. A sparse injection mechanism—comprising an attention-contribution-based gate (SKA) and an inter-layer propagation path (NetProp)—selects which tokens receive external embeddings and allows non-LookME layers to reuse retrieved embeddings. Experiments on 11 benchmarks compare LookME with Qwen2.5-VL, a CPT baseline, and text-only PLE methods Engram and MeKi. The central claim is that LookME 'consistently outperforms baselines and text-only PLE methods across diverse visual benchmarks.'","tokens_in":14534,"tokens_out":7756,"duration_ms":79178,"significance":"The architectural idea is genuinely novel and timely: extending PLE-style lookup to continuous multimodal embeddings, with a scene-to-primitive factorization that avoids flat-table routing imbalance without increasing table size, is a plausible and useful direction for edge deployment of VLMs. The paper also provides useful ablations of lookup pathways, embedding dimension, injection layers, and NetProp range. If the empirical results survive matched-training scrutiny, the contribution would be publishable and practically relevant. However, the current manuscript does not yet establish that empirical claim: the fair-comparison setup is not verifiable from the text, the promised SKA ablation is absent, and a headline claim about multi-image gains is contradicted by the paper's own table.","major_comments":[{"comment":"The fair-comparison assumption is unverifiable as stated. The paper says all models are pre-trained on the same 60M-sample unified dataset and that Engram/MeKi use 'the same hyperparameter settings as LookME', but the dataset composition, sampling ratios, training hyperparameters, and per-pathway table sizes are deferred to a supplementary that is not present in the submission. No error bars or repeated-seed results are reported. Therefore the reported +0.3–3.9 point gains over the strongest baseline could reflect data curation or tuning rather than the lookup mechanism. Please include the full training configuration for every method, data mixture weights, compute budgets, and at least three seeds per benchmark (or error bars), and specify table sizes and latency for each method.","section":"Experimental Setup"},{"comment":"The text repeatedly claims that LookME 'consistently achieves the best performance across all six task categories' and shows its 'largest gains' on multi-image evidence aggregation. Table 1 contradicts this: on BLINK, MeKi scores 49.0 while LookME scores 48.7, making LookME second-best in the Multi-Image category. Moreover, LookME's BLINK gain over the Qwen2.5-VL backbone is +0.1 (48.6→48.7), far smaller than the +2.8–3.9 gains on CV-Bench-3D and HRBench. Please correct these overstatements and rephrase the conclusions to match the data.","section":"Main Results / Table 1"},{"comment":"The section states that 'we conduct separate ablations for each component' of sparse injection (SKA and NetProp), but only NetProp results are reported (Table 3). No ablation isolates the SKA intra-layer gate (e.g., gate vs. no gate; random gate vs. attention-contribution gate). Since SKA is a core contribution, the claim that sparse injection improves the efficiency/performance trade-off is not fully supported. Please add the missing SKA ablation and report results.","section":"Effectiveness of Sparse Injection"},{"comment":"The SKA gate is described as a 'per-token, per-pathway binary gate' with 'distinct gate projections for multimodal pathways,' but Eq. (10) defines a single scalar gate g_i = σ(W_g Δ_i) with one W_g ∈ R^{d_h×1}. There is no pathway index or separate projection. Please clarify: if the gate is pathway-specific, update the equation; if it is shared, revise the text. This is important for reproducing the sparse injection behavior.","section":"Eq. (10), Intra-Layer Sparse Activation"}],"minor_comments":[{"comment":"The notation 'argTop \\tilde{T}' is not defined. It should be stated explicitly that this returns the indices of the top-\\tilde{T} relevance scores, and the subsequent use of e_{S,p} should be clarified as the table embedding at those indices.","section":"Eq. (7)"},{"comment":"The symbol R_img is overloaded: it denotes both the per-scale retrieved embeddings (R_img,s) and the fused output on the left-hand side. Rename the per-scale outputs, e.g., to \\tilde{R}_{img,s}, to avoid confusion.","section":"Eq. (9)"},{"comment":"The update R*_{(k)} = R*_{(k-1)} W_k^p + R*_{(k-1)} uses the same symbol on the right-hand side in two roles; please clarify whether the residual connection is intended and define W_k^p (dimensions, initialization, and whether it is a bottleneck).","section":"Eq. (12)"},{"comment":"Typo: 'controled' should be 'controlled'.","section":"Section 'Intra-Layer Sparse Activation'"},{"comment":"The 'LLM Params (Backbone / External)' column does not specify whether 'External' counts table parameters, ROM storage, or active parameters. Please provide per-pathway table sizes and a breakdown of storage vs. active overhead for Engram, MeKi, and LookME.","section":"Table 1"},{"comment":"The limitation section is a single sentence about continual pre-training. Given the systems-oriented claims, the paper should report concrete latency, ROM, and memory measurements, and discuss failure cases where the lookup degrades performance (e.g., the BLINK result).","section":"Limitation"}],"recommendation":"major_revision","confidential_remarks":"The novelty is real and the method is plausible, but the paper currently falls short on reproducibility: the supplementary is missing, no error bars are reported, and the SKA ablation is absent. These are fixable within the manuscript's scope, and I do not see an inherent flaw in the architecture. If the authors supply the missing details and correct the overstatements, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real architectural step forward, and the paper mostly knows what it is doing; the evidence section is not yet strong enough to support the “consistently outperforms” language.\n\nWhat’s new: LookME extends PLE-style lookup beyond discrete token IDs to continuous image and cross-modal embeddings. The two-level scene-to-primitive retrieval is a sensible response to the routing-imbalance problem that a flat lookup table would face, and the sparse injection (SKA gate plus NetProp) gives a coherent way to control cost. The ablation tables support the core mechanism: adding the image pathway helps, adding the hidden-state pathway helps, and the head-tail propagation choice is reasonable. The PCA initialization from the backbone embedding matrix carries a self-referential flavor, but because the tables are trained and injected through learned pathways, I don’t see a circularity problem.\n\nSoft spots, in proportion: first, the abstract and conclusion say LookME “consistently” outperforms baselines, but Table 1 shows MeKi beating LookME on BLINK (49.0 vs 48.7), and the text specifically claims the largest gains on multi-image evidence aggregation. That is a direct internal inconsistency and should be fixed. Second, the fair-comparison assumption cannot currently be checked: the experimental setup says all models are pretrained on the same 60M-sample dataset and that baselines use the same hyperparameters, but the corpus composition, sampling ratios, and training details are deferred to a supplementary that is not present. Per-method table sizes, ROM storage, latency, and active-parameter overhead are also missing. Third, there are no error bars or repeated-seed results, and the headline configuration appears to be selected on the evaluation benchmarks. These issues are addressable rather than fatal, but they mean the quantitative claim is conditional.\n\nThe limitation section only mentions from-scratch training and does not acknowledge the missing artifacts or the fair-comparison gap. I’d want that acknowledged.\n\nWho this is for: anyone working on parameter-efficient VLM scaling or PLE-style memory. The architectural idea deserves serious referee time, but only if the authors release code, data recipes, table budgets, and variance numbers. As it stands, I would not trust the precise gains yet.\n\nRecommendation: send it to peer review with a clear request for major revision on the experimental reporting, rather than desk-reject.","headline":"A genuinely new extension of PLE to continuous visual and cross-modal tokens, but the empirical case is undermined by a missing supplement, no variance reporting, and a direct contradiction in the multi-image benchmark.","tokens_in":14948,"tokens_out":1922,"would_cite":true,"duration_ms":24086,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LookME is the first framework to apply lookup-based external-embedding enhancement to multimodal (image and cross-modal) tokens in vision-language models, and it reports consistent gains over text-only PLE methods across 11 visual benchmark","keywords":["vision-language models","per-layer embedding (PLE)","multimodal embedding lookup","hierarchical two-level retrieval","sparse injection","ROM storage","edge deployment","visual token enhancement"],"falsifier":"Retrain the strongest text-only PLE baseline and LookME under strictly identical conditions—same 60M-sample corpus with disclosed sampling ratios, same total table parameter budget, same optimizer and compute—then compare on the 11 benchmarks; if the text-only baseline matches or beats LookME, the claim that multimodal lookup drives the improvement is falsified.","tokens_in":13940,"feed_emoji":"🔍","tokens_out":7032,"duration_ms":61721,"temperature":0.7,"pith_summary":"LookME sets out to show that per-layer embedding (PLE) scaling—storing large external knowledge tables in ROM and doing lightweight lookups—can work for the visual token stream of a vision-language model, not just for text tokens. The paper introduces a hierarchical two-level lookup that first routes continuous image and cross-modal embeddings to coarse 'scenes', then softly combines the most relevant 'scene-primitives', plus a sparse injection scheme that gates which tokens get enhanced and propagates retrieved embeddings across layers. If the claim holds, VLM capacity could be scaled on edge devices by offloading knowledge to ROM and retrieving only relevant entries on demand, avoiding both the memory cost of dense models and the latency of loading experts. The empirical case is made on 11 visual benchmarks, where LookME consistently outperforms the base model and text-only PLE baselines, with the largest gains on fine-grained visual perception tasks.","feed_headline":"Lookup memory lifts vision-language models on 11 benchmarks","feed_subtitle":"By routing visual tokens to scene-level memory, the model beats text-only baselines on perception-heavy tasks.","key_machinery":"The central mechanism is the hierarchical two-level lookup table: an external embedding table reorganized as N_s scenes by N_p scene-primitives, replacing a flat |V|-way routing. Level-1 routes a projected continuous embedding to its nearest scene center via cosine similarity; Level-2 uses a shared MLP to score and softly combine the top-T primitives within that scene, so an arbitrary continuous embedding gets a relevant external representation without discrete IDs. Around this, LookME wraps three lookup pathways (text by ID, image by multi-scale two-level lookup, cross-modal by two-level lookup on hidden states), a Gumbel-Sigmoid per-token gate driven by the layer's attention contribution,","core_discovery":"LookME's central claim is that continuous multimodal embeddings can be made lookup-addressable without discrete IDs, and that doing so is what PLE-style methods have been missing in VLMs. The paper argues that text-only lookup leaves the dominant visual pathway unenhanced, and proposes organizing a large external embedding table as a two-level structure of scenes and scene-primitives. Image embeddings are projected into the table's latent space, routed to the best scene by cosine similarity, and then combined with the top scene-primitives via soft weights—a design that avoids the routing imbalance and parameter under-utilization of a flat lookup. The same two-level lookup is applied to decod","pith_inferences":["If the scene-primitive factorization generalizes, the same two-level lookup could be applied to other continuous token streams—audio, video, or latent features—without redesign, since any continuous embedding can be routed to learned scene centers.","The attention-contribution gate (SKA) is a cheap, general heuristic: tokens that self-attention already updates most are the ones that benefit from external memory. This rule could be lifted out and tested in other memory-augmented architectures as a standalone selector.","Because all comparisons share one backbone and one self-curated 60M-sample corpus, the cleanest test of whether multimodal lookup itself causes the gains is to swap in a second base VLM and a fixed public data mixture; the paper does not report such a transfer experiment.","The single benchmark where a text-only PLE method edges out LookME (BLINK) is a multi-image reasoning set, which suggests cross-modal lookup alone may not fully cover compositional multi-image inference—an area where a dedicated inter-image pathway might be needed."],"forward_implications":["If LookME is correct, the PLE scaling paradigm extends from text to vision, letting VLMs grow knowledge capacity without proportional GPU memory growth.","Text-only PLE baselines underperform the base model on most benchmarks in this study, implying that multimodal lookup pathways—not just larger tables—are what a memory-augmented VLM needs.","Sparse injection via attention-conditioned gating and cross-layer propagation lets most layers benefit from one retrieval, so the efficiency penalty of lookup can be contained.","The largest gains appear on fine-grained perception and multi-image tasks, suggesting ROM-resident lookup memory is most valuable where evidence alignment and detail matter.","The ablation shows text, image, and cross-modal pathways each add independent performance, so future PLE designs should treat modality as a first-class axis of the lookup."],"fun_headline_variants":["Scene-level memory lookup boosts vision-language models","Two-level lookup memory enhances VLMs","LookME: first multimodal lookup for VLMs","Lookup-based embeddings improve VLM efficiency","Hierarchical lookups lift VLMs beyond text-only"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported advantage rests on the assumption that all compared models were trained with the same data mixture, table-size budgets, and tuning hyperparameters; the paper defers those details to a supplementary document that is not present, so the gains could in principle come from the training recipe rather than from multimodal lookup.","fun_headline_variants_meta":{"raw":{"variants":["Scene-level memory lookup boosts vision-language models","Two-level lookup memory enhances VLMs","LookME: first multimodal lookup for VLMs","Lookup-based embeddings improve VLM efficiency","Hierarchical lookups lift VLMs beyond text-only"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1590,"prompt_tokens":791,"completion_tokens":799,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":731}},"tokens_in":535,"tokens_out":799,"duration_ms":8382,"temperature":1.0,"reasoning_tokens":731,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T06:28:49.363565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the strongest text-only PLE baseline and LookME under strictly identical conditions—same 60M-sample corpus with disclosed sampling ratios, same total table parameter budget, same optimizer and compute—then compare on the 11 benchmarks; if the text-only baseline matches or beats LookME, the claim that multimodal lookup drives the improvement is falsified.","supporting_citations":[],"review_version":1}