{"id":"768c60a4-b887-41fc-86d7-541c4dff4903","arxiv_id":"2608.06972","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GEB evaluates multimodal embeddings by decoding answers from the embedding alone, revealing information bottlenecks and model rankings that differ from retrieval-style benchmarks.","lead":"This paper introduces GEB, a benchmark that asks a language model to answer visual questions using only a frozen embedding and the question text, with no access to the image. It finds that current embeddings preserve far less answer-relevant detail than a model with the image, and that question-conditioned embeddings recover much more information.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-model GEB rankings and category bottleneck conclusions rest on an untested assumption that the Qwen3-0.6B decoder is a neutral readout; a non-Qwen decoder rerun is needed before trusting Table 4.","rationale":"The paper has real strengths: matched-embedding controls (Table 5) demonstrate dependence on sample-specific information, the leakage audit is thorough, the Gemini-3-Flash answerability filter is disclosed and framed honestly as defining an answerable pool, and the decoder sensitivity and data-scale studies are useful first steps. However, the single most load-bearing concern is readout neutrality and adequacy, which the reader also identified. The Qwen3-0.6B decoder is from the same family as the two top-scoring models, and no cross-family decoder or same-readout oracle is tested. This matters because the benchmark's central contribution is comparative: GEB rankings are claimed to reveal information bottlenecks beyond what MMEB-V2 captures. If the Qwen decoder maps Qwen-family embedding spaces more easily into its hidden space, the ranking differences in Table 4 and the claim that fine-grained text/document information is lost from embeddings would be at least partly artifacts of the readout. The absence of an oracle control leaves open the alternative that the small 0.6B decoder simply cannot read exact strings from any input. The paper's own supplementary acknowledges that a different training seed could fall outside the reported item-sampling intervals, so single-run scores add a second layer of uncertainty. These concerns do not invalidate the benchmark concept or the qualitative control results; they make the cross-model rankings and category-level bottleneck conclusions conditional on readout validation. The reader's CONDITIONAL verdict is appropriate, and the proposed decoder-family rerun is the concrete check that would settle the issue.","tokens_in":15672,"tokens_out":7447,"duration_ms":76542,"concrete_test":"Rerun the full GEB protocol on the 900-item test set with a decoder from a different family (e.g., Llama-3.2-1B-Instruct) using the same adapter architecture, LLaVA-NeXT 738K training data, one-epoch budget, loss masking, and evaluation pipeline as Table 4. Compare the resulting VL-joint model ordering and category gaps to Table 4, and additionally train three seeds each for Qwen3-VL-Embedding-8B and UME-R1-2B to bound optimization variance. If Qwen3-VL-Embedding-8B no longer leads and the scene-text versus natural-image gap collapses below roughly 10 points, the rankings and bottleneck conclusions are substantially readout artifacts; if the pattern persists, the family-alignment concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's utility rests on the readout being a neutral probe across embedding models. The decoder is Qwen3-0.6B (Section 3.2), and the top VL-joint scores in Table 4 come from Qwen3-VL-Embedding-2B/8B, the same model family. The supplementary's readout-adequacy checks only vary decoder size within the Qwen3 family (0.6B vs 1.7B) and training-data scale; they never test a decoder from another family, and no oracle control (the same trained decoder fed original-image-derived visual features) is run. Consequently, the low scene-text and visual-document scores (e.g., 23.67 and 13.42 for the best visual-only models) could reflect the decoder's inability to read exact strings from any input rather than an information bottleneck in the embeddings. The central claim that GEB exposes information bottlenecks that separability-based evaluation does not capture, and the specific model orderings in Table 4, therefore rest on an unexamined interaction between embedding space and decoder family. Additionally, each model-mode row comes from a single training run (Supplementary S4.2), so reported differences such as 32.92 vs 33.21 are within item-sampling noise and could also be training noise; the paper itself notes a different seed could fall outside the reported intervals.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Generative Embedding Benchmark (GEB), which evaluates frozen multimodal embedding models by training a decoder to answer visual questions using only the embedding and the question text, without access to the original image or intermediate visual features. The benchmark has a curated 1,800-item development split and a 900-item held-out test split across natural images, scene text, and visual documents. Seven public embedding models are evaluated in visual-only and vision-language joint modes under a common decoder (Qwen3-0.6B) and training recipe. The main results are that visual-only scores span a narrow range (28.25–33.21), VL-joint encoding raises VLM-based model scores substantially (best 65.56 for Qwen3-VL-Embedding-8B), natural-image information is more recoverable than scene text or visual-document information, and GEB rankings differ from MMEB-V2 rankings. Control experiments show that matched, sample-specific embeddings outperform text-only, zero, and shuffled inputs.","tokens_in":15946,"tokens_out":6978,"duration_ms":68731,"significance":"The paper proposes a genuinely complementary evaluation axis to separability-based benchmarks: instead of asking whether an embedding can rank candidates, GEB asks whether answer-relevant content can be recovered from the embedding by a generative readout. The protocol is carefully staged, with development-set selection, a held-out test set, source-attrition reporting, image-identity leakage audits, and multiple control conditions. The authors also disclose important limitations, including the answerability-filter dependence and the fact that bootstrap intervals reflect item sampling rather than training-run variation. If the decoder-neutrality concern can be addressed, the benchmark would be a useful contribution to the community, and the paper's concrete observations about scene-text and visual-document recoverability would be valuable.","major_comments":[{"comment":"The cross-model rankings and the central claim that GEB exposes information bottlenecks rest on the assumption that the shared Qwen3-0.6B decoder is a neutral readout across embedding families. The top-scoring VL-joint models in Table 4 are Qwen3-VL-Embedding-2B and Qwen3-VL-Embedding-8B, which share the Qwen3-VL family with the decoder LM, while the readout-adequacy checks in Supplementary S2 only vary decoder size within the Qwen3 family and training-data scale. The paper also lacks an oracle control in which the same trained decoder receives image-derived visual features or image tokens from the same visual encoder family; without such a control, the low scene-text and visual-document scores (e.g., 23.67 and 13.42 for the best visual-only models) could reflect a decoder-side inability to reproduce exact strings from any input rather than an embedding-side information bottleneck. I request either a control that feeds original-image visual features through the same adapter/decoder pipeline as an upper bound, or a rerun of the main comparisons with a decoder from a different LM family, to separate decoder-family alignment from embedding content.","section":"§3.2 and Table 4"},{"comment":"Each model–mode row in Table 4 comes from a single training run, and the bootstrap intervals in S4.2 are explicitly stated to quantify test-item sampling variation, not optimization variation; the paper itself notes that a decoder retrained with a different seed could score outside the reported intervals. Under this uncertainty, several load-bearing comparisons are not established: the visual-only range from 28.25 to 33.21 and the 'nearly tied' characterization of Qwen3-VL-Embedding-8B versus 2B fall within the item-sampling CIs, and the VL-joint ordering changes relative to MMEB-V2 (e.g., Qwen3-VL-Embedding-2B at 44.87 versus UME-R1-2B at 51.45) could be training noise rather than a real difference. I ask for at least one independent seed for the main model–mode rows, or a formal seed-variance estimate, and for ranking claims to be restricted to differences that survive both item-sampling and seed variation.","section":"§5.3, Table 4, and Supplementary S4.2"},{"comment":"The visual-only and VL-joint conditions differ not only in whether the embedding is conditioned on the question but also in training-instance construction: visual-only uses one embedding shared across the full conversation, while VL-joint expands each conversation into turn-level instances, increasing the training set from about 736,900 to about 4,601,635 instances (S4.1). The gain attributed to query-conditioned encoding is therefore confounded with supervision format and data quantity. Although §5.3 acknowledges that 'this gain reflects the complete mode-specific pipeline rather than encoder conditioning alone,' the Discussion interprets the gain in terms of query-conditioned compression. I request an ablation that trains a visual-only decoder on the same turn-level instance format, reusing the same frozen image embedding for each turn, or otherwise equates the supervision budget, so that the mode comparison isolates encoder conditioning.","section":"§3.3 and §5.3"}],"minor_comments":[{"comment":"In the visual-only block, the Scene and Doc entries for VLM2Vec-V2 are rendered as '11.6015.19'; a missing separator makes the value ambiguous and should be fixed.","section":"Table 4"},{"comment":"The sentence 'A leakage audit confirms that no image-identity group crosses the two splits: ... are all zero (Table 2)' refers to a table that actually reports per-source candidate attrition; the zero-overlap audit is described in Supplementary S3. Please correct the cross-reference.","section":"§4.2"},{"comment":"The comparison with MMEB-V2 would benefit from stating explicitly that the archived MMEB-V2 scores may come from different checkpoints or evaluation versions than the GEB runs, to avoid implying that the same model instances are being compared on the two benchmarks.","section":"§5.5"},{"comment":"The two-slot late-fusion interface for CLIP and SigLIP is mentioned only briefly; a short example of how the two vectors are concatenated or otherwise combined in the adapter would improve reproducibility.","section":"Supplementary S4.1"}],"recommendation":"major_revision","confidential_remarks":"The main decision risk is the decoder-family confound: if the authors can supply a non-Qwen decoder control or an oracle upper-bound control using the same readout, the central claim about embedding-side bottlenecks will be substantially stronger. The single-run uncertainty is also important for the cross-model ranking claims. The benchmark design is otherwise careful, and the paper is transparent about its limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful benchmark paper with a real design flaw in the cross-model comparison. The idea—measure what a frozen embedding lets a generator recover, rather than retrieval separability—is genuinely new as a systematic benchmark. The paper does a lot right: fixed decoder recipe, held-out test split, leakage audits, matched/zero/shuffled controls, clear category breakdowns. The controls show the score depends on the actual embedding, not language priors. That part is solid.\n\nThe soft spot is the decoder. They use Qwen3-0.6B as the readout for all models, and the two best-scoring models in Table 4 are Qwen3-VL-Embedding-2B/8B—same family. Their own readout adequacy checks only vary decoder size within Qwen3 (0.6B vs 1.7B) and training data scale; they never test a decoder from another family, and there is no oracle control where the same trained decoder is fed original-image-derived visual features. So the low scene-text and visual-document scores could partly be the decoder's inability to read exact strings from any input, not an embedding bottleneck. That undercuts both the category-level conclusion and the specific model ordering in Table 4. The stress-test note is right about this.\n\nAlso, each model–mode row is a single training run. The paper admits a different seed could fall outside the reported intervals. Given the visual-only range is only about 5 points (28.25 to 33.21), training noise alone could reorder the middle of the pack. The bootstrap CIs only cover item sampling, not optimization variance. That's a minor-to-moderate issue; for a benchmark meant to rank models, it matters.\n\nWhat's still good: the dataset curation is careful—answerability filtering by Gemini-3-Flash is disclosed and reasonable, and the leakage audit (image-hash, perceptual hash, LLaVA-NeXT overlap) is thorough. The two encoding modes (visual-only vs VL-joint) are a genuinely useful distinction, and the qualitative multi-question interrogation is illustrative, not load-bearing.\n\nBottom line: as a proof-of-concept evaluation axis, it deserves peer review. The authors should add a non-Qwen decoder (or ensemble) and a couple of training seeds before the rankings are trusted. I would cite it as a methodological reference, not for its specific leaderboard.","headline":"A genuinely useful benchmark idea and careful dataset, but the Qwen-family decoder makes the cross-model rankings—and part of the category story—untrustworthy as-is.","tokens_in":16501,"tokens_out":1890,"would_cite":true,"duration_ms":17971,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes the Generative Embedding Benchmark (GEB), which scores dense embeddings by how much answer-relevant content a decoder can recover from them, arguing that separability-based benchmarks miss information bottlenecks.","keywords":["generative embedding benchmark","embedding evaluation","generative readout","information bottleneck","multimodal embeddings","visual question answering","separability vs recoverability"],"falsifier":"Train the identical GEB readout recipe with a decoder from a different language-model family (for example, a Llama-based 0.6B model) and compare the new ranking against Table 4; if Qwen3-VL-Embedding models no longer lead while UME-R1 or VLM2Vec improve, then the original ranking is confounded by encoder–decoder family alignment rather than by information content alone.","tokens_in":15463,"feed_emoji":"🧠","tokens_out":5192,"duration_ms":45173,"temperature":0.7,"pith_summary":"The paper argues that standard embedding benchmarks, which score representations by their ability to separate correct answers from distractors in a gallery, do not tell us whether the content needed to generate an answer actually survives compression into a dense vector. To test this directly, the authors introduce the Generative Embedding Benchmark (GEB): a frozen embedding, together with the textual question, is fed to a decoder that must produce the answer without ever seeing the original image. Each of seven public embedding models receives the same decoder architecture, training data, and optimization budget, so answer quality becomes an operational measure of recoverable content. On the held-out test set, visual-only scores cluster tightly (28.25–33.21), while question-conditioned joint encoding lifts the best model to 65.56, still far below the 84.30 of a reference that sees the image. The paper's central conclusion is that strong retrieval performance does not imply preservation of the fine-grained information downstream generators need.","feed_headline":"Retrieval winners can still lose answer-critical details","feed_subtitle":"A generative readout of frozen vectors exposes information bottlenecks that separability benchmarks miss.","key_machinery":"The generative readout is the central mechanism: a frozen embedding is layer-normalized and projected by a two-layer MLP (width 1024) into the hidden space of Qwen3-0.6B, where it replaces a special <EMBED_PAD> token at the start of the first user turn. The decoder is fine-tuned end-to-end with a fixed recipe (one epoch, learning rate 1e-4, effective batch 512, assistant-token loss), and answer quality under this readout is defined as generative information. GEB adds a curated 2,700-item VQA set (1,800 dev, 900 test) balanced across natural images, scene text, and visual documents, with an answerability filter retaining items a reference VLM answers correctly.","core_discovery":"GEB measures what the authors call generative information: the answer-relevant content a fixed generative readout can recover from a frozen embedding. The readout is a two-layer adapter plus Qwen3-0.6B, trained on LLaVA-NeXT 738K with the embedding model frozen. In visual-only mode the image is embedded once and reused across questions; in vision-language joint mode the embedding is computed from image and question together. Controlled interventions show that matched embeddings outperform text-only, zero, and shuffled inputs by 11–43 points, and the two encoding modes produce different model rankings that do not reproduce the MMEB-V2 ordering. Category-level scores show natural-image content is far more recoverable than scene text or visual-document content, pinpointing where fixed-capacity embeddings lose exact strings, numbers, and layout.","pith_inferences":["Because the decoder and the top-scoring embedding models come from the same Qwen3-VL family, the cross-model ranking may partly reflect encoder–decoder hidden-space alignment; testing a decoder from a different family would reveal whether the ranking is an artifact of that alignment.","The answerability filter uses a single reference VLM (Gemini-3-Flash), so GEB measures recoverability within an answerable pool; sample-level scores may not transfer to items that this reference cannot answer.","One could use the GEB protocol to train embedding models directly: since the readout is differentiable, a loss on generative answer quality could shape embeddings toward preserving generator-relevant information, not just separability.","Category-level scores provide a per-question diagnostic that could be used to allocate embedding capacity dynamically—for instance, more slots or longer vectors for scene-text and document images."],"forward_implications":["If GEB rankings are taken seriously, a model's position on separability benchmarks such as MMEB-V2 does not predict how much of its information a generator can use; models with lower retrieval scores can rank higher on generative readout.","The large visual-only-to-VL-joint gains (e.g., Qwen3-VL-Embedding-8B from 33.21 to 65.56) indicate that question-conditioned encoding reallocates embedding capacity toward query-relevant content, at the cost of reusability.","Scene text and visual documents show that fixed-capacity embeddings lose exact strings, numbers, and layout more readily than coarse scene semantics, so applications that rely on OCR-like detail need embeddings with adaptive capacity or explicit slots.","The control experiments imply that GEB scores are not explained by decoder language priors: shuffled or blank inputs drop scores by roughly 40 points in joint mode.","A lightweight query-conditioned adapter over a precomputed image embedding could reclaim part of the VL-joint selectivity without sacrificing offline precomputation."],"supporting_citations":[{"why":"Supplies the MMEB-V2 comparison table that GEB rankings are shown not to reproduce.","marker":"(Meng et al. 2025)"},{"why":"Provides two of the evaluated embedding models, Qwen3-VL-Embedding-2B and -8B.","marker":"(Li et al. 2026)"},{"why":"Is the training corpus (LLaVA-NeXT 738K) used for the shared decoder readout.","marker":"(Liu et al. 2024a)"},{"why":"Supplies the evaluation harness, answer-processing procedures, and metrics used to score GEB.","marker":"(Zhang et al. 2025)"},{"why":"Is the source of the decoder LM (Qwen3-0.6B) architecture and chat template.","marker":"(Yang et al. 2025)"},{"why":"Defines the reference-VLM answerability filter used to construct the GEB item pool.","marker":"(Google 2025)"},{"why":"Provides the VLM2Vec embedding model and the original MMEB formulation that motivates the comparison axis.","marker":"(Jiang et al. 2025)"}],"fun_headline_variants":["Embeddings hide answer details from generative readers","Frozen vectors pass separability tests yet lose key info","Generative readout reveals what embeddings truly preserve","Benchmark shows embeddings bleed answer-critical content","What survives in a dense embedding? Less than you think"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decoder is assumed to be a neutral probe that reads all embedding spaces equally; in practice it is a Qwen-family model, and the top-scoring embedders are also Qwen3-VL models, so a family-specific alignment could inflate their scores.","fun_headline_variants_meta":{"raw":{"variants":["Embeddings hide answer details from generative readers","Frozen vectors pass separability tests yet lose key info","Generative readout reveals what embeddings truly preserve","Benchmark shows embeddings bleed answer-critical content","What survives in a dense embedding? Less than you think"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1452,"prompt_tokens":988,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":405}},"tokens_in":604,"tokens_out":464,"duration_ms":5373,"temperature":1.0,"reasoning_tokens":405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:22:16.370813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical GEB readout recipe with a decoder from a different language-model family (for example, a Llama-based 0.6B model) and compare the new ranking against Table 4; if Qwen3-VL-Embedding models no longer lead while UME-R1 or VLM2Vec improve, then the original ranking is confounded by encoder–decoder family alignment rather than by information content alone.","supporting_citations":[{"cited_title":"Training","cited_arxiv_id":null,"evidence_quote":"Supplies the MMEB-V2 comparison table that GEB rankings are shown not to reproduce."},{"cited_title":"2025 , address=","cited_arxiv_id":null,"evidence_quote":"Is the source of the decoder LM (Qwen3-0.6B) architecture and chat template."}],"review_version":1}