{"id":"6302c824-9932-4271-8795-c85ece8b5ebf","arxiv_id":"2412.06774","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A self-supervised method learns continuous image tokens in the text-embedding space of a frozen T2I diffusion model, capturing both semantics and visual details.","lead":"ViLex maps images into the token embedding space of a frozen text-to-image diffusion model, creating a 'visual lexicon' of continuous tokens that can reconstruct images and be combined with text prompts. This single representation aims to bridge image generation and understanding, enabling zero-shot personalization and stronger vision-language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's VLM gains may reflect extra SigLIP fine-tuning rather than ViLex's language-space reconstruction mechanism; a matched control is missing.","rationale":"The paper provides genuine evidence for reconstruction quality and zero-shot DreamBooth-like behavior, and the VLM improvements are consistent across many benchmarks. My concern is narrower but load-bearing: the central claim that ViLex, as a language-space visual representation, improves understanding is underdetermined because no control isolates the mechanism. The reader's weakest assumption was that a reconstruction-only objective through a frozen 64x64 T2I model suffices for transferable semantics. I agree with that concern, and the supplement's own limitation about missing small objects due to the low-resolution base model supports it. My sharper version is that even the observed gains on the 15 benchmarks do not yet prove the specific reconstruction objective is responsible, because the comparison includes additional fine-tuning compute and uses pre-pooling ViT features rather than the ViLex tokens themselves. The supplement's Table A1 quantifies the step-count confound, and Table A2 shows reconstruction quality and understanding can move in opposite directions, further suggesting that the reconstruction objective is not automatically the driver of semantic transfer. I am not proposing rejection; the method is plausible and the reported numbers are internally consistent. I recommend keeping the conditional verdict and adding one controlled ablation: a same-compute, same-architecture fine-tune of the same SigLIP backbone with a non-language-space reconstruction objective. If that control matches ViLex's gains, the understanding claim must be reframed; if it does not, the central claim becomes substantially stronger.","tokens_in":19772,"tokens_out":7604,"duration_ms":85228,"concrete_test":"Train a control encoder identical to ViLex SigLIP in every respect (same SigLIP-So400M initialization, same WebLI data, same 300K or 600K steps, batch size 2048, same optimizer, and the same attention-pooling overhead) but replace the frozen text-to-image decoder with a standard pixel-space reconstruction head, e.g., a VAE or MAE decoder, so the objective no longer passes through the CLIP/Imagen language space. Evaluate the frozen pre-pooling ViT features in the PaliGemma protocol on the same 15 benchmarks and compute reconstruction FID. If this control closes most of the performance gap versus Original SigLIP in Table 3, then ViLex's language-space mechanism is not the cause of the understanding gains; if the gap persists, the reconstruction-through-T2I design is validated as load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 claims that ViLex improves a strong SigLIP encoder across 15 vision-language benchmarks, and this is a core pillar of the 'simultaneous reconstruction and understanding' claim. However, Table 3 compares a SigLIP-So400M checkpoint that has received 300K-600K additional WebLI training steps with a reconstruction loss through a frozen Imagen decoder against the untouched official SigLIP checkpoint. The supplement's own Table A1 shows that increasing ViLex fine-tuning from 150K to 600K steps raises many downstream scores (e.g., COCOcap 140.5 to 141.5, SciQA 86.1 to 89.3), so training duration is a known confound. The paper does not rule out the possibility that any comparable continued pretraining of SigLIP, with its original contrastive objective or with a simpler pixel-space reconstruction objective, would produce similar or larger gains. Additionally, the features used for the understanding evaluation in Table 3 are the pre-pooling ViT features, not the ViLex text-space tokens; the downstream gains therefore do not directly demonstrate that the Visual Lexicon embeddings themselves are the cause. What is established is that a ViT initialized from SigLIP can be fine-tuned with this objective without destroying semantic usefulness; what is not established is that the language-space/T2I-decoder mechanism is load-bearing for the reported understanding improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ViLex, a ViT-based image encoder with attention pooling that maps images into n continuous embeddings in the token-embedding space of a frozen text encoder (OpenCLIP ViT-H/14) that conditions a frozen 64x64 Imagen text-to-image model. Training uses the diffusion denoising reconstruction loss plus a TailDrop strategy that randomly drops the last k tokens to encourage earlier tokens to carry richer semantics. The authors report three main results: (i) image reconstruction with lower FID/IS than DeDiffusion and image-guided DALL-E 3, including single-token reconstructions; (ii) zero-shot DreamBooth-style editing by concatenating ViLex tokens with natural-language prompts; and (iii) improved vision-language understanding when replacing SigLIP in the PaliGemma VLM with ViLex features, reported across 15 benchmarks. The supplement provides human-study instructions, ablations over training steps, pooling layers, and encoder initializations, and explicitly acknowledges that small objects can be missed because the base T2I model operates at 64x64 resolution.","tokens_in":20008,"tokens_out":6564,"duration_ms":65596,"significance":"If the central claims hold, ViLex offers an appealing unification: a single continuous token sequence in language space that can serve both as a prompt for a frozen text-to-image model and as a vision representation for downstream understanding. The reconstruction comparison against DeDiffusion is well posed, since DeDiffusion uses the same encoder architecture and the same Imagen decoder; the TailDrop mechanism is simple and makes the token budget adaptive; and the supplement contains useful ablations. However, the paper's central 'simultaneous reconstruction and understanding' claim is not yet established: the understanding gains in Section 4.3 are confounded with additional fine-tuning, and the state-of-the-art claim in Table 5 rests on unmatched comparisons. The paper would still be significant as a representation-learning recipe even if the mechanism is not fully isolated, but the current evidence requires revision.","major_comments":[{"comment":"The headline claim that ViLex improves a strong SigLIP vision encoder across 15 benchmarks is confounded by the amount of additional training. Table 3 compares an untouched official SigLIP checkpoint with a checkpoint that was further trained for 300K or 600K steps on WebLI through a frozen Imagen decoder; the main text in §4.1 says 300K steps, while Supplement Table A1 reports 150K and 600K checkpoints. Supplement Table A1 itself shows that extending the ViLex fine-tuning from 150K to 600K steps raises many downstream scores (e.g., SciQA from 86.1 to 89.3 and RefCOCOg-test from 60.2 to 62.6), so training duration alone is a known confound. A matched control is missing: the same initialization, data, and step budget should be trained with an alternative objective (e.g., continued SigLIP contrastive pretraining or a pixel-space/VAE reconstruction objective) and evaluated with identical hyperparameter sweeps. In addition, the features used for Table 3 are the pre-pooling ViT patch features, not the ViLex text-space tokens; the results therefore establish that ViLex-style fine-tuning does not destroy semantic usefulness, but they do not establish that the language-space/T2I-decoder mechanism is responsible for the reported gains.","section":"§4.3, Table 3 and Supplement Table A1"},{"comment":"The claim of 'SOTA performance across multiple visual question answering tasks' is not supported by the comparisons shown. ViLex is evaluated in a frozen-encoder PaliGemma-2B setup with 336 tokens at 224px, whereas the competing models use 7B-13B language models, higher resolutions (336–448px), 576–1280 tokens, and are fine-tuned on the target datasets, as the asterisks in Table 5 indicate. These differences in model scale, input resolution, token budget, and training data dominate the VQA score differences, so no conclusion about state-of-the-art status can be drawn without matched ablations at a fixed scale. The claim should be softened to 'competitive with much larger systems' or supported with controlled comparisons.","section":"§4.3, Table 5"},{"comment":"The quantitative reconstruction results are reported without uncertainty estimates. Table 1 gives FID and IS for single runs, and Table 2 reports 91–98% human win rates without stating the number of participants, the number of image-prompt pairs, how stimuli were sampled, or inter-annotator agreement. Since the central 'higher fidelity' claim rests on these numbers, the paper should report standard errors or confidence intervals for FID/IS (e.g., across multiple random subsets or generation seeds) and full human-study statistics. Without these details, the magnitude of the reported margins is difficult to assess.","section":"§4.2, Tables 1 and 2"},{"comment":"The FID values reported for ViLex are mutually inconsistent across tables. Table 3 reports FID 2.38 for ViLex SigLIP, Table 1 reports FID 2.07 for the 75-token ViLex configuration, and Supplement Table A2 reports FID 2.58 for the 5-layer attention-pooling configuration described in the main text. The reconstruction protocol (token count, input resolution, evaluation set, whether the pooling layer is used, and the number of generation seeds) is not specified consistently across these tables, so the reader cannot reconcile the numbers. Please state the exact protocol for each FID column or remove the redundant values.","section":"Supplement A3, Table A2 and main Tables 1 and 3"}],"minor_comments":[{"comment":"The image encoder is described in the main text as 'SigLIP@224 [88] ... ViT-base' but in the supplement as 'SigLIP-So400M@224'; these are different models, so the architecture actually used must be stated consistently.","section":"§4.1 and Supplement A1"},{"comment":"The main text says training runs for 300K steps, while the supplement says doubling from 300K to 600K improves performance and Table A1 evaluates 150K and 600K checkpoints; the main-text tables should identify which checkpoint underlies each reported number.","section":"§4.1 and Supplement A1"},{"comment":"The symbol k is used both for the number of patch tokens and for the number of tokens dropped by TailDrop; please use distinct notation for these two quantities.","section":"§3.1"},{"comment":"The text states that ViLex improves the baseline 'by over 1 point margin on a variety' of tasks, but Table 3 shows several tasks with gains below 1 point (e.g., VQAv2 +0.2, OKVQA +1.3, MSRVTT +1.3); the wording should match the data.","section":"§4.3, Table 3"},{"comment":"The VAE baseline in Table 4 is not described: the table does not state which VAE tokenization is used, how many tokens are fed to the VLM, or whether the VAE encoder is frozen; without this information the '>8 times lower' comparison is hard to interpret.","section":"§4.3, Table 4"}],"recommendation":"major_revision","confidential_remarks":"This is a Google DeepMind paper with no released code or checkpoints, so reproducibility depends on the precision of the experimental protocol. The core reconstruction experiment against DeDiffusion is the strongest part of the paper and is reasonably controlled. The main risk to the paper's central claim is the missing matched control in Section 4.3: the understanding gains may be attributable to additional fine-tuning rather than to the language-space/T2I reconstruction mechanism. I recommend major revision rather than rejection because the recipe is plausible and the core evidence is partly sound, but the understanding claim needs a controlled study before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ViLex maps images to continuous embeddings in the text-token space of a frozen T2I diffusion model, trained with a reconstruction loss through that model. The core idea is a clean, useful step beyond DeDiffusion, which is stuck with discrete tokens. The reconstruction numbers are convincing: even one ViLex token beats DeDiffusion on FID/IS with the same Imagen backbone, and the zero-shot DreamBooth demos are genuinely impressive. The TailDrop and TFG additions are sensible small contributions. This part of the paper deserves serious attention.\n\nThe understanding section is where the evidence gets softer, and the stress-test note lands. Table 3 compares a SigLIP checkpoint that has been fine-tuned with the ViLex objective for hundreds of thousands of WebLI steps against the untouched official SigLIP. The supplement's own Table A1 shows a steady climb from 150K to 600K steps on many benchmarks, so training duration is a real confound. Without a control—say, SigLIP continued pretraining with its original contrastive loss, or with a pixel-space reconstruction objective—you can't attribute the downstream gains to the language-space/T2I decoder mechanism. And the features used in that table are the pre-pooling ViT features, not the ViLex tokens, so the understanding improvements don't directly validate the Visual Lexicon itself. That doesn't sink the paper; it means the claim \"ViLex improves understanding\" is currently \"continued pretraining through a diffusion decoder improves understanding,\" which is still interesting but less specific.\n\nOther soft spots are moderate. The human study reports 95-98% win rates with no participant count or stimulus sampling details; FID/IS have no error bars; Table 5's \"SOTA\" phrasing overreaches because the baselines are not evaluated in the same protocol. No code or data release, which limits reproducibility. The supplement's own note about missing small objects is honest and points to a real limitation of the 64x64 base model.\n\nOverall the generation/reconstruction contribution is solid and new, and the understanding claim is plausible but under-evidenced. I'd send this to review, and the main revision request would be a matched training control plus evaluation details. Researchers working on vision encoders, diffusion inversion, or multimodal systems will want to cite it.","headline":"A useful continuous-token alternative to DeDiffusion with convincing reconstruction results, but the VLM gains need a matched training control before the understanding claim is fully established.","tokens_in":20544,"tokens_out":3205,"would_cite":true,"duration_ms":33046,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ViLex asks whether a single image representation can both reconstruct pixels and understand scenes, and answers by mapping images into the text-token space of a frozen text-to-image diffusion model.","keywords":["visual language","text-to-image diffusion","image reconstruction","vision encoder","self-supervised learning","text embedding space","zero-shot generation","representation learning"],"falsifier":"Train the same ViLex pipeline with a higher-resolution base T2I model, for example a 256x256 Imagen, and measure small-object recall and fine-detail reconstruction; if the representation quality stays flat despite the richer decoder, the encoder objective is the bottleneck, while a sharp increase would tie the current claims to the 64x64 teacher. Alternatively, probe a frozen single-token ViLex embedding on the Describable Textures dataset: if it cannot discriminate texture classes above chance, the claim that one ViLex token preserves fine visual detail is falsified.","tokens_in":19537,"feed_emoji":"🖼️","tokens_out":5908,"duration_ms":61425,"temperature":0.7,"pith_summary":"The paper introduces ViLex, an image encoder whose outputs live directly in the text-embedding space of a frozen text-to-image diffusion model. ViLex is trained only to reconstruct the input image through that frozen decoder, so the same representation must carry high-level semantics and fine visual detail. The authors report that even a single ViLex token reconstructs images with lower FID than a text-based inversion baseline, and that using ViLex as a vision encoder improves a vision-language model across 15 benchmarks relative to a strong SigLIP baseline. The central wager is that a generative decoder can serve as a teacher for understanding features.","feed_headline":"Images turned into text tokens re-create images better than captions","feed_subtitle":"New visual \"language\" retains details words miss and lifts vision-language models on 15 benchmarks.","key_machinery":"The load-bearing object is the ViLex token sequence, produced by attention pooling over ViT patch features and placed into the text-embedding lookup space of a frozen text encoder, so a frozen diffusion U-Net treats the tokens exactly like text. The training loss is the standard diffusion denoising objective through that frozen decoder; TailDrop randomly drops the last k tokens during training to push semantic information into the earlier tokens; and Text-Free Guidance interpolates between ViLex-only and ViLex-plus-text conditioning at inference. The frozen decoder is a 64x64 Imagen model, which sets a hard resolution bottleneck on the visual information the representation must carry.","core_discovery":"ViLex is a 'visual lexicon': a sequence of continuous tokens that behave like text tokens for a frozen text-to-image model but encode visual content that words cannot express. The encoder, a ViT initialized from SigLIP followed by five attention-pooling blocks, projects patch features into the index-to-embedding lookup space of the T2I model's text encoder, so the frozen diffusion U-Net can consume them as if they were text. Training minimizes a standard denoising objective, backpropagating through the frozen text encoder and U-Net into the vision encoder; a TailDrop schedule randomly drops trailing tokens, forcing the leading tokens to carry the semantic core. At inference the token count becomes a dial between coarse semantics (one token) and fine detail (75 tokens). The paper claims this setup beats text-based image inversion in reconstruction fidelity, enables zero-shot DreamBooth-style personalization by concatenating ViLex tokens with natural-language prompts, and improves a SigLIP-initialized vision encoder on captioning, visual question answering, and referring segmentation.","pith_inferences":["If ViLex transfers to other frozen generative models, the apparent dichotomy between understanding features and generative features becomes a matter of training target rather than architecture, suggesting a general recipe for visual-language pretraining.","The paper's FID comparison uses different token counts for DeDiffusion and ViLex; a fairer bit-budget-equal test would hold the total token count constant, and the paper's one-token result predicts ViLex would still win.","The 64x64 base model is a likely ceiling: a higher-resolution frozen T2I decoder should improve small-object recall and fine-detail reconstruction, a testable extension the supplement itself gestures toward.","ViLex tokens could double as a retrieval index, since images become searchable in text space by cosine similarity to caption embeddings, yielding a text-to-image retrieval procedure that inherits the T2I model's semantic granularity."],"forward_implications":["ViLex tokens can be typed directly into any text prompt, turning subject-driven generation into a zero-shot operation that needs no per-object fine-tuning.","Vision encoders trained with the ViLex objective improve in both reconstruction and understanding, so the same objective can upgrade existing encoders such as SigLIP.","The token count becomes a user-settable fidelity dial, letting systems trade semantic compactness for fine detail as the task requires.","Because ViLex tokens share a space with text, a single sequence can interleave visual and textual content for a frozen T2I model, enabling re-contextualization, art rendition, and accessorization without adapter modules.","Representation learning for understanding can draw its training signal from a frozen generative decoder alone, with image-only training barely hurting downstream vision-language performance."],"supporting_citations":[{"why":"DeDiffusion is the direct baseline for image-to-text-to-image inversion, using the same encoder architecture and T2I model, and supplies the discrete-token comparison ViLex must beat.","marker":"[76]"},{"why":"Imagen is the frozen text-to-image diffusion model used as the decoder and teacher in the ViLex autoencoder framework.","marker":"[60]"},{"why":"CLIP defines the text-encoder vocabulary lookup space that ViLex embeddings inhabit and the 77-token context limit that constrains the token budget.","marker":"[53]"},{"why":"SigLIP provides the pretrained ViT initialization for the ViLex encoder and is the strong vision-encoder baseline that ViLex improves upon in reconstruction FID and VLM benchmarks.","marker":"[88]"},{"why":"PaliGemma is the vision-language model framework used to evaluate ViLex tokens and the fine-tuned SigLIP encoder on downstream understanding tasks.","marker":"[6]"},{"why":"DALL-E 3 serves as an image-guided generation baseline in the human study for layout, semantic, and style consistency.","marker":"[5]"},{"why":"DreamBooth is the per-subject fine-tuning approach that ViLex aims to match in identity-preserving generation without any test-time training.","marker":"[58]"},{"why":"WebLI is the training data source, providing the image and image-text pairs used during ViLex pretraining.","marker":"[11]"}],"fun_headline_variants":["ViLex: image tokens beat text embeddings at reconstruction","Visual Lexicon: encoding images as text, not captions","Single ViLex token beats text embeddings in fidelity","Zero-shot DreamBooth with image-as-text ViLex tokens","ViLex lifts vision-language models on 15 benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that training only to reconstruct images through a frozen 64x64 text-to-image model forces the embeddings to carry both semantic and detailed visual information that transfers to understanding tasks; if the reconstruction loss instead biases the encoder toward low-level memorization, or if the low-resolution decoder drops information such as small objects, the claimed gains over SigLIP would be overstated.","fun_headline_variants_meta":{"raw":{"variants":["ViLex: image tokens beat text embeddings at reconstruction","Visual Lexicon: encoding images as text, not captions","Single ViLex token beats text embeddings in fidelity","Zero-shot DreamBooth with image-as-text ViLex tokens","ViLex lifts vision-language models on 15 benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2986,"prompt_tokens":999,"completion_tokens":1987,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1922}},"tokens_in":615,"tokens_out":1987,"duration_ms":15565,"temperature":1.0,"reasoning_tokens":1922,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:17:49.574746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same ViLex pipeline with a higher-resolution base T2I model, for example a 256x256 Imagen, and measure small-object recall and fine-detail reconstruction; if the representation quality stays flat despite the richer decoder, the encoder objective is the bottleneck, while a sharp increase would tie the current claims to the 64x64 teacher. Alternatively, probe a frozen single-token ViLex embedding on the Describable Textures dataset: if it cannot discriminate texture classes above chance, the claim that one ViLex token preserves fine visual detail is falsified.","supporting_citations":[{"cited_title":"De-diffusion makes text a strong cross- modal interface","cited_arxiv_id":null,"evidence_quote":"DeDiffusion is the direct baseline for image-to-text-to-image inversion, using the same encoder architecture and T2I model, and supplies the discrete-token comparison ViLex must beat."},{"cited_title":"Photorealistic text-to-image diffusion models with deep language understanding","cited_arxiv_id":null,"evidence_quote":"Imagen is the frozen text-to-image diffusion model used as the decoder and teacher in the ViLex autoencoder framework."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP defines the text-encoder vocabulary lookup space that ViLex embeddings inhabit and the 77-token context limit that constrains the token budget."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"SigLIP provides the pretrained ViT initialization for the ViLex encoder and is the strong vision-encoder baseline that ViLex improves upon in reconstruction FID and VLM benchmarks."},{"cited_title":"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation","cited_arxiv_id":null,"evidence_quote":"DreamBooth is the per-subject fine-tuning approach that ViLex aims to match in identity-preserving generation without any test-time training."}],"review_version":1}