{"id":"212099b2-eade-4484-a2a1-616d1dc30bf6","arxiv_id":"2506.18902","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"jina-embeddings-v4 is a multimodal embedding model built on Qwen2.5-VL with LoRA adapters and dual single/multi-vector outputs, and it introduces the Jina-VDR benchmark for visually rich retrieval.","lead":"This paper introduces jina-embeddings-v4, a 3.8-billion parameter model that turns text, images, and code into searchable vectors, and it presents a new benchmark for visually rich document retrieval. The reported results are strong on visual document search and competitive elsewhere, though the paper's broad state-of-the-art claim goes beyond what its own tables show.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Jina-VDR state-of-the-art claim rests on a self-built benchmark whose test tasks may overlap with the training data; without a contamination check, the reported margins over other visual retrievers are not interpretable.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: Jina-VDR is an author-built benchmark with probable training-data overlap and synthetic queries, so the reported visual-retrieval supremacy is not yet credible. I agree with the CONDITIONAL verdict because the concern is concrete but addressable: a contamination analysis and a re-run on an overlap-free subset could settle it. I do not see grounds for outright rejection, since the model is released, the architecture and training procedure are described in enough detail to be reproduced, and the ViDoRe results are competitive even if not decisively superior. The abstract's broader 'state-of-the-art performance on both single-modal and cross-modal retrieval tasks' is internally overstated relative to the paper's own tables (e.g., voyage-code-3 beats jina-embeddings-v4 on COIR, nllb-clip-large-siglip beats it on Crossmodal3600, and gemini-embedding-001 leads MMTEB), but the paper's more specific and defensible contribution is the visually rich retrieval capability, so the fairness of Jina-VDR is the single condition on which the key claim hinges.","tokens_in":25179,"tokens_out":3923,"duration_ms":42533,"concrete_test":"Compute exact and near-duplicate overlap (perceptual hashing) between every Jina-VDR test image set and the training corpora named in Section 5.2.1, especially VDR multilingual and any released mined-hard-negative data, then recompute the Table A2 average nDCG@5 for jina-embeddings-v4 (dense and multi) and the strongest baseline, dse-qwen2-2b-mrl-v1, on the overlap-free subset. If the model's lead over baselines shrinks by more than a few points or disappears, the state-of-the-art claim on Jina-VDR is not supported and must be withdrawn or heavily qualified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central visual-retrieval claim (Section 7.3: \"provides the current state-of-the-art, in both single- and multi-vector mode\" on Jina-VDR) depends entirely on Jina-VDR being a fair, uncontaminated benchmark. That condition is not established, and the paper contains concrete reasons to doubt it. Section 5.2.1 states that multimodal hard negatives were mined from \"VDR multilingual\" and other \"curated multimodal datasets,\" while Section 6.4 says Jina-VDR \"extends the ViDoRe benchmark\" and re-purposes existing datasets (ChartQA, PlotQA, CharXiv, TableVQA, etc., per Table A1). If the same document images appear on both sides of the train/test boundary, the scores in Table A2 partly reflect memorization rather than retrieval quality. Section 6.3 further reveals that many Jina-VDR queries were generated with Qwen2 and filtered by LLMs; jina-embeddings-v4 is built on Qwen2.5-VL, so these synthetic queries may be systematically easier for the proposed model than for baselines. The paper reports no overlap statistics, no evaluation code, and no variance or significance estimates. Without these, the margin between jina-embeddings-v4 and dse-qwen2-2b-mrl-v1 (e.g., 73.98 vs. 67.25 dense, 80.55 vs. 63.80 late in Table 3) cannot be attributed to retrieval quality. An independent check of overlap and sensitivity to the synthetic subset is therefore the load-bearing test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces jina-embeddings-v4, a 3.8-billion-parameter multimodal embedding model built on Qwen2.5-VL-3B-Instruct. The model uses task-specific LoRA adapters for asymmetric retrieval, text matching, and code search, and supports both single-vector (Matryoshka-truncatable 2048-dim) and multi-vector late-interaction outputs. Training uses a joint InfoNCE/CoSENT/MRL objective on text-text and text-image pairs. The paper also introduces Jina-VDR, a benchmark extending ViDoRe with 30 additional visually rich document retrieval tasks, and evaluates the model on Jina-VDR, ViDoRe, CLIP Benchmark, MTEB/MMTEB, LongEmbed, CoIR, and STS benchmarks.","tokens_in":25521,"tokens_out":5893,"duration_ms":50089,"significance":"If the results hold, the model provides a practical unified text-image embedding solution, and the Jina-VDR benchmark could become a useful resource for visually rich retrieval. The paper clearly describes the architecture and training procedure, and the released benchmark and model links (Hugging Face) enable reproducibility. The strongest evidence is on ViDoRe and Jina-VDR visual retrieval, where the model outperforms ColPali, DSE, and CLIP-style baselines by large margins. However, the headline 'state-of-the-art' claim is broader than the evidence: on several text benchmarks the model trails gemini-embedding-001 and voyage-code, and the Jina-VDR margin is vulnerable to train/test overlap because the benchmark reuses datasets named in the training section. The paper's own Section 7.1 language ('broadly comparable') is more accurate than the abstract.","major_comments":[{"comment":"The abstract's unqualified claim that jina-embeddings-v4 'achieves state-of-the-art performance on both single-modal and cross-modal retrieval tasks' is contradicted by Table 3. On MMTEB the model scores 66.49 versus gemini-embedding-001's 67.71; on MTEB-en it scores 55.97 versus 64.35; on COIR it scores 71.59 versus voyage-code-3's 77.33 and gemini's 73.11; on LEMB it scores 67.11 versus 78.35. The model is best-in-class only on STS-en and on the visual benchmarks. Please revise the abstract and Section 7.1 to say 'competitive' or 'state-of-the-art on visually rich retrieval' rather than a general SOTA claim.","section":"Abstract; Table 3"},{"comment":"The central visual-retrieval claim (Section 7.3: 'provides the current state-of-the-art, in both single- and multi-vector mode' on Jina-VDR) is not interpretable without a contamination analysis. Section 5.2.1 states that multimodal hard negatives were mined from 'VDR multilingual' and other curated multimodal datasets, while Section 6.1 lists re-purposed datasets (ChartQA, PlotQA, CharXiv, TableVQA, MPMQA, etc.) that appear as Jina-VDR tasks in Table A1. The paper reports no overlap statistics between training data and Jina-VDR test tasks, no evaluation code, and no variance or significance estimates. If the same documents appear on both sides of the training/test boundary, the margins in Table 3 (e.g., dense 73.98 vs. 67.25 for dse-qwen2-2b-mrl-v1) could partially reflect memorization. Please provide a per-task overlap analysis and report results with and without overlapping tasks, or restrict the SOTA claim to tasks verified to be disjoint.","section":"Section 5.2.1; Section 6.1; Table A1; Table 3"},{"comment":"Section 6.3 states that many Jina-VDR queries were generated with Qwen2 and filtered by LLMs, while jina-embeddings-v4 is built on Qwen2.5-VL. Because the query distribution may be systematically closer to the model's prior, the benchmark may overstate the proposed model's advantage over baselines. Please report a sensitivity analysis that excludes LLM-generated/synthetic tasks, or at least report per-task results separately for human-annotated, re-purposed, and synthetic subsets.","section":"Section 6.3"},{"comment":"Several Jina-VDR tasks are very small (e.g., ramen_benchmark_jp has 29 queries, shanghai_master_plan has 57, jina_2024_yearly_book has 75, student-enrollment has 1000 but near-random scores), so the averaged nDCG@5 numbers in Table A2 and Table 3 are unstable. Reporting no confidence intervals or significance tests makes it difficult to judge whether the 73.98 vs. 67.25 dense margin is real. Please add bootstrap confidence intervals or per-task significance tests, and consider weighting by task size in the average.","section":"Table A2 and Table A1"}],"minor_comments":[{"comment":"The model name 'jina-embedings-v2-code' is a typo for 'jina-embeddings-v2-code'; please fix it.","section":"Table 3"},{"comment":"The sentence 'These tests include re-purposed existing datasets, new manually-annotated datasets, and generated synthetic data' is missing a period at the end.","section":"Section 6"},{"comment":"The cross-modal alignment score definition should state whether the cosine similarities are averaged over the 1K samples per dataset or over all pairs; Table 4 would benefit from a standard deviation.","section":"Section 8.2"},{"comment":"The notation in Equation (2) says qi,pj∈B, but B is a batch of pairs; please clarify that the division is by the number of query tokens t of qi.","section":"Section 5.1"},{"comment":"Some dataset names in Table A1 (e.g., jinaai/ChartQA, jinaai/CharXiv-en) appear to be re-hosted versions of existing datasets; please state explicitly whether the splits differ from the originals.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's self-citations are heavy, and introducing a new benchmark in the same paper that evaluates on it creates a conflict-of-interest concern; however, external benchmarks (ViDoRe, CLIP, MTEB) provide independent ground. The abstract overclaims text SOTA; the authors should be asked to align claims with their own tables. The Jina-VDR contamination question is the load-bearing issue, and an independent overlap check is feasible because the benchmark and model are released."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real systems paper with a genuinely useful model and a load-bearing evaluation problem. The external-benchmark results are strong enough to justify the work, but the paper's own Jina-VDR benchmark cannot support the SOTA claim it is asked to carry without a contamination check.\n\nThe genuinely new pieces: a 3.8B unified encoder on Qwen2.5-VL that does text, image, and code retrieval; single- and multi-vector outputs via Matryoshka and late interaction; three LoRA adapters; and a new benchmark, Jina-VDR. The model and benchmark are released. That is a concrete engineering contribution, and the evaluation breadth is a real strength: ViDoRe, CLIP Benchmark, MMTEB, MTEB, COIR, LongEmbed, plus the modality-gap and cone-effect analysis. On external benchmarks the model is competitive and often best-in-class, especially on ViDoRe (84.11 dense, 90.17 late) and on code retrieval. Credit where due.\n\nThe soft spot is exactly the one the stress-test note names. Section 5.2.1 says the retrieval adapter was trained on 'VDR multilingual' and 'curated multimodal datasets.' Section 6.4 says Jina-VDR extends ViDoRe and re-purposes existing datasets like ChartQA, PlotQA, CharXiv, and TableVQA. Those same document families plausibly appear in training. The paper reports no overlap statistics, no evaluation code, and no variance or significance numbers. Without those, the Jina-VDR margins (73.98 vs. 67.25 dense; 80.55 vs. 63.80 late) are not interpretable as retrieval-quality differences. This is not a minor footnote; it is the paper's central SOTA claim.\n\nSecond, the abstract says state-of-the-art on single- and cross-modal retrieval, but the paper's own tables show gemini-embedding-001 ahead on MMTEB and voyage-code ahead on code. That is an overclaim by the paper's own data, easy to fix, should be fixed.\n\nVerdict: the model is worth engaging and the paper deserves a serious referee, but only with a contamination analysis or a narrowed claim. If Jina-VDR is cleaned or the claims are re-scoped to ViDoRe and external benchmarks, this is a solid accepted paper. If not, the headline result is unsupported.","headline":"Solid systems contribution with a real benchmark-confounding problem: the headline Jina-VDR SOTA claim is not interpretable without a contamination check, but the model and external-benchmark results are worth engaging.","tokens_in":26083,"tokens_out":2154,"would_cite":true,"duration_ms":20744,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single 3.8-billion-parameter multimodal model, trained once with one joint loss, gives state-of-the-art retrieval for text, code, and visually rich documents in both single-vector and multi-vector output modes.","keywords":["jina-embeddings-v4","multimodal embeddings","visual document retrieval","late interaction","multilingual retrieval","code retrieval","LoRA adapters","cross-modal retrieval"],"falsifier":"Build an independent visual-document retrieval benchmark from documents and human queries created after the model's training data cutoff, with no author involvement, and rerun the reported comparisons; if the margin over the strongest prior visual retrievers narrows or reverses, the state-of-the-art claim is an artifact of benchmark overlap or synthetic-query style.","tokens_in":24994,"feed_emoji":"🔍","tokens_out":10110,"duration_ms":93802,"temperature":0.7,"pith_summary":"jina-embeddings-v4 is presented as a single model that can replace the usual stack of a text embedder, an image embedder, and an OCR-plus-retrieval pipeline: one 3.8-billion-parameter vision-language backbone maps text and images into one semantic space, and three small task-specific adapters tailor it to retrieval, semantic similarity, and code search. The core claim is that this one model is the current state of the art for retrieving visually rich documents—screenshots, tables, charts, maps, and mixed-media pages—while staying competitive with specialized text and code embedders. To demonstrate this, the paper introduces Jina-VDR, a multilingual benchmark of more than thirty visual-retrieval tasks, and reports the highest scores on it and on the existing ViDoRe benchmark, with further gains when using multi-vector and late-interaction embeddings. If the claim holds, the practical consequence is that one model, one index, and one embedding space can serve text retrieval, cross-modal search, and visual document search without per-task pipelines.","feed_headline":"One 3.8B embedding model claims best visual-document retrieval","feed_subtitle":"One shared backbone searches tables, charts, and screenshots directly, plus text and code, without OCR.","key_machinery":"The load-bearing object is a unified multimodal language model: images are converted by a vision encoder into a sequence of tokens that the language-model decoder processes exactly like text tokens, so both modalities share contextual attention and the same embedding space. On top are three mechanisms: mean pooling that produces 2048-dimensional single-vector embeddings, trained with a Matryoshka-style loss so they can be truncated to 128 dimensions; a projection layer that produces 128-dimensional per-token multi-vector embeddings, scored by late interaction, that is, summing over query tokens their maximum cosine similarity to any document token; and three 60-million-parameter LoRA adapters for asymmetric retrieval, symmetric text matching, and code search, chosen at inference time. The joint training loss combines InfoNCE contrastive losses on dense and late-interaction similarity matrices plus a KL term that distills late-interaction scores into dense scores, so a single training run teaches both output modes.","core_discovery":"The central discovery is that the modality gap of dual-encoder models is not inevitable: if images are first turned into token sequences by a vision encoder and then passed through the same transformer decoder as text, the model uses a shared embedding region for both modalities. Trained with a joint contrastive loss over dense and late-interaction similarity matrices, the model reaches cross-modal alignment scores around 0.71–0.72 on standard image-text datasets, roughly double a dual-encoder baseline, and reports 84.11 (dense) and 90.17 (late interaction) average nDCG on ViDoRe, and 73.98 and 80.55 on Jina-VDR, the highest scores in the comparison. The paper states that this makes jina-embeddings-v4 the current state of the art, in both single-vector and multi-vector mode, for visually rich document retrieval.","pith_inferences":["Editorial inference: the benchmark's own LLM-generated and template-based queries may understate the difficulty of real user queries; the natural follow-up is a third-party held-out set of human-typed queries on the same documents, which could be built without retraining the model.","Editorial inference: because dense scores are distilled from late-interaction scores, the single-vector mode likely inherits late-interaction's sensitivity to query phrasing; one can test whether truncating dense vectors to 128 dimensions preserves late-interaction rankings better than it preserves dense cosine rankings.","Editorial inference: the recipe is not modality-specific; audio, video, or 3D data could in principle be tokenized into the same shared decoder and trained with the same dense-plus-late joint loss, suggesting a path to one universal embedder rather than per-modality models.","Editorial inference: if the shared-encoder design is what shrinks the modality gap, then improving the backbone's multilingual text coverage should transfer directly to cross-modal retrieval in low-resource languages; measuring image-to-text retrieval in languages absent from the current evaluation would separate backbone coverage from alignment quality."],"forward_implications":["Visual document search becomes a single forward pass on the rendered image: no OCR, no layout parsing, and no separate text-extraction pipeline.","One frozen backbone with small adapters covers asymmetric retrieval, symmetric similarity, and code search; adding a new task or domain means training a 60-million-parameter adapter, not a 3.8-billion-parameter model.","Multi-vector and late-interaction mode is consistently more accurate than single-vector on visual documents (80.55 vs 73.98 average on Jina-VDR), so applications can trade storage and computation for precision by switching output modes.","Because text, image, and code embeddings live in the same space, a single index can serve text-to-text, text-to-image, and image-to-text queries without realignment or re-ranking across separate spaces.","If Jina-VDR is adopted, visual retrieval evaluation shifts from English and French question answering to multilingual, non-question queries over maps, diagrams, advertisements, and mixed-media pages."],"supporting_citations":[{"why":"Supplies the ViDoRe benchmark and the ColPali late-interaction visual retriever that jina-embeddings-v4 must outperform, and defines the late-interaction scoring used in comparisons.","marker":"[Faysse et al., 2025]"},{"why":"Provides the task-LoRA training recipe and the jina-embeddings-v3 baseline that the new model extends and compares against on text benchmarks.","marker":"[Sturua et al., 2024]"},{"why":"Provides the 3.8-billion-parameter vision-language backbone whose shared text-image processing path is the architectural foundation of the model.","marker":"[Bai et al., 2025]"},{"why":"Introduces ColBERT and late-interaction and multi-vector similarity, the mechanism the multi-vector output mode reproduces.","marker":"[Khattab and Zaharia, 2020]"},{"why":"Supplies the InfoNCE contrastive loss that drives both pair training and task-specific training on text and text-image pairs.","marker":"[van den Oord et al., 2018]"},{"why":"Supplies Matryoshka Representation Learning, which makes the single-vector embeddings truncatable from 2048 to 128 dimensions with small accuracy loss.","marker":"[Kusupati et al., 2022]"},{"why":"Supplies LoRA, the parameter-efficient adapter mechanism that lets one frozen backbone specialize into three task models.","marker":"[Hu et al., 2022]"},{"why":"Supplies Wiki-SS, one of the multimodal datasets used to mine hard negatives for the asymmetric retrieval adapter.","marker":"[Ma et al., 2024]"}],"fun_headline_variants":["Shared backbone beats dual encoders on visual documents","No OCR needed: 3.8B model retrieves images and text directly","Multimodal embeddings: one model, no modality gap","Vision tokens join text in one decoder for retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The visual-retrieval state-of-the-art claim stands on the assumption that Jina-VDR is a fair, uncontaminated benchmark, even though the same authors built it and trained the retrieval adapter on some of the same or closely related multimodal data, including VDR multilingual and other curated hard negatives.","fun_headline_variants_meta":{"raw":{"variants":["Shared backbone beats dual encoders on visual documents","No OCR needed: 3.8B model retrieves images and text directly","Multimodal embeddings: one model, no modality gap","Vision tokens join text in one decoder for retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3224,"prompt_tokens":866,"completion_tokens":2358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":2290}},"tokens_in":482,"tokens_out":2358,"duration_ms":15931,"temperature":1.0,"reasoning_tokens":2290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:40:33.573871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an independent visual-document retrieval benchmark from documents and human queries created after the model's training data cutoff, with no author involvement, and rerun the reported comparisons; if the margin over the strongest prior visual retrievers narrows or reverses, the state-of-the-art claim is an artifact of benchmark overlap or synthetic-query style.","supporting_citations":[],"review_version":2}