{"id":"1e0713fb-f80f-40e2-82a8-9d83779e4a9a","arxiv_id":"2509.08897","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single multimodal retriever with a gated recurrent Transformer cell, one-token output, and three-layer sampling achieves strong results on M2KR and M-BEIR and improves RAG-VQA accuracy.","lead":"This paper presents ReT-2, a retrieval model that can search across collections where both queries and documents contain images and text, using a recurrent Transformer with gating to blend visual and textual features from multiple layers. It reports top results on the M2KR and M-BEIR benchmarks and faster retrieval than several prior systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"M2KR SOTA claim rests on potentially unfair baseline protocol: PreFLMR/FLMR rows in Tables III/IV are not marked as re-run on ReT-2's document-image-augmented splits.","rationale":"The reader's conditional verdict hinges on the same load-bearing concern I identify: the M2KR comparison is unfair if PreFLMR/FLMR numbers come from text-only-document splits while ReT-2 uses document images. This is not a minor detail; the augmentation is described as aligning the benchmark with the paper's multimodal setting, and ReT-2 is trained on the augmented data. The absence of a † marker on the key baseline rows is strong evidence they were not re-run. My concrete test would settle this directly. A secondary concern is that the layer-selection analysis in Section V-C uses the M2KR test splits to choose layer indices, which is a form of test-set leakage; however, that affects the efficiency/robustness narrative more than the core SOTA claim. The M-BEIR comparison is fairer (UniIR was reproduced with SigLIP2), and the RAG experiments are separate. Thus the verdict should remain conditional: accept only after the baseline protocol is checked.","tokens_in":30270,"tokens_out":3180,"duration_ms":36078,"concrete_test":"Using the released ReT-2 code and augmented M2KR splits, re-run PreFLMR with CLIP ViT-L (the exact config from Table III) on the augmented document pool and evaluate on the same test queries. If PreFLMR's average R@K rises from the reported 57.4 to roughly match ReT-2's 64.1, the SOTA claim is an artifact of the protocol mismatch. A complementary check: evaluate ReT-2 on the original text-only-document M2KR splits; if its average drops substantially below 64.1, the extra visual signal is the source of the gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim on M2KR depends on comparing ReT-2 against PreFLMR and FLMR under the same data protocol. Section V-A states that ReT-2 augments the OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA splits by attaching images to reference documents. However, in Table III and Table IV, the PreFLMR and FLMR rows without a dagger (†) are cited from the original papers, which used text-only documents. The dagger is used only for the authors' own SigLIP2 reproductions and other re-runs. This means ReT-2 is trained and evaluated with an extra visual signal on those document sets, while the baselines are not. The gap in Table III is large (e.g., ReT-2 CLIP ViT-L unfrozen 67.9 vs. PreFLMR 57.4), so the claimed improvement could be explained by the protocol difference rather than by the architecture. The reader's weakest assumption is therefore correct: without re-running the baselines on the augmented splits, the M2KR SOTA claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReT-2, a unified multimodal retrieval model that extends the authors' earlier ReT architecture. ReT-2 shares query/document encoder weights, uses a single fused token instead of 32 tokens, samples only three backbone layers (early, middle, late) for the recurrent fusion cell, and injects global pooler features. The authors evaluate on M2KR (with document-image-augmented splits for four datasets) and M-BEIR local, and additionally test the retriever in retrieval-augmented VQA with off-the-shelf MLLMs. They report consistent gains over ReT, PreFLMR, FLMR, and UniIR, along with faster inference than late-interaction baselines. Ablations in Table II support the main architectural choices, and code and models are released.","tokens_in":30568,"tokens_out":10461,"duration_ms":119036,"significance":"If the claims hold, ReT-2 is a useful, efficient alternative to late-interaction multimodal retrievers and to much larger MLLM-based retrievers, achieving near-SOTA on M-BEIR and strong RAG-VQA results at a fraction of the inference cost. The paper's strengths are its systematic ablations, the reproduced UniIR baseline on M-BEIR, the efficiency analysis, and the public release of code and models. However, the main M2KR state-of-the-art claim currently rests on an unfair protocol comparison: the PreFLMR/FLMR rows in Tables III and IV are cited from the original text-only-document benchmark while ReT-2 is evaluated on the document-image-augmented splits described in Section V-A. In addition, the method's handling of missing modalities, which is central to the 'universal' claim and to several M-BEIR tasks, is not specified, and the layer-pruning decision is based on test-split activations. These issues are fixable but require substantial revision.","major_comments":[{"comment":"The M2KR SOTA comparison is not apples-to-apples. Section V-A states that ReT-2 augments the OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA splits by attaching images to reference documents. However, the PreFLMR and FLMR rows in Tables III and IV that lack the † marker are not re-run under this protocol; the dagger is reserved for the authors' own reproductions. Thus ReT-2 sees an extra visual signal on four of the eight M2KR datasets, while the cited baseline numbers come from the original text-only-document benchmark. The large gaps (e.g., 67.9 vs. 57.4 for CLIP ViT-L in Table III) could be partly due to this protocol difference. The authors must either re-run PreFLMR/FLMR on the augmented splits or clearly separate the augmented-split comparison from the original-benchmark comparison before claiming state-of-the-art on M2KR.","section":"Section V-A, Tables III and IV"},{"comment":"The paper claims that ReT-2 'can also handle scenarios with missing modalities' and evaluates on M-BEIR tasks that are unimodal on one side (e.g., q_T→d_V, q_T→(d_V,d_T)). However, the method description defines the recurrent cell, gating, and global feature injection always in terms of both a visual representation E_V and a textual representation E_T. There is no description of how a missing image or missing text is represented (zero vector, masked cross-attention, separate token, or something else). This is a reproducibility gap for a central claim of the paper: without this detail, the M-BEIR results in Table V cannot be reconstructed from the text. Please specify the missing-modality protocol explicitly.","section":"Section IV-A, Eqs. (4)-(9); Section V-A"},{"comment":"The layer-pruning selection uses test-split information. In the Layer Pruning subsection, the authors state that the choice of three representative layers is 'further supported by an empirical analysis of the average gate activations of ReT, conducted on the InfoSeek and Encyclopedic-VQA test splits of M2KR.' The selected layer indices are then used as a fixed component of ReT-2 and evaluated on the same test splits. This constitutes tuning a hyperparameter on the test set, which can inflate the reported numbers. The selection should be made on validation splits, or the authors should argue that the layer choice is architecture-agnostic and not fitted to the test data.","section":"Section V-C, Fig. 5"}],"minor_comments":[{"comment":"The abstract claims 'consistently achieves state-of-the-art performance across diverse settings,' but on M-BEIR local, LamRA achieves an average of 56.6 vs. ReT-2 SigLIP2's 56.4 (Table V). The paper acknowledges this in Section V-D, so the abstract should be qualified (e.g., 'state-of-the-art among non-MLLM methods').","section":"Abstract and Table V"},{"comment":"The dimension notation for W_final is inconsistent. Eq. (8) writes W_final ∈ R^{d×d}, but Section V-B says its dimension is 'set to match d with the dimension of the global features.' Since the recurrent hidden size (1,024 according to Section V-B) can differ from the backbone pooler size (e.g., 768 for CLIP ViT-L), Eq. (9)'s addition h_L + E_V + E_T is dimensionally unclear. Please clarify whether W_final projects to the global feature dimension.","section":"Section IV-A, Eqs. (8)-(9)"},{"comment":"The row label 'PreFLMR [2]' should be 'PreFLMR [18]'; [2] is the authors' own ReT paper. The same typo appears in the qualitative results section.","section":"Table VII"},{"comment":"Minor language issue: 'enabling a more effectively evaluation' should be 'enabling a more effective evaluation.'","section":"Section V-A"},{"comment":"The column headers ('Training Info', 'Inference Time (ms)', 'All↓', '#Tokens') are hard to parse. Please use clearer subheadings (e.g., 'GPUs', 'Hours', 'Forward (ms)', 'Retrieval (ms)', 'Total (ms)').","section":"Table VI"}],"recommendation":"major_revision","confidential_remarks":"The protocol mismatch in Tables III and IV is the main substantive issue: the M2KR SOTA claim is not currently established. I would expect the authors to re-run the baseline models on the augmented splits or to restructure the paper so that the augmented-split results are clearly separated from the original M2KR benchmark numbers. The missing-modality specification and the test-split-based layer selection also need to be addressed. If these points are resolved, the paper would likely be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ReT-2 is a genuine improvement over the authors' earlier ReT, with clean ablations and a useful efficiency story. But the paper's central SOTA claim on M2KR has a fairness problem that needs fixing before it's citable.\n\nWhat's genuinely new: collapsing the 32-token matrix output to a single token, always sampling three backbone layers (early/mid/late), injecting global pooler features, and sharing query/document encoder weights. The rank-collapse analysis motivating token reduction is a nice piece of evidence; the ablation table walks through each change and the story holds. The M-BEIR comparison against a reproduced UniIR baseline (SigLIP2) is the right way to do it, and ReT-2 does beat it convincingly. Efficiency numbers are believable: one token, three layers, no late interaction.\n\nThe soft spot is the M2KR SOTA claim. Section V-A says they augmented OVEN, InfoSeek, E-VQA, and OKVQA documents with images, which is fine for their own model. But the PreFLMR and FLMR rows in Tables III and IV without the dagger are the original numbers, trained and evaluated on text-only documents. So ReT-2 gets an extra visual signal on those splits that the baselines didn't have. The gap on InfoSeek (67.9 vs 57.4 with unfrozen CLIP ViT-L) is large enough that the protocol difference, not the architecture, could be doing the work. This doesn't sink the paper—the M-BEIR results and the ablations stand on their own—but the abstract's 'consistently achieves state-of-the-art' line overreaches until the baselines are re-run on the augmented protocol.\n\nTwo smaller things: the layer-selection analysis (Fig. 5) uses the test split of InfoSeek/E-VQA to pick layers, so the chosen indices are in-sample. It's a minor leak since the choice is coarse (three fixed groups), but worth declaring. And the RAG-VQA numbers are real but should be read with the knowledge base sizes in mind—ReT-2 uses a 15.9M-doc index where others used 2M.\n\nBottom line: this is worth a serious referee. The design is sensible, the evaluation is mostly careful, and code/models are out. I'd send it to review, but the M2KR baseline protocol needs to be fixed or the SOTA claim softened. I'd cite the M-BEIR result in the near term, not the M2KR comparison.","headline":"Solid architectural extension of ReT, but the headline M2KR SOTA claim is not yet supported because the PreFLMR/FLMR baselines were not re-run on the image-augmented splits.","tokens_in":31071,"tokens_out":1758,"would_cite":true,"duration_ms":19757,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReT-2 claims that one recurrent fused token can represent multimodal queries and documents, setting state-of-the-art results on M2KR and M-BEIR at lower cost.","keywords":["universal multimodal retrieval","recurrent Transformer","LSTM gating","multi-layer feature fusion","single-token embedding","M2KR benchmark","M-BEIR benchmark","retrieval-augmented generation"],"falsifier":"Re-run PreFLMR and FLMR on the image-augmented M2KR splits for OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA (i.e., provide the same document images to their document encoders or mark them as text-only), and compare with ReT-2; if either baseline surpasses ReT-2 under identical document inputs, the claimed M2KR state-of-the-art does not stand. Also, evaluating ReT-2 on the M-BEIR global setting would settle whether the single-token design truly generalizes to the full 5.6M-document pool.","tokens_in":30146,"feed_emoji":"🔎","tokens_out":6600,"duration_ms":62901,"temperature":0.7,"pith_summary":"This paper tries to establish that universal multimodal retrieval—matching queries and documents that mix images and text—can be done with a compact, single-token embedding produced by a recurrence-augmented Transformer. ReT-2 fuses features from early, middle, and late layers of a vision and a text encoder using LSTM-style gates, then adds global backbone features to the final token. If correct, it would show that heavy late-interaction machinery and massive MLLM-based retrievers are unnecessary for state-of-the-art multimodal search, and that retrieval quality directly improves downstream question answering. The paper backs this with results on M2KR and M-BEIR local across eight task configurations, plus RAG-VQA gains with off-the-shelf MLLMs.","feed_headline":"Single fused token beats bigger multimodal retrievers","feed_subtitle":"LSTM-style gates fuse image and text layers, cutting memory and lifting RAG-VQA accuracy.","key_machinery":"The load-bearing component is the recurrent fusion cell: a Transformer block whose hidden state evolves over backbone layers under three gates (forget, text-input, vision-input), with cross-attention into the normalized state from layer-specific visual and textual features. The final state is a single k×d token (k=1) that sums global pooler features from both backbones; this token replaces the 32-token late-interaction matrix of ReT and enables a plain dot-product InfoNCE loss.","core_discovery":"ReT-2 is a unified retrieval model that encodes a multimodal query or document as a single token by running a recurrent Transformer cell over three sampled layers of frozen (or fine-tuned) visual and textual backbones. The cell applies an LSTM-inspired forget gate and separate input gates for text and vision, so the model can decide how much of each modality and each layer to keep. The paper reports that this design—with shared query/document weights, layer pruning, and global feature injection—outperforms the previous 32-token ReT and existing baselines on M2KR and M-BEIR, while reducing inference time and memory. It also shows that ReT-2 as a retrieval backbone improves accuracy of LLaVA-M","pith_inferences":["If the single-token result holds beyond these benchmarks, the 32-token late-interaction design shared by several retrievers may be over-parameterized; testing ReT-2 against a rank-regularized variant of ReT would isolate whether the gain comes from token compression or from the recurrent gating itself.","The gate activations shown in Figure 5 provide a per-layer, per-modality relevance profile; one could use those statistics as a cheap diagnostic to predict which datasets benefit most from visual versus textual context, or to guide automatic layer selection for new backbones.","The paper evaluates only the local M-BEIR setting; extending ReT-2 to the global 5.6M-document pool would test whether the single fused token retains discriminative power at billion-scale candidate sets, where late-interaction methods often shine or fail."],"forward_implications":["Universal multimodal retrieval is achievable without MLLM-scale models: ReT-2's best variant (SigLIP2 ViT-L backbone) matches or exceeds several 7B-parameter MLLM-based retrievers on M-BEIR local while using far fewer parameters and half the inference time.","The single-token representation eliminates the need for fine-grained late-interaction scoring (the max-over-tokens operator), replacing it with a standard dot-product and cutting memory and latency.","Layer pruning to three representative layers (early, middle, late) is sufficient: it preserves accuracy on M2KR and improves average score by +0.1 over using all layers, and enables the same recurrent cell to work across backbones of different depths.","Retrieval quality is a decisive factor in knowledge-intensive VQA: using ReT-2 to retrieve top-3 documents raises the accuracy of off-the-shelf MLLMs on Encyclopedic-VQA and InfoSeek compared to using CLIP/SigLIP2, UniIR, ReT, or PreFLMR as the retriever."],"fun_headline_variants":["One token to retrieve them all: LSTM gates fuse text and image","Recurrent Transformer compresses multimodal query into one token","LSTM-style gating makes multimodal retrieval faster and lighter","Single fused token beats 32-token retrievers on M2KR and M-BEIR","ReT-2: one token for all multimodal retrieval, faster than ever"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The M2KR state-of-the-art comparison assumes that the PreFLMR and FLMR numbers cited in Tables III and IV were produced under the same document-image-augmented splits used for ReT-2 (Section V-A augments OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA with document images), but those baseline rows carry no dagger marking as re-runs, so if they come from the original text-only-document benchmark, ReT-2 may be receiving extra visual evidence the baselines did not get.","fun_headline_variants_meta":{"raw":{"variants":["One token to retrieve them all: LSTM gates fuse text and image","Recurrent Transformer compresses multimodal query into one token","LSTM-style gating makes multimodal retrieval faster and lighter","Single fused token beats 32-token retrievers on M2KR and M-BEIR","ReT-2: one token for all multimodal retrieval, faster than ever"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":3869,"prompt_tokens":737,"completion_tokens":3132,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":3037}},"tokens_in":481,"tokens_out":3132,"duration_ms":25499,"temperature":1.0,"reasoning_tokens":3037,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:01:25.429665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run PreFLMR and FLMR on the image-augmented M2KR splits for OVEN, InfoSeek, Encyclopedic-VQA, and OKVQA (i.e., provide the same document images to their document encoders or mark them as text-only), and compare with ReT-2; if either baseline surpasses ReT-2 under identical document inputs, the claimed M2KR state-of-the-art does not stand. Also, evaluating ReT-2 on the M-BEIR global setting would settle whether the single-token design truly generalizes to the full 5.6M-document pool.","supporting_citations":[],"review_version":1}