{"id":"174a0af0-d432-4fbd-a25b-b50a4577ff73","arxiv_id":"2606.19960","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Stellar reduces memory and latency by 1-2 orders of magnitude for multimodal document retrieval via lexical filtering with a fine-tuned MLLM and disk-backed late interaction without loss in effectiveness.","lead":"Stellar is a retrieval framework that stores most token-level embeddings on disk and loads only a small candidate set into memory for late interaction in multimodal document search. This targets the memory bottleneck in current RAG systems and could make large-scale multimodal retrieval practical on standard hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"LRF recall preservation is the load-bearing assumption; no other internal inconsistency found in the described architecture.","rationale":"Reader correctly isolated the filtering assumption as weakest; full-text description of LRF (fine-tuning MLLM as sparse encoder + balanced clustering for DLI) does not introduce additional soundness problems or hidden assumptions beyond that one. The concrete_test directly measures whether the assumption lands.","tokens_in":1727,"tokens_out":320,"duration_ms":10760,"concrete_test":"On the newly introduced large-scale dataset, compute LRF recall@1000 (fraction of documents that would have been ranked in top-1000 by full late interaction that survive the lexical filter) and compare against the end-to-end nDCG@10; if the gap exceeds 3 points on any of the four benchmarks, the 1-2 order-of-magnitude claim no longer holds without effectiveness loss.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim requires that the fine-tuned MLLM sparse encoder in LRF produces a candidate set whose recall is high enough that the subsequent DLI stage (late interaction on the reduced set) matches full-corpus late-interaction effectiveness. Section 3.2 and the experimental claims rest on this; if lexical representations systematically drop relevant multimodal documents (e.g., when visual content dominates relevance), effectiveness is compromised even if memory/latency numbers look good. The paper reports end-to-end metrics but the critical intermediate recall@K for LRF alone is the untested link.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Stellar, a multimodal document retrieval framework for natural language queries that combines Lexical Representation-based Filtering (LRF) — fine-tuning an MLLM as a sparse encoder for candidate reduction — with Efficient Disk-backed Late Interaction (DLI) that stores token embeddings on disk and loads only necessary candidates via a balanced clustering layout and cost model. It claims 1-2 orders of magnitude reductions in memory and query latency versus prior late-interaction methods on four benchmarks plus a new large-scale dataset, while preserving retrieval effectiveness.","tokens_in":1828,"tokens_out":511,"duration_ms":8847,"significance":"If the central effectiveness claim holds, Stellar would materially improve the practicality of token-level late-interaction retrieval in RAG pipelines by addressing the memory and latency bottlenecks that currently limit deployment at scale. The introduction of a new large-scale multimodal dataset is a concrete contribution that could support future work.","major_comments":[{"comment":"§3.2 (LRF) and §4 (experiments): the headline claim that effectiveness is uncompromised rests on the assumption that LRF produces a candidate set with sufficiently high recall that the subsequent DLI stage matches full-corpus late-interaction performance. The manuscript reports only end-to-end metrics; the critical intermediate recall@K (or equivalent) for the LRF sparse-encoder stage alone is not provided, leaving the load-bearing assumption unverified.","section":"§3.2 and §4"},{"comment":"§4 (experimental setup): the abstract and results claim reductions “without compromising retrieval effectiveness,” yet no details are given on the precise baselines, statistical significance testing, or whether any post-hoc threshold tuning was performed on the LRF candidate-set size; this information is required to assess whether the reported effectiveness numbers are robust.","section":"§4"}],"minor_comments":[{"comment":"The abstract states positive outcomes on “four real-world benchmarks and a newly presented large-scale dataset” but provides no metric names, dataset sizes, or baseline references; adding one sentence with these specifics would improve readability.","section":"Abstract"},{"comment":"Notation for the cost model in DLI is introduced without an explicit equation number; cross-referencing the model to a numbered equation would aid reproducibility.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback on our manuscript. We address each of the major comments below and outline the revisions we will make.","responses":[{"response":"We agree that the intermediate recall of the LRF stage is important to report for full transparency. While the end-to-end results demonstrate that Stellar achieves comparable effectiveness to full late-interaction methods, adding the LRF recall metrics will better support our claims. We will include these results in the revised manuscript, showing high recall at the chosen candidate sizes.","revision_made":"yes","referee_comment":"[§3.2 and §4] §3.2 (LRF) and §4 (experiments): the headline claim that effectiveness is uncompromised rests on the assumption that LRF produces a candidate set with sufficiently high recall that the subsequent DLI stage matches full-corpus late-interaction performance. The manuscript reports only end-to-end metrics; the critical intermediate recall@K (or equivalent) for the LRF sparse-encoder stage alone is not provided, leaving the load-bearing assumption unverified."},{"response":"We will provide additional details in the experimental setup section, including the exact list of baselines compared, results of statistical significance tests (e.g., p-values from t-tests), and clarification that the candidate set size was selected based on validation data without post-hoc tuning on the test sets. This will address concerns about robustness.","revision_made":"yes","referee_comment":"[§4] §4 (experimental setup): the abstract and results claim reductions “without compromising retrieval effectiveness,” yet no details are given on the precise baselines, statistical significance testing, or whether any post-hoc threshold tuning was performed on the LRF candidate-set size; this information is required to assess whether the reported effectiveness numbers are robust."}],"tokens_in":1418,"tokens_out":363,"duration_ms":28985,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Stellar tackles the memory cost of multi-vector late interaction for multimodal documents by first using a fine-tuned MLLM as a sparse encoder to filter down to a small candidate set, then loading only the relevant token embeddings from disk via a balanced clustering layout. The two pieces are LRF for the filtering and DLI for the storage and loading.\n\nThe concrete combination is the main new element here. Prior work has used late interaction, disk storage, and sparse filtering separately; putting an MLLM-based lexical filter in front of disk-backed late interaction for natural-language multimodal queries looks like a targeted engineering step. The abstract reports 1-2 order-of-magnitude drops in memory and query latency across four benchmarks plus a new large-scale dataset, with no drop in end-to-end effectiveness.\n\nThat addresses a real deployment pain point for RAG systems. The approach is straightforward and the claimed gains are large enough to matter if they hold.\n\nThe load-bearing assumption is that LRF keeps recall high enough that the reduced set still contains the relevant documents. If the sparse lexical representations miss cases where visual content drives relevance, the later late-interaction stage never gets a chance. The abstract gives end-to-end numbers but does not mention intermediate recall@K for LRF alone, so it is hard to tell whether the filtering step is doing the job or whether the gains come partly from easier queries. The new dataset also needs scrutiny on construction and baseline selection.\n\nThe paper is for engineers building large multimodal RAG pipelines who need lower hardware costs. A reader in that position would find the reported trade-offs useful. It is coherent on its own terms and deserves a serious referee to examine the filtering metrics and experimental details.","headline":"Stellar pairs MLLM sparse lexical filtering with clustered disk storage for late interaction to cut memory and latency in multimodal retrieval, but the filtering recall is the untested link that needs checking.","tokens_in":2318,"tokens_out":431,"would_cite":false,"duration_ms":17765,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Stellar reduces memory overhead and query latency for multimodal document retrieval by 1-2 orders of magnitude without losing effectiveness.","keywords":["multimodal document retrieval","late interaction","lexical filtering","disk-backed storage","scalable retrieval","MLLM","RAG systems","token embeddings"],"falsifier":"Running the LRF step on a benchmark and finding that for some queries the gold relevant document is not in the filtered candidate set.","tokens_in":2636,"feed_emoji":"💾","tokens_out":503,"duration_ms":17079,"temperature":0.7,"pith_summary":"The paper shows how to make multi-vector retrieval scalable by keeping most token embeddings on disk and using a fine-tuned multimodal model to filter down to a small candidate set before late interaction. This addresses the high memory cost that has limited deployment of accurate retrieval methods in RAG systems. If the approach works, large-scale multimodal search becomes feasible on ordinary hardware while preserving the quality of current state-of-the-art methods. Experiments on four benchmarks plus a new large dataset back the claim.","feed_headline":"Stellar slashes retrieval memory and latency by 10-100x","feed_subtitle":"Disk storage plus MLLM lexical filtering keeps effectiveness while making large-scale multimodal search practical.","key_machinery":"Lexical Representation-based Filtering (LRF) using a fine-tuned MLLM as sparse encoder combined with Efficient Disk-backed Late Interaction (DLI) using balanced clustering for on-disk storage.","core_discovery":"Stellar introduces Lexical Representation-based Filtering that fine-tunes an MLLM as a sparse encoder for effective document pruning, paired with Efficient Disk-backed Late Interaction that uses balanced clustering to lay out token embeddings on disk and a cost model to load only needed ones into memory for late interaction.","pith_inferences":["Success would shift design focus from in-memory indexes to hybrid disk-memory architectures for retrieval.","Further gains might come from optimizing the clustering or cost model for specific hardware.","Testing on even larger synthetic datasets could reveal where the filtering step starts to drop recall."],"forward_implications":["Multimodal document retrieval can now operate on corpora much larger than current memory limits allow.","Production RAG systems can adopt late-interaction methods without requiring specialized high-RAM servers.","Query response times improve enough to support interactive applications at scale.","The same framework may extend to other embedding-based retrieval tasks that suffer from multi-vector memory costs."],"fun_headline_variants":["Stellar stores token embeddings on disk for scalable retrieval","MLLM lexical filtering reduces candidate set for late interaction","Balanced clustering guides disk storage of token embeddings","Disk-backed late interaction lowers memory and latency"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The fine-tuned MLLM used for lexical filtering will not exclude any truly relevant documents from the small candidate set passed to late interaction.","fun_headline_variants_meta":{"raw":{"variants":["Stellar stores token embeddings on disk for scalable retrieval","MLLM lexical filtering reduces candidate set for late interaction","Balanced clustering guides disk storage of token embeddings","Disk-backed late interaction lowers memory and latency"]},"model":"grok-4.3","cost_usd":0.005687,"raw_usage":{"total_tokens":2717,"prompt_tokens":669,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":56874500,"prompt_tokens_details":{"text_tokens":669,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1991,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":669,"tokens_out":57,"duration_ms":14516,"temperature":1.0,"reasoning_tokens":1991,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T15:48:03.610870+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the LRF step on a benchmark and finding that for some queries the gold relevant document is not in the filtered candidate set.","supporting_citations":[],"review_version":1}