{"id":"3bf43f2b-0fca-46d6-a3a7-38557003d3c7","arxiv_id":"2505.11908","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An LLM-driven lexical search loop with an importance-based sufficiency check beats embedding-based RAG baselines on long-context QA benchmarks while cutting storage and preparation cost dramatically.","lead":"This paper presents ELITE, a retrieval method for long-context question answering that skips vector embeddings during search and instead asks a language model to generate search terms, pull matching sentences, and iteratively broaden the search until enough evidence is collected. It reports better accuracy than embedding-based RAG baselines on NovelQA and Marathon with much lower storage and preparation time, though an internal scoring step quietly reintroduces embeddings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 'Ours' results likely rest on a 10-way majority vote (voter_num=10, Fig. 1), while baselines are single-pass; until a single-vote ablation is shown, the central accuracy comparison is confounded.","rationale":"The reader's weakest assumption is lexical coverage: that LLM-generated target terms will lexically overlap answer-bearing sentences. I view that as a robustness limitation, not the primary threat to the central claim: if lexical recall failed on a subset of questions, the iterative expansion in Section 4.3 might still rescue them, and the reader's concern would weaken or strengthen the claim only after an error analysis. The majority-vote confound attacks the validity of the headline accuracy comparison directly: the paper's own Section 5 lists voter_num=10 and Figure 1 includes a 'Majority Vote', but the method section never explains this component. This is an internal reproducibility and controlled-comparison failure, not a disagreement with external consensus. I recommend keeping the reader's CONDITIONAL verdict because the issue is addressable: a voter_num=1 ablation and matched baseline voting would settle it, and the efficiency and storage claims are separately plausible. If the ablation were run and the single-vote result still held, the paper would be much stronger; if not, the central claim should be regarded as unsupported.","tokens_in":10425,"tokens_out":12112,"duration_ms":126556,"concrete_test":"Re-run Table 2 under two conditions: (i) ELITE with voter_num=1, i.e., a single greedy answer with no majority vote, and (ii) each baseline with 10-way majority voting. Do this on NovelQA and Marathon at LLaMA-3.1-8B and 70B, reporting accuracy and cost. If ELITE single-vote no longer beats its baselines, or if baselines with voting match or exceed ELITE's 10-vote scores, the central accuracy claim is an ensemble artifact rather than a retrieval effect. If ELITE with voter_num=1 still wins, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 ('Experimental Setting') declares voter_num=10 for ELITE and states that all baselines 'strictly follow their original settings'. Figure 1 places a 'Majority Vote' immediately before 'Final Answer', yet Section 4 never defines voter_num or analyzes its effect. If voter_num is the number of sampled answers aggregated by majority vote, then every 'Ours' row in Table 2 is a 10-way self-consistency ensemble, while MiniRAG, RAPTOR, vanilla, and the cited leaderboard entries are single generations. Majority voting over LLaMA outputs on long-context multiple-choice QA is known to add several accuracy points, so the 12.3-point NovelQA gap over RAPTOR at 70B and the 3.3-point Marathon gap cannot be attributed to the retrieval component without an ablation. The headline claim that 'our approach outperforms strong baselines' is therefore not supported by a controlled comparison. If voter_num instead refers only to votes in the sufficiency judge, the manuscript must say so; the current text is ambiguous and the implementation is not reproducible as written.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ELITE, a retrieval-augmented generation framework for long-context multiple-choice QA that avoids offline dense indexing and explicit graph construction. The method iteratively (i) has an LLM generate lexical search terms from the question and from prior retrieval results, (ii) collects sentences by word overlap and expands them with symmetric context windows, (iii) computes an 'importance' score for each chunk from the sensitivity of the LLM's output distribution to character-level noise, and (iv) uses an LLM judge with these importance scores to decide whether to stop or explore further; final answers are produced after majority voting. Experiments on NovelQA and Marathon with LLaMA-3.1/3.2 models report accuracy gains over internal, vanilla, MiniRAG, and RAPTOR baselines and large reductions in preprocessing time and storage overhead, claiming near-zero preparation cost, a 1.0x storage ratio, and roughly two orders of magnitude runtime savings on long documents.","tokens_in":10661,"tokens_out":5792,"duration_ms":56594,"significance":"If the claims were fully supported, ELITE would be a useful contribution: it shows that a lexical, LLM-driven retrieval loop can be competitive with learned dense retrieval on long-context QA, and it provides concrete efficiency numbers for avoiding offline indexing. The paper does not fit parameters to the test sets, so the headline accuracies are external benchmark results rather than circular in-sample numbers. However, two load-bearing issues currently prevent accepting the central claims: the method is not actually embedding-free as stated because the importance computation uses cosine similarity over embedded texts, and the Table 2 comparison is confounded by an undefined ten-way majority vote that likely gives ELITE a self-consistency advantage not shared by the baselines. The derivation of the importance measure from KL divergence to cosine similarity is also not rigorous.","major_comments":[{"comment":"The paper's central claim of being embedding-free is contradicted by the implementation sentence 'we used cosine similarity between embedded result texts to calculate the similarity.' Even if the retrieval stage itself uses lexical overlap, the framework as a whole relies on embeddings in its importance-based sufficiency judge, so the abstract's 'embedding-free retrieval framework' and contribution 1 ('eliminates reliance on embedding models and dense indexing') are overstatements. Please either rework the terminology to 'without embedding-based retrieval' and clearly separate the embedding-based importance heuristic, or remove embeddings from the pipeline; as written, the central claim is internally inconsistent.","section":"Section 4.2, Eqs. (3)-(4); Abstract"},{"comment":"The configuration line 'voter_num=10' is never defined in Section 4, and Fig. 1 inserts a 'Majority Vote' step before the final answer. If this means ten answer samples are aggregated, then every 'Ours' row in Table 2 is a ten-way self-consistency ensemble, while MiniRAG, RAPTOR, vanilla, and the cited leaderboard entries are single-pass systems. Majority voting is known to improve LLM accuracy on long-context multiple-choice QA, so the 12.31-point NovelQA gap over RAPTOR at 70B and the 3.29-point Marathon gap cannot be attributed to the retrieval component without a single-vote ablation. Please report Ours with voter_num=1, or clarify exactly what voter_num controls and show that it does not materially affect the reported accuracy.","section":"Section 5, Experimental Setting; Fig. 1; Table 2"},{"comment":"The transition from the expected-KL definition to the cosine-similarity Monte Carlo estimate is asserted rather than derived. The justification that output distributions 'maintain the same functional form with variance determined by the fixed temperature parameter' does not by itself imply that expected KL divergence is monotonically related to mean cosine similarity of output texts. Please either provide a rigorous derivation with the exact functional form assumed, or present the importance score as a heuristic and validate it empirically (for example, by showing that high-importance chunks are the ones whose removal changes the correct answer), and remove the word 'objective' from the claims about this metric.","section":"Section 4.2, Eqs. (3)-(4)"},{"comment":"Because the same LLM generates the search terms (4.1), produces the output distributions used for importance (4.2), and decides sufficiency (4.2), the stopping criterion is fully self-referential. No experiment shows that when the judge says 'enough,' the retrieved chunks actually contain the answer, or that the iterative loop would continue if the answer were absent. Please add an analysis of the sufficiency judge's reliability, for example by checking answer containment or lexical overlap between the final retrieved set and gold evidence, and include an ablation that disables the judge (e.g., fixed iteration count) to demonstrate that the judge contributes positively to accuracy.","section":"Section 4.2 and Section 5.1"},{"comment":"The claim of 'ranking 2nd overall on both benchmarks' compares LLaMA-3.1-70B against proprietary systems such as GPT-4 and Claude under different inference conditions; this cross-model leaderboard comparison is not a controlled evaluation of the retrieval method. The accuracy differences in Table 2 are the only place where the retrieval component is compared with same-backbone baselines, and the contribution 3 claim of 'outperforming all baseline methods at every scale' should be limited to those same-backbone settings or explicitly labeled as an uncontrolled leaderboard comparison.","section":"Section 5.1 and Contributions"}],"minor_comments":[{"comment":"The special handling of counting questions is described only in prose; no prompt, no separate accuracy number, and no ablation are provided. Add the counting prompt and report the counting-question subset accuracy separately.","section":"Section 4.1"},{"comment":"The context expansion window (five sentences before and after each retrieved sentence) is not listed in the Experimental Setting; add it to the configuration list so that the method is reproducible.","section":"Section 4.1"},{"comment":"The notation \\theta(c⊕q) is ambiguous because \\theta is earlier defined as the model parameters; write the model function explicitly and specify how the output text is embedded and averaged when computing cosine similarity.","section":"Section 4.2, Eq. (4)"},{"comment":"For 'Our Method,' the preparation time is nonzero (0.004-0.085s) even though no offline index is built; clarify what this preparation includes and whether the reported retrieval time includes all LLM calls for term generation, importance scoring, and sufficiency judging.","section":"Table 3"},{"comment":"The flowchart is hard to read; the 'Not Enough/Expansion' and 'Enough' transitions are not clearly connected to the iterative loop, and the boxes for components such as 'Importance Calculation' and 'Retrieval-Reduction Agent' are underspecified.","section":"Figure 1"},{"comment":"The footnote says code is available at a GitHub repository, but no URL is given; provide the repository link and runnable configuration files so that voter_num, prompts, and the noise-perturbation procedure are reproducible.","section":"Abstract footnote"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea, but I would not accept it in current form. The voter_num issue is the most serious: if it is majority voting, the main accuracy table is not a controlled comparison, and a single-vote ablation is essential. The use of embeddings in the importance module also needs to be reconciled with the 'embedding-free' framing, and the importance approximation needs either a real derivation or a downgrade to heuristic status. If the authors provide these, the paper could become a solid contribution to retrieval-augmented generation for long-context QA."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know before you spend time on this: the central accuracy comparison is probably confounded. The paper sets voter_num=10, Figure 1 shows a Majority Vote before the final answer, and Section 4 never defines voter_num or reports a single-vote ablation. If voter_num is 10 sampled answers aggregated by majority vote, then every \"Ours\" row in Table 2 is an ensemble, while MiniRAG, RAPTOR, vanilla, and the leaderboard baselines are single generations. On multi-choice long-context QA, that alone can be worth several points. The reported 12.3-point gap over RAPTOR at 70B on NovelQA cannot be attributed to the retrieval component without a controlled comparison. The stress-test note holds up on reading.\n\nWhat is genuinely new and useful: the idea of replacing offline indexing with an iterative loop where the LLM generates search terms, lexical overlap collects sentences, an importance-based judge decides sufficiency, and expansion happens breadth- and depth-wise. The efficiency story is compelling if it holds — near-zero preparation time and 1.0x storage are big practical wins for single-query-per-document scenarios. The importance score is a real twist, and the paper is honest that it is only auxiliary evidence.\n\nSoft spots beyond the vote issue: the paper calls itself \"embedding-free\" but Section 4.2 explicitly uses cosine similarity between embedded result texts for the importance calculation. That is a naming problem at least, and a conceptual one if the framework is presented as not relying on embeddings at all. The derivation from Eq. 3 to Eq. 4 is hand-waved; the monotonic relationship between KL divergence and embedding similarity is assumed, not shown. There is also an internal inconsistency: the text says five sentences of context on each side, but the experimental setting says neighbor_num=2. No error bars, no ablations of recall_index, deep_search_num, or voter_num, and no code artifact despite the footer claiming a GitHub repo. The paper also does not cite earlier LLM-in-the-loop iterative retrieval work like Self-RAG, which it should have engaged with.\n\nIf the authors fix the evaluation — add a single-vote ablation, define voter_num, release prompts and code, and reconcile the embedding usage — the efficiency contribution alone would merit a real look. As it stands, the accuracy claims are not supported by a controlled comparison, but the core idea is worth refereeing rather than desk-rejecting.\n\nCandidly, I would not cite this in its current form, but I would bring it up in a reading group to discuss evaluation pitfalls in retrieval papers. Send it to reviewers who insist on ablations.","headline":"The accuracy numbers in Table 2 are likely a 10-way majority-vote ensemble, not a fair single-pass comparison, so the headline retrieval wins are not yet established.","tokens_in":11173,"tokens_out":2586,"would_cite":false,"duration_ms":28389,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that retrieval for long-context question answering can be done without embeddings or offline indexing, replacing them with iterative LLM-driven lexical search guided by an importance-based sufficiency judge.","keywords":["retrieval-augmented generation","embedding-free retrieval","long-context question answering","iterative search","lexical overlap retrieval","importance score","implicit knowledge graph","efficient RAG"],"falsifier":"Construct a long-context QA set where every answer sentence paraphrases or renames the key entities from the question so no question-derived term appears in the gold sentence. Run ELITE and a strong embedding retriever on it: if ELITE's accuracy falls toward the no-retrieval baseline while the embedding retriever holds, the lexical-coverage assumption is the limiting factor. A smaller probe is to take NovelQA questions, replace answer-bearing names with synonyms, and measure the drop.","tokens_in":10204,"feed_emoji":"🔍","tokens_out":5216,"duration_ms":45761,"temperature":0.7,"pith_summary":"The paper argues that effective retrieval for long-context question answering does not need vector embeddings or any offline index. It presents ELITE, a retrieval loop in which an LLM generates search terms, a word-overlap pass collects matching sentences, and an importance-based judge decides whether the collected evidence is sufficient, expanding the search when it is not. On the NovelQA and Marathon benchmarks, ELITE with LLaMA-3.1-70B reaches 71.27% and 68.46% accuracy, surpassing graph- and embedding-based retrieval baselines and ranking second overall on both leaderboards. Because it never builds an index, its preparation time is near zero and its storage overhead is 1.0x, in contrast to baselines that multiply storage 10- to 20-fold. The central claim is that LLM reasoning plus iterative lexical search can match or beat dense retrieval while being far cheaper.","feed_headline":"No embeddings: LLM search beats RAG baselines on long-context QA","feed_subtitle":"ELITE uses iterative LLM search plus an importance judge, cutting storage and runtime by over 10 times.","key_machinery":"The load-bearing mechanism is the iterative exploration loop built on two components: a word-overlap collector, $R = \\operatorname{topK}(\\{(s_i, \\operatorname{overlap}(s_i, T)) \\mid s_i \\in S\\})$, which retrieves sentences by exact term matching; and an importance-based sufficiency judge, whose score $Imp(c,q;\\theta)=1-\\operatorname{AvgSim}(\\theta(c\\oplus q), \\theta(\\varepsilon_\\lambda(c)\\oplus q))$ estimates how much a chunk drives the model's answer by measuring output sensitivity to character-level noise. The judge's verdict gates breadth-wise and depth-wise term expansion, which emulates graph propagation without constructing a graph. The importance score does the argument's heavy lifting: it turns the subjective question of whether enough evidence has been collected into a measurable quantity, but the whole chain still depends on the collector's lexical matches.","core_discovery":"ELITE's central discovery is that a retrieval system can replace embedding similarity with an explicit generate-collect-judge-expand loop. For a question, the LLM first produces phrases likely to carry the answer; sentences containing those phrases are scored by raw lexical overlap and the top K, expanded with neighboring sentences, are passed to a sufficiency judge. The judge combines the model's own assessment with an objective importance score: the expected change in model output when a chunk is corrupted by character-level noise, estimated as $1 - \\operatorname{AvgSim}(\\theta(c\\oplus q), \\theta(\\varepsilon_\\lambda(c)\\oplus q))$. If the evidence is judged insufficient, the LLM generates a broader term list (breadth-wise) and a deeper list derived from retrieved chunks (depth-wise), traversing an implicit knowledge graph without materializing it. This loop repeats until sufficiency or a limit, then the model trims and answers. The paper demonstrates that this pipeline outperforms MiniRAG and RAPTOR at every model scale tested and approaches proprietary systems.","pith_inferences":["A testable extension: mutate the benchmark by paraphrasing answer-bearing sentences so no generated term matches lexically; if ELITE's accuracy drops sharply while embedding baselines hold, the lexical-overlap bridge is the binding constraint.","The importance score could be reused outside sufficiency judging, for example as a training signal for query-expansion models or as a diagnostic for which retrieved chunks actually change an LLM's answer.","The method's reliance on exact term overlap suggests it may transfer better to technical or entity-dense domains, where answers reuse question vocabulary, than to heavily paraphrased or non-English text; measuring that boundary would clarify where embedding-free retrieval is appropriate.","Because the search loop is prompt-driven, the same machinery could be used for tasks beyond QA, such as evidence-based fact-checking or multi-document summarization, by swapping the sufficiency criterion."],"forward_implications":["RAG can operate with zero offline indexing: preparation time for documents over 2M tokens drops from thousands of seconds (19,337.6s for MiniRAG, 6,423.2s for RAPTOR) to 0.085s.","Storage overhead can be eliminated: ELITE reports a fixed 1.0x expansion ratio versus 10.6x-19.6x for graph-based baselines.","Retrieval policy becomes inspectable and modifiable at test time: because search terms are generated text, a user can see why a document was retrieved and can steer the term list, unlike with dense vectors.","Performance scales with base model capability: ELITE improves monotonically from 1B to 70B parameters and beats both MiniRAG and RAPTOR at every scale on both benchmarks.","The method's accuracy approaches proprietary systems: 71.27% on NovelQA versus 71.80% for the leading GPT-4-0125-preview, and 68.46% on Marathon versus 78.59% for GPT-4."],"supporting_citations":[{"why":"Defines retrieval-augmented generation, the framework ELITE challenges with an embedding-free alternative.","marker":"[11]"},{"why":"Graph-light RAG baseline that ELITE compares against on accuracy, runtime, and storage.","marker":"[5]"},{"why":"Recursive tree-structured retrieval baseline whose preparation time and storage overhead ELITE contrasts with its own.","marker":"[18]"},{"why":"Source of the NovelQA benchmark and its 2,305 long-context questions used in the main evaluation.","marker":"[21]"},{"why":"Source of the Marathon benchmark and its 1,530 long-context questions used in the main evaluation.","marker":"[22]"},{"why":"Jina embedding model examined in the limitations analysis and used as a strong retrieval baseline.","marker":"[20]"},{"why":"NV-Embed model tested in Table 1 to show dense retrievers can favor syntactically similar distractors.","marker":"[10]"}],"fun_headline_variants":["Embedding-free LLM retrieval beats RAG baselines on long-context QA","ELITE replaces embeddings with LLM iteration, cutting storage 10x","LLM iterative search without embeddings outperforms RAG on QA","No embeddings, no graph: ELITE's LLM loop beats RAG, cuts storage","Drop embeddings: ELITE's LLM search beats RAG and trims storage 10x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the answer-bearing sentences will contain at least some of the words the LLM generates as search terms, so if the answer is expressed in completely different vocabulary from the question, the word-overlap collector retrieves nothing and the expansion loop must stumble onto the right terms.","fun_headline_variants_meta":{"raw":{"variants":["Embedding-free LLM retrieval beats RAG baselines on long-context QA","ELITE replaces embeddings with LLM iteration, cutting storage 10x","LLM iterative search without embeddings outperforms RAG on QA","No embeddings, no graph: ELITE's LLM loop beats RAG, cuts storage","Drop embeddings: ELITE's LLM search beats RAG and trims storage 10x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1608,"prompt_tokens":940,"completion_tokens":668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":563}},"tokens_in":556,"tokens_out":668,"duration_ms":6406,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:44:55.451031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a long-context QA set where every answer sentence paraphrases or renames the key entities from the question so no question-derived term appears in the gold sentence. Run ELITE and a strong embedding retriever on it: if ELITE's accuracy falls toward the no-retrieval baseline while the embedding retriever holds, the lexical-coverage assumption is the limiting factor. A smaller probe is to take NovelQA questions, replace answer-bearing names with synonyms, and measure the drop.","supporting_citations":[],"review_version":1}