{"id":"2b211b55-0269-4372-8237-d1492be0c371","arxiv_id":"2611.16488","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Selective LLM evaluation guided by a disagreement-based active learning model recovers most retrieval quality at a fraction of energy cost.","lead":"This paper proposes LAAM, a hybrid retrieval system that uses active learning to approximate LLM ranking with far less energy. It achieves 87% of the accuracy of full LLM reranking on long narrative queries while using only 3.3% of the energy, allowing practitioners to trade accuracy for energy budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"LLM relevance judgments on TREC Robust 2004 may reflect memorization rather than inference, undermining the claim that LAAM approximates a superior ranking function.","rationale":"The reader's weakest_assumption focused on the greedy selection policy lacking theoretical guarantee. However, the empirical comparisons (Table 2, Figure 6) show that the greedy policy outperforms alternatives, so this is not the most fragile point. A more fundamental concern is the potential memorization in the LLM's relevance judgments due to training data overlap. The paper acknowledges this limitation (b242-b245) but does not fully resolve it. If the LLM's advantage on narrative queries is partly memorization, then LAAM's performance which approximates those potentially inflated judgments may not generalize. This is load-bearing because the central claim (b218) directly depends on the LLM being a superior ranker that LAAM can efficiently approximate. The suggested test (repeat on a post-cutoff dataset) would directly assess whether the observed gains are robust or rely on memorization. The reader's verdict of CONDITIONAL already acknowledges concerns; this test is a concrete way to satisfy that condition. No change to the verdict is necessary, as the condition remains appropriate.","tokens_in":17231,"tokens_out":1125,"duration_ms":24593,"concrete_test":"Replicate the main experiment (narrative queries, K=50 and K=100) on a dataset whose documents and relevance judgments post-date DeepSeek's training cutoff (e.g., TREC 2023 or a private collection). If the P@10 gap between DeepSeek and BM25 narrows significantly compared to Table 1, then the generalizabillity of LAAM's gains is weakened. If the gap remains comparable, the memorization concern is mostly dismmissed for the current result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on DeepSeek being a genuinely superior ranking function whose judgments LAAM can efficiently approximate. However, DeepSeek's training data may include the queries and documents of TREC Robust 2004, as the authors acknowledge (Section 6, [b242-b245]). If DeepSeek's relevance scores reflect memorized patterns rather than pure inference, the apparent gain over BM25 on narrative queries where the gap is largest could be partially artifactual. The authors mitigate by noting that on short queries LAAM does not consistently beat BM25, but this does not rule out selective memorization for verbose queries that more closely resemble training examples. If the advantage of exhaustive DeepSeek reranking shrinks on a corpus not seen during training, then LAAM's ability to recoup 87% of that advantage would also shrink, and the system's value as a general solution would be weakened. This is load-bearing because it questions whether the fundamental premise (that the LLM is a semantically superior ranker for long queries) holds on genuinely unseen data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes LAAM (LLM Active Approximation Model), a hybrid retrieval framework that uses a lightweight Gradient Boosting model trained via active learning to approximate the relevance judgments of an LLM (DeepSeek) for reranking documents. The goal is to reduce the energy cost of exhaustive LLM reraning while preserving most of its retrieval quality, especially for verbose queries. On TREC Robust 2004, for narrative queries LAAM at 100 LLM calls achieves 87% of the P@10 of full DeepSeek reranking (0.459 vs 0.526) while consuming an estimated 3.3% of the per-query energy. LAAM is statistically significantly better than BM25 and statistically indistinguishable from DeepSeek on P@1 and P@10 for narrative and description queries, though DeepSeek remains better on MAP.","tokens_in":16839,"tokens_out":4352,"duration_ms":35407,"significance":"The problem of making LLM-based retrieval computationally feasible is timely and important. LAAM offers a pragmatic approach by learning a surrogate model that concentrates LLM evaluations on informative documents, exposing a continuous energy-accuracy trade-off. The evaluation includes statistical tests and ablation studies, and the energy measurements are partially hardware-measured. If the reported energy savings are robust, the framework could enable cost-aware retrieval for long queries, a scenario where LLMs provide the clearest semantic advantage but have been prohibitive.","major_comments":[{"comment":"The central efficiency claim—that LAAM consumes only 3.3% of the energy of full DeepSeek reranking—rests on an estimated rather than directly measured energy cost for DeepSeek. The authors acknowledge this limitation in §6 but the abstract and conclusions present the figure without the necessary caveat. Moreover, the paper does not clarify whether the energy measurements of the LAAM pipeline include the cost of (re)training the Gradient Boosting model at each iteration. If retraining is not included, the reported energy for LAAM is an underestimate. Please either provide direct measurements, expand the sensitivity analysis of the estimation, or qualify the 3.3% figure prominently, and specify whether model training energy is accounted for.","section":"§4.2.6, §5.2, Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The authors acknowledge that TREC Robust 2004 may be in DeepSeek's training data, so some relevance judgments may reflect memorization rather than inference. They mitigate this by noting that on short queries LAAM does not consistently beat BM25. However, this does not rule out selective memorization for verbose narrative queries—the very setting where the claim of LLM superiority is largest. If DeepSeek's advantage over BM25 on narrative queries is partially artifactual, then LAAM's ability to recoup 87% of that advantage becomes less meaningful. The authors should discuss the magnitude of this risk more thoroughly and, if possible, provide evidence (e.g., a subcorpus analysis or evaluation on a collection not seen by DeepSeek) that the gains reflect genuine semantic reasoning.","section":"§6, lines [b242–b245]","authors_acknowledge":"Testing the framework on newer or private collections that are less likely to overlap with LLM training data would provide stronger evidence that the observed gains generalize beyond this benchmark.","authors_acknowledge_p":0.71},{"comment":"The abstract states LAAM is statistically indistinguishable from DeepSeek on average precision, and the strongest claim (b218) says LAAM 'in some cases surpasses [exhaustive LLM reranking].' While for narrative P@1 at K=50 LAAM is numerically higher (0.771 vs 0.674), the post-hoc tests confirm the difference is not stastically significant (the paper itself says LAAM is indistinguishable). The word 'surpasss' is therefore misleading. Please revise to indicate that apparent advantages are not stastically significant.","section":"Table 1, Abstract, line [b218]","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The positioning relative to BAGEL and TS-SetRank is useful, but it would be clearer to explicitly state that LAAM is the only one that learns a lightweight surrogate predictor, while the others address sampling or block selection.","section":"§2.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Reporting the Frriedman test p-values is appropriate, but consider including the p-values of the post-hoc pairwise comparisons (or a letter superscript) directly in the table for eader reference.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The parameters of the initial sampling strategies (e.g., a,b,c,β) are described as 'chosen arbitrarily.' A brief note on whether the conclusions are sensitive to these choices would be helpful.","section":"§4.2.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The x-axis is logarithmic; ensure the axis labels and ticks are clearly legible in black-and-white printing.","section":"Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The prompt used for DeepSeek is given; consider adding a note on whether the same prompt structure was used uniformly for all three query verbosity levels (title, description, narrative).","section":"Appendix A","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is a solid, well-executed empirical study. The core idea — train a gradient boosting surrogate to predict LLM relevance scores, using disagreement with BM25 to guide labelling — is not a paradigm shift but the implementation and evaluation are thorough. The novelty lies in combining disagreement-based active learning with a lightweight learned predictor, whereas prior work (BAGEL, TS-SetRank) focussed on sampling for the LLLM rather than learning a surrogate.\n\nThe paper's main strength is the extensive ablation: initial sampling strategies, feature sets, embedding models, selection policies. The central result — 87% of DeepSeek's P@10 at 3.3% of its energy on narrative queires — is clearly supported by diminishing returns. The direct energy measurement for their own pipeline (perf) and the use of a published estimation method for DeepSeek are appropriate.\n\nSoft spots are few but real. First, the claim that LAAM 'surpasses' full DeepSeek in some cases is not backed by the stats: they report 'indistinguishability' on P@1 and P@10, and DeepSeek is significantly better on MAP. The word 'surpasses' shold be removed or reserved for cases with proper support. Second, the energy figure for DeepSeek is estimated, not measured, so the exact 3.3% is approximate; the authors acknowledge this but it should be highlighted in any refereed version. Third, the memorization concern (DeepSeek may have seen TREC during training) is acknowledged and partially mitgated by the null results on short queires, but it remains an open question on truly unseen data — exactly as the authors state in their limitations.\n\nOverall this is a useful contribution for ther IR community, giving practitioner a concrete method to trade accuracy for energy. It should be acceptedf or peer review after toning down the 'surpasses' claim. I would probably cite it as a state-of-the-art baseline for active learning in retrieval.","headline":"A practical active learning framework for approximating LLM ranks with clear energy-accuracy trade-offs; \"surpasses\" claim needs softening.","tokens_in":17305,"tokens_out":1557,"duration_ms":13896},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Active learning with a lightweight surrogate predictor recovers most LLM reranking quality at a fraction of the energy cost.","keywords":["active learning","LLM ranking","energy-efficient retrieval","long queries","document reranking","gradient boosting","BM25","TREC Robust 2004"],"falsifier":"If another active learning acquisition function (e.g., variance-based uncertainty sampling) were shown to consistently outperform the greedy disagreement policy on this task, or if a similar surrogate trained on a different LLM or dataset failed to approach the full reranker's accuracy, the paper's claim that the specific selection policy recovers most retrieval quality would be weakened.","tokens_in":18026,"feed_emoji":"⚡","tokens_out":2123,"duration_ms":17814,"temperature":0.7,"pith_summary":"The paper tries to establish that a retrieval system can approach the accuracy of an expensive large language model (LLM) reranker by actively learning a cheap surrogate predictor that approximates the LLM's relevance judgments. The key insight is that most documents in a candidate list do not need to be evaluated by the LLM; a small, strategically chosen set suffices to train a model that reorders the rest. The authors demonstrate on TREC Robust 2004 using DeepSeek as the LLM that for verbose narrative queries the surrogate achieves 87% of the full reranker's precision at 10 while consuming only 3.3% of the per-query energy. The method exposes a continuous energy–accuracy tradeoff, allowing practitioners to adjust the number of LLM evaluations to fit a computational budget.","feed_headline":"Active learning cuts LLM reranking energy by 97% while keeping most accuracy","feed_subtitle":"Selective document evaluation guided by a lightweight predictor recovers 87% of full reranking quality on verbose queries.","key_machinery":"The LLM Active Approximation Model (LAAM): a gradient boosting regressor that predicts an LLM's relevance score (1–1000) from a feature vector combining TF-IDF cosine similarity, query-term overlap, and sentence-transformer embeddings of the query and document. During active learning, the system selects the document with the largest absolute difference between its predicted LAAM score and its BM25 score (greedy disagreement), labels it with the LLM, and incrementally retrains the model. This pure-exploitation policy concentrates LLM effort on documents where lexical and semantic signals diverge most.","core_discovery":"The central claim is that a gradient-boosting regression model, trained on a small number of LLM-labeled documents selected by a greedy disagreement-based active learning policy, can approximate the ranking of an exhaustive LLM reranker well enough to yield statistically indistinguishable precision at 1 and 10 for verbose queries, while cutting energy consumption by over 95%. The paper shows that this holds across three query verbosity levels, with the largest gains on long narrative queries where lexical retrieval is weakest. The energy–accuracy curve exhibits strong diminishing returns, meaning most improvements come from the first few dozen LLM evaluations.","pith_inferences":["The paper's strongest results are on narrative queries; the method shows no benefit over BM25 on short title queries, implying that a hybrid system should dynamically route queries: short ones to lexical retrieval, long ones to the LAAM pipeline. This is acknowledged as future work but left implicit as a design consequence.","The energy comparison relies on a remote API estimate for DeepSeek rather than direct measurement; if local inference hardware becomes more efficient, the relative energy advantage of LAAM may shrink, though the cost savings in API calls would remain.","The greedy disagreement policy is a heuristic; the paper explicitly notes that alternative strategies (e.g., uncertainty sampling) remain unexplored. A principled comparison would clarify whether the observed gains are tied to the specific selection rule or generalise to other active learning approaches.","The TREC Robust 2004 dataset is public and may overlap with LLM training data, making the absolute relevance judgments potentially optimistic. Testing on a private or newer collection would strengthen the claim that the surrogate truly learns reasoning rather than memorized patterns."],"forward_implications":["Retrieval systems can treat the number of LLM evaluations as a tunable resource, enabling a smooth transition from cheap lexical search to expensive semantic reranking depending on query complexity and budget.","The active learning loop can be extended to transfer knowledge across queries, potentially reducing the per-query labeling cost further by reusing a meta-model.","The framework is directly applicable to retrieval-augmented generation (RAG) pipelines, where it could lower latency and energy consumption by replacing exhaustive LLM reranking with a learned surrogate.","The finding that dense semantic embeddings dominate the feature importance suggests that cheaper embedding models (e.g., MiniLM) can be substituted with only a small loss in accuracy, further reducing computational overhead.","LAAM's reliance on a simple binary disagreement signal opens the door to investigating theoretically grounded acquisition functions (e.g., uncertainty sampling or Bayesian optimization) that might improve sample efficiency or provide guarantees."],"supporting_citations":[{"why":"BM25 as the baseline lexical retrieval method that LAAM builds upon.","marker":"[30]"},{"why":"DeepSeek V3 as the LLM whose relevance judgments LAAM approximates.","marker":"[23]"},{"why":"TREC Robust 2004 collection used for all experiments.","marker":"[37]"},{"why":"Gradient boosting regression, the core predictor of LAAM.","marker":"[11]"},{"why":"Active learning framework that motivates the iterative labeling strategy.","marker":"[33]"},{"why":"BAGEL, a related budget-constrained LLM reranking method using Gaussian processes, used as a point of comparison.","marker":"[19]"},{"why":"TS-SetRank, another budget-constrained approach that selects documents based on contextual relevance uncertainty.","marker":"[15]"},{"why":"Infrastructure-aware energy estimation methodology used for DeepSeek inference.","marker":"[17]"}],"fun_headline_variants":["Active learning approximates LLM reranking at 3% of the energy","Selective LLM evaluation cuts energy 97% with 87% precision","Greedy active learning targets informative docs, saves 97% energy","87% reranking quality for just 3.3% of the energy cost","Lightweight surrogate replaces costly LLM reranking via active learning"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The greedy document selection policy, which always picks the document with the largest difference between the predicted LAAM score and the BM25 score, assumes that this disagreement signal is the most informative criterion for reducing surrogate error; no theoretical guarantee or empirical comparison to uncertainty-based sampling is provided.","fun_headline_variants_meta":{"raw":{"variants":["Active learning approximates LLM reranking at 3% of the energy","Selective LLM evaluation cuts energy 97% with 87% precision","Greedy active learning targets informative docs, saves 97% energy","87% reranking quality for just 3.3% of the energy cost","Lightweight surrogate replaces costly LLM reranking via active learning"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"cost":0.000307468,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000307468,"upstream_inference_prompt_cost":0.000111188,"upstream_inference_completions_cost":0.00019628},"total_tokens":1700,"prompt_tokens":999,"completion_tokens":701,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":602}},"tokens_in":999,"tokens_out":701,"cost_source":"openrouter_usage","duration_ms":6816,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":602,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T14:47:41.522103+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If another active learning acquisition function (e.g., variance-based uncertainty sampling) were shown to consistently outperform the greedy disagreement policy on this task, or if a similar surrogate trained on a different LLM or dataset failed to approach the full reranker's accuracy, the paper's claim that the specific selection policy recovers most retrieval quality would be weakened.","supporting_citations":[{"cited_title":"Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval","cited_arxiv_id":null,"evidence_quote":"BM25 as the baseline lexical retrieval method that LAAM builds upon."},{"cited_title":"DeepSeek-V3 Technical Report","cited_arxiv_id":"2412.19437","evidence_quote":"DeepSeek V3 as the LLM whose relevance judgments LAAM approximates."},{"cited_title":"Overview of the trec 2004 robust track","cited_arxiv_id":null,"evidence_quote":"TREC Robust 2004 collection used for all experiments."},{"cited_title":"Greedy function approximation: a gradient boosting machine.Annals of statistics, pages 1189–1232, 2001","cited_arxiv_id":null,"evidence_quote":"Gradient boosting regression, the core predictor of LAAM."},{"cited_title":"Springer Science & Business Media, 2011","cited_arxiv_id":null,"evidence_quote":"Active learning framework that motivates the iterative labeling strategy."},{"cited_title":"Bayesian active learning with gaussian processes guided by llm relevance scoring for dense passage retrieval","cited_arxiv_id":null,"evidence_quote":"BAGEL, a related budget-constrained LLM reranking method using Gaussian processes, used as a point of comparison."},{"cited_title":"Contextual relevance and adaptive sampling for llm-based document reranking","cited_arxiv_id":null,"evidence_quote":"TS-SetRank, another budget-constrained approach that selects documents based on contextual relevance uncertainty."},{"cited_title":"How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference","cited_arxiv_id":null,"evidence_quote":"Infrastructure-aware energy estimation methodology used for DeepSeek inference."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}