{"id":"dfed3f56-31fc-4adc-b43a-f5c70eb427b2","arxiv_id":"2412.11832","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"DCRF uses zero-shot LLM prompts to pick the best ranking among eight rerankers per query, beating each individual reranker on most datasets and approaching RankGPT-level quality, but the reported scores select the best prompt per dataset from test labels.","lead":"This paper proposes DCRF, a pipeline that runs eight search rerankers in parallel and uses an untrained large language model to decide which reranker's ranking to return for each query. The headline result, near-parity with the much costlier RankGPT, rests on a per-dataset choice of the best prompt made after seeing test answers, so the practical claim is weaker than stated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 already falsifies the universal-superiority claim: even with per-dataset prompt selection from test labels, DCRF trails BM25 on Touche, DeBERTa on Signal1m, and MonoT5-3B on NFCorpus.","rationale":"The reader's verdict is REJECT, and my analysis agrees. The single most load-bearing premise is the zero-shot rank evaluator's reliability, and the strongest evidence against it is internal: Table 1's own results. I considered whether the per-dataset prompt selection (Section 4.4) is the deeper issue; it is a real leakage problem because the reported system is not a fixed pipeline and requires test labels to choose among four prompts. But the more decisive objection is that even after exploiting that leakage, DCRF fails on three datasets. On Touche, BM25's 44.22 is 10.5 points above the best DCRF in Table 1 (Vicuna-13b, 33.72). On Signal1m, DeBERTa-v3's 33.41 beats the best DCRF (ChatGLM2-6b, 32.71). On NFCorpus, MonoT5-3B's 38.97 beats the best DCRF (LLaMA3-70b, 37.95). These numbers are not sensitive to prompt choice because we are taking the max over all reported prompt variants. Therefore the abstract's 'outperforming each single model across all queries and corpora' is falsified as written. A fixed, zero-shot, no-label version would presumably be worse. The proposed concrete check—running one pre-registered prompt without test-set selection—would settle the residual question of how much of Table 1 depends on label leakage, but it cannot rescue the claim already contradicted by Table 1. The verdict should remain REJECT.","tokens_in":15202,"tokens_out":5719,"duration_ms":48328,"concrete_test":"Re-run DCRF with a single pre-registered prompt (Appendix B default Passage-Pointwise-Complex) on all 10 datasets for each of the six LLMs, without per-dataset prompt selection, and compare against BM25, DeBERTa-v3, and MonoT5-3B. If any single reranker beats the fixed-prompt DCRF on any dataset (expected on Touche, Signal1m, and NFCorpus), the zero-shot 'all queries and corpora' claim is falsified; additionally report per-query win rates on Touche to show how often the evaluator fails to select BM25's clearly superior rank.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Section 4.4) is that DCRF outperforms each single retrieval model across all queries and corpora using a zero-shot evaluator. The paper's own Table 1 refutes this. Using the per-dataset best of four prompts (selected after seeing test labels, Section 4.4), the best DCRF on Touche is Vicuna-13b at nDCG@10 33.72, while BM25 scores 44.22; even the best black-box variant in Table 3 (Claude-3.5, 34.83) stays far below BM25. On Signal1m, the best DCRF is ChatGLM2-6b at 32.71, below DeBERTa-v3's 33.41. On NFCorpus, the best DCRF is 37.95, below MonoT5-3B's 38.97. Since these counterexamples remain even under the favorable 'oracle prompt per dataset' protocol, they are not artifacts of a poorly chosen prompt; they show the zero-shot LLM rank evaluator fails to identify the best reranker exactly on datasets where sparse or distilled models dominate. Additionally, Section 4.4's statement that the authors 'choose the most effective prompting strategy for each dataset' means the reported DCRF is not a fixed, deployable system but a per-dataset oracle over prompts, contradicting the no-labeled-data framing. The abstract promise and the Table 1 numbers cannot both be true.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Distributed Collaborative Retrieval Framework (DCRF) that combines eight existing retrievers/rerankers (BM25, Contriever, GTR, MonoBERT, MonoT5 variants, DeBERTa-v3, RankLLaMA) and then uses an LLM prompted with one of four zero-shot strategies to select the best ranked output for each query. The authors claim that DCRF outperforms each single model across all queries and corpora, is comparable to RankGPT and ListT5, and is more efficient, all without relying on labeled data. Experiments are conducted on TREC-DL19/DL20 and eight BEIR datasets with six open-source LLMs and three black-box LLMs as evaluators.","tokens_in":15545,"tokens_out":6279,"duration_ms":55162,"significance":"If the central claims were correct, the paper would make a useful contribution: a modular, training-free way to combine diverse retrieval models and an LLM-based rank evaluator that selects the best result per query would be attractive for practical IR systems. The paper also includes useful ingredients, such as four explicitly described prompting strategies, an upper-bound analysis with supervised metrics, and a cost comparison against listwise rerankers. However, the main empirical claim is contradicted by the paper's own results, and the evaluation protocol selects prompts on test labels, so the current evidence does not support the headline contribution.","major_comments":[{"comment":"The claim that DCRF \"outperforms each single model across all queries and corpora\" is falsified by Table 1. Even under the paper's most favorable protocol, in which the best prompting strategy is selected per dataset from the test labels, DCRF's best result on Touche is 33.72 (Vicuna-13b) while BM25 obtains 44.22; on Signal1m the best DCRF is 32.71 (ChatGLM2-6b) while DeBERTa-v3 obtains 33.41; and on NFCorpus the best DCRF is 37.95 while MonoT5-3B obtains 38.97. The black-box results in Table 3 do not fix this: Claude-3.5 reaches only 34.83 on Touche, still far below BM25. Because these counterexamples appear even when the prompt is chosen oracle-style from the test set, they are not artifacts of a poorly chosen prompt; they contradict the title and the abstract's universal claim. The paper should either remove the \"all queries and corpora\" claim or restrict it to \"most datasets\" and discuss the failure cases explicitly.","section":"Abstract; §4.4, Table 1"},{"comment":"The reported DCRF is not a fixed, deployable system. Section 4.4 states that \"we choose the most effective prompting strategy for each dataset on the specific LLMs,\" and indeed every DCRF value in Table 1 equals the per-column maximum across the five prompt rows for the corresponding LLM in Table 5. This means the prompting strategy is selected after inspecting the test labels, so the label-free, zero-shot framing is not tested as stated. A practitioner facing a new corpus would not know which prompt to use without labeled data. The authors should either fix a single prompt per LLM (e.g., the Passage-Pointwise-Complex default mentioned in Appendix B) and report those results, or clearly label the Table 1 numbers as an oracle-prompt upper bound and relegate the unsupervised claims to a separate, properly validated protocol.","section":"§4.4; Table 5; Appendix B"},{"comment":"The claim that positional bias is \"minimal\" is not supported by the presented analysis. Figure 5 only reports how often each reranker is selected on Dbpedia-Entity; it does not measure whether the LLM evaluator's selected rank matches the label-defined best rank for each query. A model could be selected frequently yet incorrectly, and equal selection frequencies across positions do not establish that selection is correct. The authors should report per-query agreement between the LLM-selected rank and the best rank according to nDCG@10, or the nDCG loss of the selected rank relative to the oracle, across datasets and LLMs. Without such a correctness measure, the bias analysis is not informative about evaluator quality.","section":"§4.6.2, Figure 5"},{"comment":"The inference-cost comparison appears to omit the number of candidate ranks that must be evaluated. To choose among Nranks = 8 reranker outputs, the passage-based strategies must score passages in each candidate rank, so the cost is at least Nranks * k * Lp * TLLM for the passage-based methods, not the Lp * TLLM shown in Table 4. The rank-pairwise entry Nranks * k * Lp * TLLM is also inconsistent with the pairwise procedure, which would require comparisons across pairs of ranks. These formulas should be corrected before the efficiency advantage over RankGPT can be assessed.","section":"§4.6.1, Table 4"}],"minor_comments":[{"comment":"Figure 1 and its caption contain the typo \"queires\" for \"queries,\" and the frequency definition would benefit from a precise statement of how ties are broken when two models obtain the same best score on a query.","section":"Section 1; Figure 1"},{"comment":"The method name \"InPairs\" should be \"InPars\" (as in the reference list), and Table 1's \"DeBERTa-3v\" should be \"DeBERTa-v3.\"","section":"Section 4.4"},{"comment":"The passage-based evaluation methods are said to use NDCG, MAP, and MRR to combine per-passage LLM scores, but the text does not explain how the LLM scores are converted into a ranked list or how ties are handled; this should be clarified.","section":"Section 3.2.1"},{"comment":"The prompt text contains \"ranked responsed\" and other grammatical errors; the prompts should be proofread because they are part of the method's reproducibility.","section":"Appendix A.6"},{"comment":"Limitation (3) says \"we haven't further analyse\" and \"improve the its domain generalization ability\"; these should be corrected.","section":"Section 6"}],"recommendation":"reject","confidential_remarks":"The central claim is contradicted by the paper's own Table 1 even under the most favorable oracle-prompt protocol, and the protocol itself undermines the zero-shot contribution because the best prompt per dataset is chosen from test labels. These are load-bearing problems that affect the title, abstract, and main evaluation. I do not see how a minor revision could address them without substantially changing the claims and rerunning the experiments with a fixed prompt policy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the paper's central claim is false on its own numbers. DCRF is supposed to \"outperform each single model across all queries and corpora,\" but Table 1 shows BM25 beating every DCRF variant on Touche (44.22 vs best 33.72), DeBERTa-v3 beating DCRF on Signal1m (33.41 vs 32.71), and MonoT5-3B beating DCRF on NFCorpus (38.97 vs 37.95). Those are not edge cases; they are exactly the datasets where a single model happens to dominate.\n\nWhat is genuinely new: the architecture of running eight heterogeneous rerankers in parallel and using a zero-shot LLM prompt to pick the best resulting rank is a reasonable ensemble-plus-judge design. I do not see that in the cited RankGPT/ListT5 line. The four prompt templates in Appendix A are concrete and documented, the ablations span 6 LLMs and 10 benchmarks, and the paper includes an inference-cost model and a positional-bias check. That is real work.\n\nWhere it falls apart: Section 4.4 says the authors \"choose the most effective prompting strategy for each dataset on the specific LLMs.\" That means every reported DCRF score is a per-dataset oracle over four prompts, selected after seeing test labels. The results are fitted, not predicted. And even with that favorable selection, the universal-superiority claim still loses on three datasets. The paper provides no code, no error bars, and no significance tests, so the small nDCG deltas elsewhere cannot be separated from noise. The Limitations section worries about improving the evaluator but does not flag the oracle selection, which is the more serious issue.\n\nIf the protocol were fixed (one prompt pre-registered per LLM, then results reported), the ensemble idea might still be interesting as a cheap way to approach RankGPT/ListT5 quality. But as written, the abstract overpromises and the experiments cannot support it. A reviewer should ask for a major rewrite, not just minor edits.\n\nRecommendation: if this lands on your desk, send it to review only if you are willing to push for major revision. The idea clears the bar for referee time; the current numbers do not. I would not cite this version.","headline":"The paper's headline claim is disproved by its own Table 1; the ensemble idea has merit, but the reported results are an oracle over prompts.","tokens_in":16099,"tokens_out":4388,"would_cite":false,"duration_ms":38978,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DCRF claims a zero-shot LLM can pick the best ranked list per query, matching RankGPT at lower cost.","keywords":["distributed collaborative retrieval","zero-shot evaluation","LLM as evaluator","ranking selection","reranking","BEIR","TREC deep learning","prompting strategies"],"falsifier":"A decisive test is to fix the prompt selection rule ahead of time (for example, always use the passage-pointwise prompt) and measure DCRF against each single reranker on all ten datasets; if the fixed-prompt system no longer beats the best reranker on most datasets, then the reported gains come from choosing prompts on the test labels rather than from the evaluation mechanism.","tokens_in":15014,"feed_emoji":"🎯","tokens_out":6648,"duration_ms":50190,"temperature":0.7,"pith_summary":"This paper proposes a Distributed Collaborative Retrieval Framework (DCRF) that runs several retrieval models in parallel and uses a zero-shot LLM prompt to pick the best-ranked list for each query. The authors argue that no single retriever wins across all queries and corpora, so dynamic per-query selection should outperform any fixed model. With eight rerankers and four prompting strategies, DCRF reaches effectiveness comparable to listwise rankers like RankGPT and ListT5 while keeping inference cost lower. The paper also shows that an oracle evaluator with human labels would push DCRF far higher, so the framework's current ceiling is set by the zero-shot evaluator.","feed_headline":"Zero-shot LLM selects best rank per query, beating single retrievers","feed_subtitle":"Eight rerankers run in parallel; a prompted LLM picks the winner, rivaling RankGPT at lower cost.","key_machinery":"The load-bearing object is the rank evaluator built from four zero-shot prompting strategies: passage-pointwise (score each passage 0-5), passage-relwise (answer yes/no to relevance), rank-pointwise (score an entire ranked list 0-100), and rank-pairwise (compare two ranked lists and pick the better). The reranker pool spans sparse (BM25), dense bi-encoders (Contriever, GTR), cross-encoders (MonoBERT), LM-based models (MonoT5-base and MonoT5-3b), a distilled DeBERTa, and RankLLaMA. Because all rerankers run in parallel, the inference time is set by the slowest reranker plus the evaluator's forward pass, which is the basis of the efficiency claim.","core_discovery":"The central claim is that rank-oriented automatic evaluation can be delegated to any capable LLM through carefully designed prompts, and that the resulting framework selects the best reranker output on most queries. The paper reports that DCRF with LLaMA3-70b as evaluator achieves 52.93 nDCG@10 on the BEIR average, above all selected retrieval models and close to RankGPT(GPT-4) at 53.68, while requiring less inference time. In the authors' view, this shows that a trained evaluator is unnecessary: the prompt itself supplies the judgment, and the distributed parallel execution supplies the efficiency.","pith_inferences":["A fixed deployment would need to choose one prompt per dataset without seeing test labels; the paper picks the best prompt on the test set, so a fully honest zero-shot system might lose some of the reported margin.","On datasets where one reranker dominates, such as Touché with BM25, the LLM evaluator fails to select the best rank, so a hybrid rule that trusts sparse retrievers in professional domains could close that gap.","The same rank-selection prompting could be applied to choosing among outputs from other generation systems, treating each system's ranked output as a unit.","The paper's oracle upper bound suggests that a trained or better-prompted evaluator could recover more than 10 nDCG points on several datasets, which is where future work would pay off."],"forward_implications":["On most of the 10 TREC and BEIR datasets, DCRF outperforms every individual reranker it contains, showing that query-level selection beats any fixed model.","The zero-shot LLM evaluator achieves results comparable to trained listwise rankers such as ListT5 and black-box RankGPT, which would remove the need for labeled training data in the selection stage.","Because reranking runs in parallel and the evaluator only reads a small set of candidates, DCRF's inference cost stays below listwise generation methods like RankGPT.","The framework is modular: any new retriever can be added without retraining the evaluator, so the system can adapt to new domains by expanding the pool."],"supporting_citations":[{"why":"Defines the listwise ranking baseline RankGPT and the benchmark selection of 10 TREC/BEIR datasets that DCRF follows.","marker":"(Sun et al., 2023)"},{"why":"Provides the BEIR benchmark used for evaluating zero-shot retrieval and the eight BEIR tasks in the experiments.","marker":"(Thakur et al., 2021)"},{"why":"Contributes the TREC Deep Learning track datasets DL19 and DL20 used in the evaluation.","marker":"(Craswell et al., 2021)"},{"why":"Supplies the MonoT5 rerankers (base and 3B) that serve as candidate rerankers in the parallel pool.","marker":"(Nogueira et al., 2020)"},{"why":"Supplies the MonoBERT cross-encoder reranker used as one of the dense candidates.","marker":"(Nogueira and Cho, 2019)"},{"why":"Supplies the Contriever dense retriever that provides a contrastive-learning-based reranking candidate.","marker":"(Izacard et al., 2021)"},{"why":"Supplies the GTR dual-encoder reranker used as a candidate in the pool.","marker":"(Ni et al., 2021)"},{"why":"Provides the LLaMA3-8b and LLaMA3-70b models that serve as zero-shot evaluation agents in the main results.","marker":"(AI@Meta, 2024)"},{"why":"Supplies the ListT5 listwise reranker that is a key trained baseline DCRF is compared against.","marker":"(Yoon et al., 2024)"},{"why":"Supplies the DeBERTa model whose distilled variant acts as one of the eight rerankers.","marker":"(He et al., 2020)"}],"fun_headline_variants":["Zero-shot LLM ranks outputs, beats every single retriever","Distributed retriever blends 8 models, LLM picks winner","LLM prompts replace trained rerankers, match RankGPT nDCG","No training needed: prompted LLM selects best rank per query"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a zero-shot LLM prompt reliably judges which ranked list is most relevant for an unseen query; the paper's own Table 1 shows this premise fails on some datasets, for example on Touché where DCRF's evaluator (nDCG@10 32.73) picks a rank far worse than the BM25 rank (44.22).","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM ranks outputs, beats every single retriever","Distributed retriever blends 8 models, LLM picks winner","LLM prompts replace trained rerankers, match RankGPT nDCG","No training needed: prompted LLM selects best rank per query"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1276,"prompt_tokens":898,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":514,"tokens_out":378,"duration_ms":4306,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:31:52.294036+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to fix the prompt selection rule ahead of time (for example, always use the passage-pointwise prompt) and measure DCRF against each single reranker on all ten datasets; if the fixed-prompt system no longer beats the best reranker on most datasets, then the reported gains come from choosing prompts on the test labels rather than from the evaluation mechanism.","supporting_citations":[],"review_version":1}