{"id":"2e072458-dd1d-4679-a6a4-c54eb37a5faf","arxiv_id":"2412.14967","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ECLIPSE improves dense retrieval by subtracting a centroid of low-ranked documents from the relevant signal, with reported AP gains up to 19.50% on TREC collections, though gains are partly selected on the test set.","lead":"ECLIPSE subtracts the average embedding of irrelevant-looking documents from that of relevant-looking ones before selecting dimensions for dense retrieval scoring, and reports up to 19.50% AP gains on TREC benchmarks. The method is a simple and plausible extension of dimension-importance filtering, but the headline gains rely on test-set hyperparameter selection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline gains may be an artifact of selecting hyperparameters and the retained-dimension fraction on the test queries; no validation split is described and the test sets are tiny.","rationale":"The reader's weakest_assumption focused on the semantic content of bottom-ranked documents, but I believe the more load-bearing issue is the evaluation protocol. The paper does not describe a validation split and explicitly selects the best retained-dimension percentage on the test set. Because the query sets are tiny, the chance of overfitting is high. In addition, because the same test set is used for both model selection and significance testing, the reported p-values are not valid. This is not an accusation; it is a request for a standard train/validation/test protocol. The method itself is simple and plausible, and the RQ3 random-sampling analysis is a thoughtful robustness check, but the central empirical claim needs this check before acceptance. If the gains persist under a proper validation split, the paper would be considerably stronger. My verdict remains CONDITIONAL, matching the reader's overall verdict, hence UNCHANGED.","tokens_in":12962,"tokens_out":3896,"duration_ms":24787,"concrete_test":"Split each query set (DL'19, DL'20, DL-HD, RB'04) into a tuning subset and a held-out test subset. Tune k+, k-, alpha, beta, and the fraction of retained dimensions on the tuning subset only, then freeze the hyperparameters and evaluate on the held-out subset. Recompute Table 1 and the paired significance tests. If the mean AP/nDCG@10 gains over the DIME baselines shrink to non-significance or reverse on the held-out queries, the central claim is a test-set selection artifact; if the gains persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that ECLIPSE significantly improves DIME rests on Table 1, where the paper reports 'the best result among varying the percentage of retained dimensions' (Section 5). The hyperparameter section (Section 4) defines grids for k+ in {2,...,14}, k- in {2,...,6}, and alpha, beta from 0.1 to 1, but does not mention any validation split. On TREC DL'19, DL'20, and DL-HD, the test sets contain only 43, 54, and 50 queries, respectively. Selecting k+, k-, alpha, beta, and the retained-dimension fraction on the same queries that are then used to report AP/nDCG@10 and to run paired significance tests invalidates the p-values and can inflate the reported 19.50% and 22.35% improvements. The pseudo-irrelevance assumption is a modeling concern, but even if bottom documents are partially relevant, the method might still help empirically; the evaluation protocol is the load-bearing gate because without a clean held-out test set, the empirical evidence for Eq. (4) is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ECLIPSE, a training-free extension of DIME-based dimension importance estimation for dense retrieval. For a query q, it computes a \"moon\" centroid m of the bottom-ranked documents in an initial retrieval list and defines the importance of dimension i as u_q(i) = α(q_i·s_i) − β(q_i·m_i), where s is the pseudo-relevant (\"sun\") centroid or an LLM-generated document embedding (Eq. 4). The method is evaluated on TREC DL'19, DL'20, DL-HD, and RB'04 with ANCE, Contriever, and TAS-B, reporting average improvements over DIME baselines and the full-dimensional baseline (up to 19.50% and 22.35% in AP). The paper also studies the effect of the retrieved-list size k and of random sampling of the negative documents.","tokens_in":13247,"tokens_out":5014,"duration_ms":37517,"significance":"If the central empirical claim holds, ECLIPSE is a simple, generic, and potentially valuable enhancement to DIME-style dimension filtering: it adds one contrastive term to an existing importance function without retraining the encoder. The formulation in Eq. (4) is clear and the idea of using low-ranked documents as pseudo-irrelevant feedback is a natural, non-circular extension of pseudo-relevance feedback. The experimental coverage is broad (three dense retrievers, four benchmarks, PRF and LLM variants), and the ablations in RQ2/RQ3 provide useful information about the role of the negative set. However, the reported evaluation has a load-bearing methodological gap: the hyperparameters and the percentage of retained dimensions appear to be selected on the test queries, and the significance claims in the text are not fully supported by the paper's own table markers. These issues must be addressed before the main claim can be accepted.","major_comments":[{"comment":"No validation split is described anywhere in the paper. Section 5 states that Table 1 reports \"the best result among varying the percentage of retained dimensions,\" and Section 4 defines grids for k+, k−, α, and β without specifying how these are chosen. If these choices are made on the same test queries (43, 54, and 50 queries for DL'19, DL'20, and DL-HD, respectively) and the best configuration is then used for paired significance tests, the p-values are invalid and the reported 19.50% and 22.35% improvements can be inflated by selection. Please add a held-out validation split for all hyperparameters and the dimension-budget selection, or report performance across the whole grid with appropriate multiple-testing correction, and re-run significance tests on the single pre-specified configuration.","section":"Section 5, Table 1; Section 4 (Hyperparameters)"},{"comment":"The text says that \"both PRF Eclipse and LLM Eclipse show statistically significant improvement with respect to their DIME counterparts and Baseline,\" but the table's own superscripts contradict this for several cells. For example, ANCE DL'19 PRF Eclipse AP (0.406) and ANCE DL'20 PRF Eclipse AP (0.408) carry only superscript a (significant vs. Baseline, not vs. PRF DIME), and several other Eclipse cells lack superscript b. The paper should quantify how many of the 24 comparisons are significant and qualify the claim accordingly.","section":"Section 5, Table 1 (significance claims)"},{"comment":"The RQ3 experiment only shows that random sampling from the last 30, 100, or 150 documents is equivalent to using the exact bottom-k− documents; it does not test whether those documents are actually irrelevant to the query. Consequently, the Section 6 conclusion that effectiveness \"stems primarily from their low relevance scores, rather than from any specific semantic properties\" is not established by the reported experiments. Please add an explicit test with documents of known relevance (e.g., labeled non-relevant or topically unrelated documents) as the negative set, or at least discuss why the current experiment is sufficient to separate relevance from semantic content.","section":"Section 3, Eq. (3); Section 5, RQ3 (Table 3)"}],"minor_comments":[{"comment":"The claim that \"Eclipse consistently outperforms the DIMEs baseline, even with reduced dimensionality\" is contradicted by the negative improvements for TAS-B LLM Eclipse on RB'04 (-2.35% AP, -2.04% nDCG@10) and by several non-significant or negative cells for other models; please qualify the claim.","section":"Section 5, Table 2"},{"comment":"The grid for α and β is described as \"positive values increasing linearly from 0.1 up to 1\" but the step size is not specified; please list the exact values used.","section":"Section 4 (Hyperparameters)"},{"comment":"The text refers to \"Figure 3\" when discussing the query \"What is an active margin?\", but the corresponding figure is labeled Fig. 1 in the manuscript; please fix the cross-references.","section":"Section 3"},{"comment":"The phrase \"mAP(AP)\" is redundant; please use one metric name consistently.","section":"Abstract"},{"comment":"The caption states that superscripts a and b indicate significant improvement over Baseline and standard DIMEs, respectively, but some reported best values carry no superscript (e.g., ANCE DL'20 LLM Eclipse nDCG@10 0.665); please clarify whether an unmarked best value is simply not statistically significant.","section":"Section 5, Table 1 caption"},{"comment":"No code or data availability statement is included; adding one would improve reproducibility, especially for the LLM-generated pseudo-relevant documents, which are otherwise not exactly reproducible.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the evaluation protocol: if hyperparameters and the retained-dimension fraction are selected on the test sets, the headline improvements and significance tests are not trustworthy. This is fixable by introducing a validation split or reporting grid-wide results with proper multiple-testing correction. The novelty is incremental relative to DIME, but the contrastive-negative idea is reasonable for this venue. If the authors can demonstrate that the improvement holds under a clean protocol, the paper would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a legitimate, incremental idea whose headline numbers are undermined by the evaluation protocol. ECLIPSE extends DIME by subtracting a centroid of bottom-ranked documents from the relevant signal in the dimension importance function (Eq. 4). The specific application of pseudo-irrelevance feedback to dimension pruning is new, as far as I can tell, even though the algebra is close to Rocchio's negative feedback and contrastive reweighting.\n\nWhat the paper does well: it asks the right research questions and the ablations actually probe the mechanism. RQ2 varies the size of the retrieved collection and shows that adding less relevant documents helps, which supports the intuition that the bottom of the list is a useful negative signal. RQ3 randomizes the selection within the bottom and finds the exact identities matter little, an informative result that the paper interprets sensibly. The method is training-free, model-agnostic, and simple to implement on top of any DIME.\n\nThe soft spot is load-bearing and it is in Section 5: \"We report the best result among varying the percentage of retained dimensions.\" That is tuning on the test set. The hyperparameter grids for k+, k-, alpha, and beta are listed in Section 4 with no mention of a validation split. On TREC DL'19, DL'20, and DL-HD, the test sets have only 43, 54, and 50 queries, respectively. Selecting five hyperparameters on the same queries that are then used for paired significance tests invalidates the p-values and can easily inflate the headline 19.50%/22.35% improvements. The paper also does not provide code, which makes it harder to check how the selection was done. The pseudo-irrelevance assumption itself is not the main worry; the RQ3 results suggest the bottom docs' content matters little, so that mechanism could survive a clean evaluation. But as it stands, the empirical claim in Table 1 is not established.\n\nWho this is for: anyone working on dimension filtering or DIME-based dense retrieval would find the idea worth considering. It deserves a serious referee, but the revision needs to tune on a validation split, report results across all hyperparameter configurations or show stability, and release code. If those are addressed, the method might well be a useful addition to the toolbox.","headline":"ECLIPSE is a plausible incremental extension of DIME that is let down by a tuning-on-test protocol; the method deserves a serious look, but the reported gains are not yet established.","tokens_in":13729,"tokens_out":1985,"would_cite":true,"duration_ms":32099,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free step that subtracts an averaged embedding of low-ranked 'irrelevant' documents from the dimension-importance signal improves dense retrieval across four TREC benchmarks.","keywords":["dense retrieval","dimension importance estimation","pseudo-relevance feedback","pseudo-irrelevance feedback","dimensionality reduction","contrastive learning","embedding dimensions","TREC benchmarks"],"falsifier":"Run ECLIPSE beside its DIME baseline on a query set where the bottom of the initial ranking has been deliberately filled with relevant or topically related documents, and check whether ECLIPSE's AP and nDCG@10 fall at or below the baseline; a consistent drop would show that the method depends on the tail being truly irrelevant, not merely low-scoring.","tokens_in":12781,"feed_emoji":"🔍","tokens_out":9920,"duration_ms":76405,"temperature":0.7,"pith_summary":"Dimension IMportance Estimators (DIMEs) improve dense retrieval by scoring embedding dimensions according to how well they align a query with a pseudo-relevant document, but they never look at what makes a document irrelevant. ECLIPSE claims that subtracting the query's alignment with a 'moon' vector—the centroid of the lowest-ranked documents from the initial retrieval list—from the relevant alignment exposes the noisy dimensions and sharpens the dimension filter. The paper reports that this contrastive step gives statistically significant gains over PRF-based and LLM-based DIMEs and over the full-dimensional baseline on three in-domain and one out-of-domain benchmark, with average-precision improvements up to 19.50% over DIME baselines and 22.35% over using all dimensions. It further claims that what makes the negative documents useful is their low relevance score rather than their semantic content, since randomly sampling from the bottom of the list does not hurt. If this holds, ECLIPSE is a training-free plug-in improvement for existing DIME-based retrieval pipelines.","feed_headline":"Bottom-ranked documents sharpen dense retrieval's dimension filter","feed_subtitle":"A training-free contrastive step lifts average precision by up to 22% over full-dimensional baselines in TREC tests.","key_machinery":"The load-bearing object is the moon vector $m = \\frac{1}{k^{-}}\\sum_{i=0}^{k^{-}-1} d_{k-i}$, the average embedding of the bottom $k^{-}$ documents of the initial retrieval list, used in the Eclipse importance function $u_q(i) = \\alpha(q_i\\cdot s_i) - \\beta(q_i\\cdot m_i)$ alongside the relevant representative $s$. Rewriting the score as $q \\odot (\\alpha s - \\beta m)$ shows that the moon vector enters through a residual vector: it is subtracted from the relevant signal before the query reweights each dimension. The mechanism does the work of suppressing dimensions that align with low-ranked documents while preserving those shared with the query and the relevant representative.","core_discovery":"ECLIPSE's central claim is that query-dependent dimension importance should be estimated contrastively: for each dimension $i$ the score is $u_q(i) = \\alpha(q_i\\cdot s_i) - \\beta(q_i\\cdot m_i)$, where $s$ is the relevant representative embedding (the pseudo-relevant centroid for the PRF variant, or the LLM-generated document for the LLM variant) and $m$ is the moon vector, the centroid of the bottom $k^{-}$ documents from an initial ranked list of 1,000 retrieved documents. The term $q_i\\cdot m_i$ estimates how strongly that dimension carries irrelevant content, and its subtraction suppresses such dimensions. Rewriting the score as $q \\odot (\\alpha s - \\beta m)$ makes the mechanism explicit: the residual 'Eclipse vector' $\\alpha s - \\beta m$ is what remains of the relevant signal after the irrelevant centroid has been removed, and the elementwise product with $q$ then highlights the dimensions the query cares about. On DL19, DL20, DL-HARD, and Robust04, with ANCE, Contriever, and TAS-B, both variants beat their DIME counterparts and the full-dimensional baseline, with LLM-Eclipse the strongest; the gains persist at half the original dimensionality. The paper's RQ3 experiment, where randomly sampling negative documents from the lower end of the list performs like the exact bottom-$k^{-}$, supports the paper's stated conclusion that effectiveness comes from the documents' low relevance scores rather than their semantic content.","pith_inferences":["A natural extension would treat the residual vector $\\alpha s - \\beta m$ as a modified query embedding, making the contrastive subtraction available to any downstream scorer rather than only to dimension filtering.","If only the low rank of the negative documents matters, then even documents from unrelated domains, or random noise vectors scaled to the embedding distribution, might serve as the moon centroid; this is a testable prediction the paper does not run.","For queries whose bottom-ranked documents are actually relevant, or where the top-$k$ list is dominated by one subtopic, the subtraction could suppress useful dimensions, so ambiguous and multi-intent queries are the natural boundary case to test.","The mechanism is conceptually close to contrastive losses in representation learning, which suggests the same pseudo-irrelevance subtraction could be tested in late-interaction models, where the per-token signals play the role of dimensions."],"forward_implications":["Any DIME-based dense retrieval pipeline can adopt ECLIPSE without retraining the encoder, because the moon vector is built from the same initial ranked list the system already produces.","Because the gains persist when only half the embedding dimensions are kept, systems can retain fewer dimensions and still beat the full-dimensional baseline, cutting storage and score-computation cost.","LLM-Eclipse is the strongest variant in the paper's experiments, so pairing an LLM-generated relevant document with an irrelevant-document centroid is the variant that benefits most from the proposed contrastive subtraction.","The bottom tail of the ranked list functions as a reusable pseudo-negative signal even when its exact composition changes, since random sampling from the tail does not hurt.","The method transfers out of domain, with statistically significant gains on Robust04, so the contrastive subtraction is not tied to the MS MARCO training distribution."],"supporting_citations":[{"why":"introduces the DIME framework this paper extends and provides the pseudo-relevant dimension-importance baseline.","marker":"[11]"},{"why":"supplies the Rocchio relevance-feedback formulation underlying the PRF DIME's relevant centroid.","marker":"[33]"},{"why":"provides the LLM that generates the synthetic relevant document used by the LLM DIME baseline.","marker":"[28]"},{"why":"provides the ANCE dense retriever whose embeddings and initial rankings are used in the experiments.","marker":"[38]"},{"why":"provides the Contriever dense retriever used as a second evaluation model.","marker":"[15]"},{"why":"provides the TAS-B dense retriever used as a third evaluation model.","marker":"[13]"},{"why":"supplies the TREC Deep Learning 2019 query set used for in-domain evaluation.","marker":"[8]"},{"why":"supplies the TREC Deep Learning 2020 query set used for in-domain evaluation.","marker":"[7]"},{"why":"supplies the DL-HARD query set used for in-domain evaluation.","marker":"[25]"},{"why":"supplies the TREC Robust 2004 collection used for out-of-domain evaluation.","marker":"[36]"}],"fun_headline_variants":["Irrelevant docs tune dense retrieval's dimension filter","Pseudo-irrelevant centroids suppress noisy dimensions in retrieval","Bottom-ranked docs calibrate dense retrieval's dimension focus","Subtract irrelevant centroid, boost dense retrieval precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's load-bearing premise is that documents at the bottom of the initial ranked list are genuinely irrelevant to the query, so their average embedding reliably points at noisy dimensions; if the tail contains relevant or topically related documents, the subtraction can suppress useful dimensions and hurt retrieval.","fun_headline_variants_meta":{"raw":{"variants":["Irrelevant docs tune dense retrieval's dimension filter","Pseudo-irrelevant centroids suppress noisy dimensions in retrieval","Bottom-ranked docs calibrate dense retrieval's dimension focus","Subtract irrelevant centroid, boost dense retrieval precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2811,"prompt_tokens":1067,"completion_tokens":1744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":1682}},"tokens_in":683,"tokens_out":1744,"duration_ms":12611,"temperature":1.0,"reasoning_tokens":1682,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:44:42.108881+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ECLIPSE beside its DIME baseline on a query set where the bottom of the initial ranking has been deliberately filled with relevant or topically related documents, and check whether ECLIPSE's AP and nDCG@10 fall at or below the baseline; a consistent drop would show that the method depends on the tail being truly irrelevant, not merely low-scoring.","supporting_citations":[{"cited_title":"Prentice Hall, Engle- wood Cliffs, New Jersey (1971)","cited_arxiv_id":null,"evidence_quote":"supplies the Rocchio relevance-feedback formulation underlying the PRF DIME's relevant centroid."},{"cited_title":"In: Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004)","cited_arxiv_id":null,"evidence_quote":"supplies the TREC Robust 2004 collection used for out-of-domain evaluation."}],"review_version":1}