{"id":"cf0df1b0-ee4c-4e1f-ba24-27bf12d07328","arxiv_id":"2606.02814","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Supervised bi-encoder retrievers encode document-level relevance priors from annotation biases, producing a findability gap for documents lacking favored features such as comprehensiveness and mainstream topic coverage.","lead":"Neural retrievers trained on annotated query-document pairs learn a query-independent relevance prior favoring certain document types like comprehensive mainstream summaries due to biases in how annotations are collected. This creates a findability gap where some genuinely relevant documents become systematically harder to retrieve.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader correctly flagged the isolation step as the key assumption. After reviewing the full described protocol (classifier on frozen embeddings, cross-model consistency, matched controls, and generalization checks), that assumption appears adequately supported by the experimental design; no further load-bearing gap is evident.","tokens_in":1746,"tokens_out":300,"duration_ms":18431,"concrete_test":"Recompute the findability-gap statistics (e.g., rank correlation or recall@K difference) after additionally matching on document length, Flesch readability, and topic entropy using the same matching procedure already employed in the paper; if the gap remains statistically significant at the same magnitude, the isolation of the annotation-bias prior is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the empirical finding that simple linear classifiers trained on frozen bi-encoder document embeddings recover a query-independent signal predictive of annotation status, that this signal is consistent across three retrievers, generalizes to held-out documents, and produces a measurable findability gap even under matched-document controls. The abstract and described method give no indication of circularity in the construction, data leakage between the prior estimator and the retrieval evaluation, or failure of the matched comparisons to hold other document properties fixed. The LLM-based characterization of favored vs. disfavored documents is presented as supporting qualitative evidence rather than as a formal proof, which is appropriate for the claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript claims that supervised bi-encoder retrievers implicitly learn query-independent relevance priors as a side effect of training on annotated data, where annotation selection favors certain document types. These priors are recovered by training simple classifiers on frozen document embeddings, generalize to unseen documents, are consistent across three state-of-the-art retrievers, and produce a measurable findability gap (documents with lower prior scores are harder to retrieve even when relevant). The effect is weaker and less consistent for BM25, persists under matched-document controls, and is supported by LLM-based qualitative analysis showing preference for comprehensive, mainstream documents over niche or technical ones.","tokens_in":1877,"tokens_out":452,"duration_ms":17184,"significance":"If the empirical findings hold, the work is significant for information retrieval because it identifies a structural limitation of supervised retrieval: models internalize annotation biases rather than learning pure relevance. This has implications for fairness, robustness, and evaluation practices. Strengths include the multi-retriever, multi-benchmark design, matched-document controls that hold other properties fixed, cross-model consistency checks, and the combination of quantitative retrieval metrics with qualitative LLM explanations. The approach avoids circularity by using frozen embeddings and simple classifiers.","major_comments":[],"minor_comments":[{"comment":"The abstract and described method indicate use of held-out documents and matched controls, but the manuscript should explicitly report the exact data splits, number of documents per benchmark, and any statistical tests (e.g., significance of the findability gap) to allow full assessment of generalizability.","section":"Methods / Experiments"},{"comment":"Clarify the precise definition and computation of the 'findability gap' metric (e.g., how retrieval performance is compared for high- vs. low-prior documents under matched conditions) to ensure the quantitative claim is unambiguous.","section":"Evaluation"},{"comment":"The LLM-based explanations are presented as supporting evidence; include inter-annotator agreement or validation steps for the qualitative categories (comprehensive vs. niche) to strengthen that component.","section":"Qualitative Analysis"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their thorough and supportive review, which accurately captures the core claims, methodology, and implications of our work. The recommendation for minor revision is appreciated. Since no specific major comments were raised, we have no point-by-point rebuttals to provide and will incorporate minor clarifications in the revised manuscript where they improve readability.","responses":[],"tokens_in":1314,"tokens_out":87,"duration_ms":7219,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to take from this paper is that supervised neural retrievers pick up a document-level relevance prior from annotation biases in the training data, and this prior makes some genuinely relevant documents harder to retrieve than others.\n\nThe new part is the demonstration that you can recover this prior with simple classifiers on the frozen embeddings, that it generalizes to unseen documents, stays consistent across three different retrievers, and produces a measurable findability gap even when documents are matched on other characteristics. The LLM explanations add a qualitative layer by pointing to favored documents being comprehensive and mainstream while niche or technical ones are disfavored. It does well to include the BM25 comparison, where the effect is weaker, which helps argue that it's tied to the supervised training.\n\nThe soft spots are around how cleanly the probing isolates the annotation bias versus other embedding properties, and whether the matched-document controls are tight enough to rule out confounds like topic or length. The abstract describes the setup, but without seeing the exact statistical tests or ablation results, it's difficult to judge the robustness of the gap size. The claim that this is a structural limitation is reasonable but would be stronger with more quantification of real-world impact.\n\nThis work is for people studying biases in retrieval systems and how training data affects model behavior. A reader focused on IR evaluation or fairness in search and RAG would find the empirical results relevant. The paper shows clear thinking in its experimental design and engages with the literature on annotation bias, so it merits a serious referee even if revisions are needed on the details.\n\nI would recommend sending this to peer review.","headline":"Supervised bi-encoders pick up document-level annotation biases that create a measurable findability gap even for relevant items.","tokens_in":2352,"tokens_out":392,"would_cite":false,"duration_ms":22127,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Supervised neural retrievers encode query-independent relevance priors from annotation biases that create a findability gap for less favored documents.","keywords":["neural retrievers","relevance priors","annotation bias","findability gap","bi-encoder","information retrieval","supervised dense retrieval"],"falsifier":"A test in which documents matched for relevance but differing in prior score show no systematic difference in retrieval rank across the models would falsify the claim.","tokens_in":2653,"feed_emoji":"","tokens_out":650,"duration_ms":15597,"temperature":0.7,"pith_summary":"The paper investigates whether neural retrievers learn document-level preferences independent of any query. It demonstrates that these priors generalize to unseen documents and remain consistent across different supervised models. The priors arise as a side effect of training on annotated pairs where annotation protocols favor certain document types. This produces a findability gap in which genuinely relevant documents with lower prior scores are harder to retrieve. The effect is pronounced in dense retrievers but weaker in BM25, and LLM analysis links the bias to preferences for comprehensive mainstream content over niche material.","feed_headline":"Neural retrievers favor certain documents due to annotation bias","feed_subtitle":"This learned preference creates a findability gap for niche but relevant content even when it matches the query.","key_machinery":"The query-independent relevance prior, isolated by training classifiers on frozen document embeddings from the retriever representation space.","core_discovery":"Supervised bi-encoder retrievers implicitly learn a document-level relevance prior encoded in their representation space as a side effect of training on annotated data. This prior is estimated by training simple classifiers on frozen document embeddings, generalizes to unseen documents, and proves consistent across models and benchmarks. It creates a findability gap where documents with lower prior are systematically harder to retrieve even when relevant, an effect that persists under matched-document controls. LLM-based explanations show that judged-relevant documents tend to be comprehensive self-contained summaries of mainstream topics while niche or technical content is often left unjudg","pith_inferences":["Future training pipelines could mitigate the gap by deliberately including more diverse unjudged documents in annotation pools.","Standard IR evaluation may need separate metrics for document findability to isolate this effect from relevance.","The same annotation-induced priors could appear in other supervised embedding models outside retrieval.","Long-tail or technical queries may suffer disproportionately from the findability gap."],"forward_implications":["Documents with lower relevance prior are systematically ranked lower even when genuinely relevant.","The bias appears consistently across multiple state-of-the-art supervised retrievers and IR benchmarks.","The effect is weaker and less consistent in BM25 than in dense retrievers.","Annotation selection favors comprehensive mainstream documents over niche or fragmentary ones.","Retrievers rank documents according to these learned features independently of actual query relevance."],"fun_headline_variants":["Neural retrievers learn document relevance priors from annotations","Annotation bias creates findability gaps for niche documents","Supervised bi-encoders encode consistent relevance priors","Document preferences in neural retrieval stem from annotation protocols"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Training simple classifiers on frozen document embeddings isolates a query-independent relevance prior caused by annotation selection bias rather than other embedding properties.","fun_headline_variants_meta":{"raw":{"variants":["Neural retrievers learn document relevance priors from annotations","Annotation bias creates findability gaps for niche documents","Supervised bi-encoders encode consistent relevance priors","Document preferences in neural retrieval stem from annotation protocols"]},"model":"grok-4.3","cost_usd":0.004112,"raw_usage":{"total_tokens":2121,"prompt_tokens":737,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":41124500,"prompt_tokens_details":{"text_tokens":737,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1327,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":737,"tokens_out":57,"duration_ms":10726,"temperature":1.0,"reasoning_tokens":1327,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T12:19:05.776878+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A test in which documents matched for relevance but differing in prior score show no systematic difference in retrieval rank across the models would falsify the claim.","supporting_citations":[],"review_version":1}