{"id":"1a6ca74d-51e8-4c2e-b205-db9485f4c680","arxiv_id":"2605.29384","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Sparse autoencoders applied to frozen dense retrievers extract Zipfian latent vocabularies that support BM25 scoring and match or exceed the base model's performance on some tasks.","lead":"The paper shows that dense retrieval models contain internal sparse features extractable by sparse autoencoders that form Zipfian vocabularies ready for direct use in BM25 scoring. This reveals hidden structure in neural retrievers that standard dense scoring does not expose, enabling sparse retrieval without extra supervision or training.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"SAE latents produce continuous activations while standard BM25 requires discrete integer term frequencies; any implicit thresholding or scaling is post-processing contradicting the 'no adjustments' claim.","rationale":"The reader's weakest assumption matches the identified gap exactly. Full text availability allows direct inspection of the scoring implementation, which would either confirm direct unmodified use or reveal the adjustment. This moves the provisional UNVERDICTED verdict to CONDITIONAL pending that verification.","tokens_in":1666,"tokens_out":365,"duration_ms":17670,"concrete_test":"From the methods or experimental sections, extract the precise document scoring equation used with the latent vocabulary. Compare it term-by-term to the canonical BM25 formula; if tf is replaced by a continuous activation, a threshold is applied, or any learned scaling appears, recompute the reported LIMIT and in-domain scores using unmodified integer-count BM25 on binarized latents (activation > 0.5) to measure the performance drop.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim states that SAE latents yield a vocabulary 'directly suitable for classical sparse retrieval scoring via BM25' with 'no retrieval-specific adjustments'. Standard BM25 (Robertson-Sparck Jones) computes scores from integer document term frequencies tf(d,t) and collection statistics. SAE latents output real-valued activations. If the method feeds raw activations into BM25, replaces tf with activation magnitude, or applies any threshold/selection to define 'presence', this is either a non-standard scoring function or post-processing. The abstract and claim provide no explicit mapping or formula, making this the least secure link: the 'term-like' and 'unmodified' properties are asserted but not shown to survive the continuous-to-discrete conversion required by BM25.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that Sparse Autoencoders trained on frozen dense retrievers (single- or multi-vector) extract latent vocabularies exhibiting approximately Zipfian collection statistics. These latents are asserted to be directly suitable for unmodified BM25 scoring without retrieval-specific adjustments, learned expansion, or sparse supervision. The resulting sparse retriever matches or outperforms the base model's single-vector scoring and comparable SPLADE variants, while substantially outperforming the base model on the LIMIT benchmark designed to expose single-vector failures.","tokens_in":1833,"tokens_out":352,"duration_ms":26407,"significance":"If the central empirical claims hold, the work is significant because it shows that dense retrievers internally encode sparse, term-like features with classical collection statistics that can be recovered via SAEs and used for sparse retrieval. This provides a training-free bridge between dense and sparse paradigms and demonstrates that existing dense models contain more indexable structure than their default scoring functions expose. The reported LIMIT gains are noteworthy as they target a documented weakness of dense methods.","major_comments":[{"comment":"Abstract: the claim that the SAE latents are 'directly suitable for classical sparse retrieval scoring via BM25' with 'no retrieval-specific adjustments' and 'without any retrieval-specific adjustments' is load-bearing but unsupported by an explicit mapping. Standard BM25 (Robertson-Sparck Jones) requires integer document term frequencies tf(d,t); SAE outputs are continuous real-valued activations. The manuscript provides no formula showing how raw activations are used as tf without thresholding, scaling, or selection, making it impossible to verify that the procedure is unmodified.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their thorough review and for highlighting the need for greater clarity on the BM25 mapping. We address the major comment below and will revise the manuscript accordingly.","responses":[{"response":"We agree that an explicit mapping is required to substantiate the claim of using unmodified BM25. The current manuscript does not include a formula detailing the conversion from continuous SAE activations to tf values. In the revised version we will add this in the Methods section, specifying the exact procedure (including any thresholding or rounding to integer frequencies) so that readers can verify the inputs to the standard BM25 implementation. This addition will not alter the core claim that no retrieval-specific training or learned adjustments are involved.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim that the SAE latents are 'directly suitable for classical sparse retrieval scoring via BM25' with 'no retrieval-specific adjustments' and 'without any retrieval-specific adjustments' is load-bearing but unsupported by an explicit mapping. Standard BM25 (Robertson-Sparck Jones) requires integer document term frequencies tf(d,t); SAE outputs are continuous real-valued activations. The manuscript provides no formula showing how raw activations are used as tf without thresholding, scaling, or selection, making it impossible to verify that the procedure is unmodified."}],"tokens_in":1303,"tokens_out":287,"duration_ms":16893,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that sparse autoencoders trained on the internal activations of frozen dense retrievers produce a set of latent features whose collection statistics are roughly Zipfian and that can be used for BM25 scoring.\n\nThis extraction requires no retrieval-specific objective or supervision, which is the clearest new element relative to earlier SAE work in IR and to supervised sparse retriever training. The paper tests the method on both single-vector and multi-vector base models, reports that the resulting sparse scores match or exceed the dense baseline on standard tasks, and shows clearer gains on the LIMIT benchmark that was built to expose single-vector weaknesses. It also positions the results against SPLADE-style models. Those empirical patterns are the part that could matter for hybrid systems that want sparse indexes without extra training.\n\nThe soft spot is the mapping from SAE activations to BM25 inputs. SAE latents are continuous real values while standard BM25 uses integer term frequencies. The abstract claims the latents are \"directly suitable\" with \"no retrieval-specific adjustments,\" yet the stress-test concern is valid: any thresholding, scaling, or substitution of activation magnitude for tf counts as a change to the scoring function. Without the explicit conversion rule or pseudocode, it is difficult to judge how unmodified the BM25 application really is. The outperformance numbers would also be easier to evaluate with error bars and a fuller set of ablations.\n\nThis paper is for IR groups that already run dense retrievers and want a lightweight way to add sparse retrieval or to inspect what the dense model has learned. A reader focused on practical hybrid architectures or on post-hoc interpretability would get the most from the results.\n\nIt deserves peer review so the conversion details and experimental controls can be checked.","headline":"SAEs on frozen dense retrievers extract Zipfian latent terms usable in BM25, but the continuous activation to discrete frequency step is under-specified.","tokens_in":2306,"tokens_out":425,"would_cite":false,"duration_ms":27187,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Dense retrievers contain sparse vocabularies that sparse autoencoders extract and make directly usable with standard BM25.","keywords":["dense retrieval","sparse autoencoders","BM25","latent vocabulary","information retrieval","Zipfian statistics","neural retrievers","sparse retrieval"],"falsifier":"Apply the extracted latent terms to BM25 on a held-out collection such as MS MARCO and observe that effectiveness falls substantially below the original dense retriever's single-vector scores.","tokens_in":2578,"feed_emoji":"🔍","tokens_out":626,"duration_ms":17581,"temperature":0.7,"pith_summary":"The paper establishes that dense retrieval models, whether single-vector or multi-vector, learn internal representations that decompose into sparse features with Zipfian collection statistics. Sparse autoencoders trained on frozen retriever activations produce these latent terms without retrieval-specific objectives or supervision. The resulting vocabulary plugs straight into classical BM25 scoring and matches or exceeds the base model's single-vector performance on standard tasks while improving on datasets that expose single-vector weaknesses. This reveals that the models hold more indexable structure than their usual scoring functions use.","feed_headline":"Sparse autoencoders turn dense retrievers into BM25 vocabularies","feed_subtitle":"Extracted latents show Zipfian stats and support classical sparse scoring with no extra training or supervision.","key_machinery":"Latent Terms: the sparse features recovered by sparse autoencoders from dense retriever activations, which exhibit Zipfian statistics and serve as a ready vocabulary for BM25.","core_discovery":"Models trained for dense retrieval learn representations that sparse autoencoders can decompose into retrieval-ready sparse features; when the autoencoders run on frozen retrievers with no retrieval adjustments, the extracted latents form a vocabulary whose collection statistics are approximately Zipfian and therefore directly compatible with unmodified BM25 scoring.","pith_inferences":["Hybrid retrieval systems could be constructed by extracting the sparse component from an existing dense model rather than training a separate sparse encoder.","The Zipfian alignment may indicate that dense training implicitly captures term-frequency regularities that classical IR methods exploit explicitly.","The approach could be tested on collections with different term distributions to check whether the extracted vocabularies remain BM25-compatible outside the original training domain."],"forward_implications":["Sparse retrieval becomes available from any dense retriever with zero sparse supervision or expansion training.","The same procedure works on both single-vector and multi-vector dense models.","Retrieval effectiveness on the LIMIT task exceeds the base dense model's single-vector scores.","Performance matches or exceeds comparable SPLADE variants trained with explicit sparse objectives."],"fun_headline_variants":["Dense retrievers hide BM25 vocabularies revealed by sparse autoencoders","Any dense retriever yields BM25 vocabularies via sparse autoencoders","Latent terms support BM25 scoring from frozen dense retrievers","Autoencoders reveal Zipfian structure in dense retrieval models"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The autoencoder latents recovered from frozen dense retrievers are term-like enough and have collection statistics aligned enough to the corpus that they work in standard BM25 without post-processing or learned weights.","fun_headline_variants_meta":{"raw":{"variants":["Dense retrievers hide BM25 vocabularies revealed by sparse autoencoders","Any dense retriever yields BM25 vocabularies via sparse autoencoders","Latent terms support BM25 scoring from frozen dense retrievers","Autoencoders reveal Zipfian structure in dense retrieval models"]},"model":"grok-4.3","cost_usd":0.008439,"raw_usage":{"total_tokens":3784,"prompt_tokens":602,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":84387000,"prompt_tokens_details":{"text_tokens":602,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3109,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":602,"tokens_out":73,"duration_ms":23011,"temperature":1.0,"reasoning_tokens":3109,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T05:44:25.580918+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Apply the extracted latent terms to BM25 on a held-out collection such as MS MARCO and observe that effectiveness falls substantially below the original dense retriever's single-vector scores.","supporting_citations":[],"review_version":1}