{"id":"08365f89-7687-4e11-90ce-21dffde06b76","arxiv_id":"2505.01452","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Li-LSR learns a static per-token query score from word embeddings, turning query encoding into a table lookup, and reports higher mRR@10 and nDCG@10 than Splade-v3-Doc.","lead":"Sparse retrieval models like SPLADE normally keep query and document representations small, but this paper shows that relaxing that constraint improves accuracy when a modern inverted index stays fast. It then introduces Li-LSR, which replaces the query encoder with a learned per-token score lookup, reporting gains over Splade-v3-Doc on MS MARCO and BEIR.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline margins over Splade-v3-Doc depend on published numbers not re-run under Li-LSR's Seismic/eval harness; the paper's own Table 2 also shows Geng et al. exceeds Li-LSR on BEIR, undercutting the 'state-of-the-art' wording.","rationale":"The reader's weakest assumption is exactly the baseline comparability issue I find most load-bearing. The central claim is an empirical superiority claim of about one to two metric points, and those points come from comparing newly produced Li-LSR scores with published numbers for Splade-v3-Doc and Geng et al. that were not re-run under the same Seismic indexing and evaluation pipeline. Because mRR@10 and nDCG@10 are sensitive to evaluation details, this is a concrete threat to the headline margin, not a philosophical one. The internal contradiction in Table 2 (Geng et al. BEIR 50.4 vs Li-LSR-Big 48.8) reinforces that the 'state-of-the-art' and 'surpasses Geng et al. by 1.0' statements are overreaching; the method may still be a valid contribution and may beat Splade-v3-Doc, but the paper should either verify comparability or soften the claims. I also note the absence of code and checkpoints and the absence of measured Li-LSR latency, but those are secondary to the comparability point. A single re-run experiment would settle the main concern; absent that, CONDITIONAL remains appropriate.","tokens_in":9165,"tokens_out":6148,"duration_ms":62323,"concrete_test":"Obtain the published Splade-v3-Doc (and, if available, Geng et al.) model checkpoints, index MS MARCO and BEIR with the same Seismic exhaustive-search configuration used for Li-LSR (posting-list truncation 8000, alpha=0.4, centroids=1/10 list length), and evaluate with the same tokenizer and metric script. If the mRR@10 and nDCG@10 margins over Splade-v3-Doc shrink to within noise or change sign, the central claim fails. Additionally, re-read Table 2 and confirm the BEIR comparison: Geng et al. reports 50.4 vs Li-LSR-Big 48.8, so any 'surpasses Geng et al.' claim should be restricted to MS MARCO.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 3.3, Table 2) is that Li-LSR-Big improves mRR@10 by 1.0 over Splade-v3-Doc on MS MARCO and nDCG@10 by 1.8 on BEIR. For this claim to hold, the published Splade-v3-Doc and Geng et al. numbers must be directly comparable to Li-LSR numbers produced with Seismic exhaustive search and the authors' evaluation script. That comparability is not established: the baselines are not re-run in this work, they were trained with different teachers, negatives, and pre-training, and mRR/nDCG differences of about one point can arise from query processing details, tokenization, or metric implementations. This is not hypothetical: Table 2 itself shows Geng et al. reaching 50.4 nDCG@10 on BEIR while Li-LSR-Big reaches 48.8, so the text's assertion that Li-LSR-Big 'surpasses ... Geng et al. by 1.0 points' is only true on MS MARCO and false on BEIR; the abstract's 'state-of-the-art effectiveness for both in-domain and out-of-domain' is therefore contradicted by the paper's own table. No code, checkpoints, or measured Li-LSR serving latency are provided, so the inference-free efficiency claim also rests on an unmeasured lookup. The core lookup idea may well be sound, but the reported margins and SOTA wording require either a re-run of baselines under identical conditions or a scaled-back claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates the role of regularization and term expansion in learned sparse retrieval (LSR), and proposes a new inference-free method, Li-LSR, that replaces the neural query encoder with a learned per-token lookup table. The authors first train LSR variants with different losses and regularizers, varying the degree of query and document expansion, and evaluate them on MS MARCO and BEIR using the Seismic inverted index. They report that relaxing regularization improves effectiveness while Seismic still yields efficient retrieval, and they identify query encoding as a latency bottleneck. They then introduce Li-LSR, which learns a static score for each token via a linear projection of word embeddings, and compare it with existing inference-free retrievers, reporting gains over Splade-v3-Doc on both MS MARCO mRR@10 and BEIR nDCG@10. The paper claims state-of-the-art effectiveness for both in-domain and out-of-domain evaluation and argues that the lookup-based query encoding eliminates the neural encoding bottleneck.","tokens_in":9541,"tokens_out":5835,"duration_ms":57908,"significance":"If the reported results are correct, the Li-LSR lookup-based query encoding is a practical and conceptually clean contribution that could meaningfully reduce serving-side latency for learned sparse retrieval while maintaining effectiveness. The systematic study of regularization and expansion levels in Section 3.1 is also a useful empirical contribution, particularly the observation that Seismic's efficiency is resilient to denser representations. The paper's core idea is sound in principle, and the use of BEIR as an external zero-shot benchmark provides some check on overfitting to the in-domain training set. However, the central effectiveness claims are weakened by the absence of re-run baselines under the same harness, the lack of any measured Li-LSR lookup latency, and an internally inconsistent state-of-the-art claim that is contradicted by the paper's own Table 2. The contribution is promising but needs stronger empirical support before the headline claims can be accepted.","major_comments":[{"comment":"The claim that Li-LSR achieves \"state-of-the-art effectiveness for both in-domain and out-of-domain evaluation\" is contradicted by Table 2 itself: Geng et al. [11] reports a BEIR nDCG@10 of 50.4, whereas Li-LSR-Big reaches only 48.8. The sentence in Section 3.3 stating that \"Li-Lsr-Big surpasses by 1.0 points both Splade-v3 and the approach by Geng et al.\" is true only for mRR@10 on MS MARCO; on BEIR, Li-LSR-Big trails Geng et al. by 1.6 points. The abstract and conclusions should be revised to restrict the claim to comparisons against Splade-v3-Doc, or to explicitly acknowledge that Geng et al. remains superior in out-of-domain effectiveness.","section":"Section 3.3, Table 2, and Abstract"},{"comment":"The headline margins over Splade-v3-Doc and Geng et al. rely on published baseline numbers that are not re-run under the same Seismic indexing and evaluation harness used for Li-LSR. Since Li-LSR's scores are produced with Seismic exhaustive search and the authors' own metric script, differences of about one point can be sensitive to tokenization, metric implementation, or indexing parameters. The paper should re-run the baselines under identical conditions, or at minimum provide the evaluation script and a detailed comparison protocol, and ideally report variance or significance tests for the reported margins.","section":"Table 2 and Section 3.3"},{"comment":"The inference-free efficiency claim is not supported by any measurement of Li-LSR's actual query encoding latency. The paper reports 24 ms for BERT-based encoding and uses this to motivate the lookup approach, but it never measures the time of the Li-LSR table lookup, nor does it provide an end-to-end query latency comparison between Li-LSR and Splade-v3-Doc. Without such a measurement, the conclusion that Li-LSR removes the query encoding bottleneck is an assertion rather than a demonstrated result. Please include a direct latency measurement (e.g., average lookup time per query and total query processing time for Li-LSR versus a neural-encoder baseline).","section":"Sections 3.2 and 3.3"},{"comment":"The training procedure for Li-LSR is underspecified. It is not stated whether the word embedding matrix is fine-tuned during training, whether the projection layer is trained jointly with the document encoder, or whether the query-side scores are produced by the projection while documents are encoded by a standard LSR head. These details are necessary to reproduce the method and to understand whether the lookup scores are learned from scratch or distilled from a full contextual encoder. Please clarify the training objective, the architecture, and the exact role of Eq. (2) in the loss computation.","section":"Section 2.2, Eq. (2)"}],"minor_comments":[{"comment":"The capitalization of the method name is inconsistent: \"Li-Lsr\" is used in many places, while the abstract and elsewhere use \"Li-LSR\". Please standardize to \"Li-LSR\".","section":"Throughout"},{"comment":"The header entry \"Genget al.\" should be \"Geng et al.\".","section":"Table 2"},{"comment":"The dataset is referred to as \"MsMarco\" in some places and \"MS MARCO\" in others; please use a single consistent spelling.","section":"Abstract and body"},{"comment":"The three panels use different x-axis ranges for average query time, which makes it harder to compare the efficiency of Small, Medium, and Big models directly; consider using a shared axis or explicitly annotating the ranges.","section":"Figure 1"},{"comment":"The set of token indices is denoted as \"I_V\" but the vocabulary set V is not explicitly defined; please define V and the index set more clearly.","section":"Section 2.2"},{"comment":"No code, checkpoints, or links to an implementation are provided. Releasing the evaluation script and trained models would substantially strengthen reproducibility and allow the community to verify the comparability of the reported numbers.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is interesting and within the scope of SIGIR, but the current framing overstates the results. The most serious issue is that the 'state-of-the-art' claim is internally contradicted by Table 2, which is a fixable but important error. The lack of re-run baselines and the absence of any latency measurement for Li-LSR make the two headline contributions (effectiveness and efficiency) less convincing than the abstract suggests. I recommend major revision with emphasis on re-running baselines under the same harness, measuring Li-LSR latency, and revising the claims to match the evidence. I would also encourage the authors to release code, as the field increasingly expects reproducibility for empirical retrieval papers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is real: Li-LSR learns a static score per token by projecting word embeddings through a linear layer (Eq. 2), which turns query encoding into a table lookup. That is distinct from Geng et al.'s IDF weighting and from contextual SPLADE encoders, and it has clear CPU-cost appeal for production serving. The regularization study in Table 1 is also systematic—four loss/regularizer combinations crossed with three expansion levels, with BEIR as an external zero-shot check. That part is genuinely useful context for the field.\n\nThe soft spots are mostly about the headline claims. Table 2 shows Geng et al. at 50.4 nDCG@10 on BEIR while Li-LSR-Big gets 48.8, so the abstract's \"state-of-the-art effectiveness for both in-domain and out-of-domain\" is simply false as written. Section 3.3's \"surpasses ... Geng et al. by 1.0 points\" is only true on MS MARCO mRR@10, not on BEIR. That internal inconsistency needs a fix.\n\nThe other issue is comparability. The baselines are published numbers, not re-run under the same Seismic exhaustive-search harness or the authors' evaluation script. Different training recipes (single vs. multiple teachers, single vs. multiple negatives, pretraining) can easily account for margins of one to two points. There are no confidence intervals, and no code or checkpoints to allow anyone to check. The efficiency claim also leans on an unmeasured quantity: they compare BERT encoding time to Seismic retrieval time, but never report the actual end-to-end time for Li-LSR's dictionary lookup plus retrieval. That lookup is obviously cheap, but it should be measured.\n\nThese are fixable problems, not a broken method. The idea is sound; the paper just needs either a re-run of baselines under identical conditions or a scaled-back set of claims. Who is this for? Researchers working on efficient learned sparse retrieval, especially those thinking about production CPU costs. The regularization study alone has takeaway value for anyone training sparse encoders.\n\nMy recommendation: yes, send it to serious peer review. It deserves referee time. But the authors should be asked to release code and either re-run key baselines or revise the abstract and Section 3.3 to match what Table 2 actually shows.","headline":"The lookup-table idea is genuinely new and worth testing, but the SOTA claim is contradicted by the paper's own Table 2 and the baseline comparability is too loose for the headline margins.","tokens_in":10036,"tokens_out":2590,"would_cite":true,"duration_ms":25504,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned sparse retriever called Li-LSR replaces neural query encoding with a per-token score lookup and reports the best inference-free effectiveness on MS MARCO and BEIR.","keywords":["Learned sparse retrieval","inference-free retrieval","query encoding bottleneck","term expansion","regularization","Seismic inverted index","MS MARCO","BEIR"],"falsifier":"Re-run Splade-v3-Doc and the model of [11] under the same Seismic exhaustive-search indexing and single-thread evaluation code used for Li-LSR and check whether the 1.0 mRR@10 and 1.8 nDCG@10 gaps persist; if they disappear, the learned lookup is not the cause. A second check is to replace Li-LSR's learned scores with IDF or uniform weights in the same harness and measure the resulting drop.","tokens_in":8981,"feed_emoji":"⚡","tokens_out":9681,"duration_ms":90262,"temperature":0.7,"pith_summary":"The paper argues that in learned sparse retrieval the neural query encoder has become the bottleneck: with modern inverted-index engines, retrieval is fast, while encoding a query still costs tens of milliseconds. The authors introduce Li-LSR, which learns one static score per vocabulary token during training so that, at serving time, encoding a query is just a table lookup. They report that Li-LSR beats the strongest inference-free retriever, Splade-v3-Doc, by 1.0 points of mRR@10 on MS MARCO and 1.8 points of nDCG@10 on BEIR, even though Li-LSR is trained with a single teacher and a single negative per query. The same study shows that relaxing the sparsity regularizer produces more effective sparse encoders without sacrificing retrieval speed.","feed_headline":"No query encoder: table lookup beats SPLADE-v3-Doc","feed_subtitle":"Li-LSR learns one score per token at training time, cutting query latency while lifting MS MARCO and BEIR accuracy","key_machinery":"The term-score lookup table. For each token $x_i$, Li-LSR learns $s_i = \\log(1 + \\mathrm{ReLU}(w^T E_W(x_i) + b))$, a scalar from the word-embedding module alone, and scores a query by summing the table entries of its tokens (with repeated tokens contributing their score once per occurrence). The second mechanism is relaxed regularization: lowering the query and document regularization weights $\\lambda_q$ and $\\lambda_d$ in the distillation loss allows denser, more expanded document representations, which the Seismic inverted index can still retrieve in under a millisecond.","core_discovery":"On the paper's own terms, the central discovery is that a context-free score per token, learned once from the teacher, is enough to make inference-free learned sparse retrieval state of the art. Li-LSR learns a mapping from each token to a scalar via a linear projection of the token's word embedding with a positivity constraint, so the query encoder disappears at serving time and is replaced by an exact lookup in a term-score dictionary. With this mechanism, the paper's Li-LSR-Big configuration reaches 38.8 mRR@10 on MS MARCO and 48.8 nDCG@10 on BEIR, surpassing the previously published Splade-v3-Doc numbers (37.8 and 47.0) and the inference-free IDF approach of [11], despite using far simpler training data.","pith_inferences":["If the learned lookup generalizes as reported, the same idea could be applied to document encoding, turning indexing into a table lookup as well; the paper removes only the query encoder.","The learned per-token scores likely act as a corpus-specific term-importance prior similar in spirit to IDF; a testable extension is to re-learn the lookup table on each BEIR collection and measure how much of the gain comes from the table itself versus the retrieval engine.","Because serving becomes lookup plus index traversal with no GPU and no neural inference, Li-LSR-style retrieval could run on commodity CPUs at scale, an operational consequence the authors do not develop."],"forward_implications":["Query encoding latency falls from about 24 ms per query on a single CPU thread to a dictionary lookup, removing the transformer from the serving path entirely.","Relaxed sparsity (the Big configuration, roughly five times the non-expanded document size) reaches over 99.9% of peak MS MARCO effectiveness in about 0.8 ms per query under Seismic, decoupling effectiveness from retrieval speed.","A training recipe with one teacher and one negative per query is enough for Li-LSR to match or beat inference-free baselines trained with multiple teachers, multiple negatives, and additional pre-training.","On out-of-domain BEIR data, learned per-token scores improve zero-shot retrieval; combining them with IDF helps the medium model but not the large one."],"supporting_citations":[{"why":"Seismic, the inverted-index engine whose speed lets the authors relax sparsity and turns query encoding into the bottleneck.","marker":"[3]"},{"why":"SPLADE-v3-Doc, the main inference-free baseline Li-LSR is compared against and the source of the published numbers it claims to beat.","marker":"[17]"},{"why":"The inference-free IDF retriever that Li-LSR extends and improves upon, including the reported accuracy drop when query encoding is removed.","marker":"[11]"},{"why":"The SPLADE distillation setup that supplies the ranking loss and the Splade-Doc-Distill baseline.","marker":"[8]"},{"why":"MS MARCO, the dataset used for training and for the in-domain mRR@10 evaluation.","marker":"[23]"},{"why":"BEIR, the 13-dataset benchmark used for the out-of-domain nDCG@10 evaluation.","marker":"[27]"},{"why":"The pre-trained checkpoint from which the Li-LSR encoder is initialized.","marker":"[10]"},{"why":"The BERT model whose single-thread encoding time is measured to establish that query encoding is the bottleneck.","marker":"[14]"}],"fun_headline_variants":["Li-LSR: inference-free lookup beats SPLADE-v3-Doc","Query encoder dropped: Li-LSR tops MS MARCO and BEIR","Learn token scores once, then table lookup: Li-LSR new SOTA","No inference at query time: Li-LSR outperforms SPLADE-v3-Doc"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed gains over Splade-v3-Doc and the inference-free baseline of [11] assume that the published numbers for those baselines are directly comparable to the authors' own Li-LSR runs, even though the baselines were trained with different teachers, negatives, and pre-training and were not re-run under the same Seismic evaluation harness.","fun_headline_variants_meta":{"raw":{"variants":["Li-LSR: inference-free lookup beats SPLADE-v3-Doc","Query encoder dropped: Li-LSR tops MS MARCO and BEIR","Learn token scores once, then table lookup: Li-LSR new SOTA","No inference at query time: Li-LSR outperforms SPLADE-v3-Doc"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2113,"prompt_tokens":962,"completion_tokens":1151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1066}},"tokens_in":578,"tokens_out":1151,"duration_ms":11148,"temperature":1.0,"reasoning_tokens":1066,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:03:45.095038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Splade-v3-Doc and the model of [11] under the same Seismic exhaustive-search indexing and single-thread evaluation code used for Li-LSR and check whether the 1.0 mRR@10 and 1.8 nDCG@10 gaps persist; if they disappear, the learned lookup is not the cause. A second check is to replace Li-LSR's learned scores with IDF or uniform weights in the same harness and measure the resulting drop.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BEIR, the 13-dataset benchmark used for the out-of-domain nDCG@10 evaluation."},{"cited_title":"Sampling Methods for Inner Product Sketching","cited_arxiv_id":"2309.16157","evidence_quote":"The SPLADE distillation setup that supplies the ranking loss and the Splade-Doc-Distill baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MS MARCO, the dataset used for training and for the in-domain mRR@10 evaluation."}],"review_version":1}