{"id":"1b231126-7f46-49a6-a376-053066aeba45","arxiv_id":"2504.18151","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"For learned sparse retrieval, an encoder-decoder model that copies input to the decoder and max-pools MLM logits over all positions outperforms encoder-only and decoder-only backbones on MS MARCO.","lead":"This paper tests three transformer architectures for learned sparse retrieval and finds that an encoder-decoder with a multi-token decoding head works best on MS MARCO. It also shows that decoder-only models underperform at small scale but can beat a 66M encoder when scaled to 1.3B parameters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Architecture ranking is not isolated: the 0.3-point NDCG advantage of EncDec.MultiTokens over DistilSplade-max confounds architecture with model scale and pretraining, and no significance test is reported for the Table 2 comparison.","rationale":"The paper is a serious, well-reported empirical study with released code, honest limitations, and internally consistent within-family comparisons. The strongest claim, however, requires that architecture rather than scale or pretraining drives the ranking. That condition is least secure in Table 2: the headline encoder-decoder result is compared with a smaller, differently pretrained encoder-only baseline, and the difference is a near-tie without significance testing. The only same-pretraining encoder-only variant is much weaker and is dismissed with an untested head-incompatibility hypothesis, which further shows that the comparison does not cleanly separate architecture from pretraining compatibility. This matches the reader's weakest_assumption, so I agree with the conditional verdict rather than suggesting acceptance or rejection. The concern is addressable: a matched-scale, MLM-compatible encoder-only control, or at minimum a paired significance test on the 0.3 NDCG gap, would determine whether the architecture claim survives.","tokens_in":14212,"tokens_out":11327,"duration_ms":117360,"concrete_test":"Train an encoder-only LSR model with MLM-compatible pretraining and parameter scale at least matching Flan-T5-base (e.g., BERT-large-uncased, ~340M parameters) using the same MLM-MultiTokens head, 600k training steps, MarginMSE distillation from MiniLM-L-6-v2, and FLOPs regularization as EncDec.MultiTokensFlanT5-base; compare NDCG@10 on MS MARCO dev. If this encoder-only model reaches or exceeds 43.6, the reported encoder-decoder advantage is explained by scale or pretraining rather than by the encoder-decoder architecture. Also report a paired bootstrap or paired t-test over the dev queries for the 0.3-point gap between Table 2 row 6 and row 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the encoder-decoder backbone with multi-token decoding is the best of the three backbones for learned sparse retrieval. The decisive evidence is Table 2, where EncDec.MultiTokensFlanT5-base scores 43.6 NDCG@10 on MS MARCO dev versus 43.3 for DistilSplade-max (DistilBERT-base). This is a 0.3-point gap with no paired significance test: the daggers in Table 3 only test MLM-MultiTokens versus MLM-SingleToken, not the architecture comparison. More importantly, the two models differ in parameter count (Flan-T5-base ~248M vs DistilBERT-base ~66M) and pretraining objective (span-corruption encoder-decoder vs masked-LM encoder). The paper's own same-pretraining encoder-only variant, Enc.FlanT5-base (Table 2 row 3), scores only 36.0, and Section 5 RQ2 explains this as an 'incompatibility' between the MLM head and T5's encoder. That explanation is post hoc, and it means the only comparisons that hold pretraining fixed use different fractions of the same T5 checkpoint, which are not matched in parameter count either. No row in Table 2 or Table 4 isolates architecture from scale and pretraining, so the architecture ranking is not established by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates learned sparse retrieval (LSR) using transformer backbones of three architectural types: encoder-only, decoder-only, and encoder-decoder. It proposes a 'multi-tokens decoding' approach in which the full input is fed to the decoder (prepended with a start token) and an MLM head with max-pooling over token positions produces sparse lexical representations. The authors evaluate zero-shot and fine-tuned performance on MS MARCO and TREC DL 2019/2020, comparing against Splade-family baselines and various heads (MLP, MLM-single-token, MLM-multi-tokens). The main reported findings are: (1) zero-shot LLMs produce poor sparse representations, with term expansion actually hurting recall; (2) decoder-only models become competitive only when scaled to large sizes; (3) copying the encoder input to the decoder improves over single-token decoding for encoder-decoder models; and (4) the encoder-decoder backbone with multi-tokens decoding achieves the best effectiveness among the three backbone types, with lower FLOPs than DistilSplade-max. The paper releases code and reproduces baseline numbers.","tokens_in":14465,"tokens_out":8486,"duration_ms":75731,"significance":"If the architecture ranking were cleanly established, the paper would provide a useful recipe for applying larger pretrained models to learned sparse retrieval and would extend the LSR literature beyond small encoder-only models. The head comparison within the encoder-decoder family (MLM-MultiTokens vs. MLM-SingleToken vs. MLP, Table 3) is a clean within-model contribution with significance tests. The zero-shot analysis is falsifiable and adds evidence to the growing picture that LLM logits are not directly suited for sparse retrieval. However, the central claim about the encoder-decoder architecture being 'best among the three backbones' is currently confounded with model scale, pretraining objective, vocabulary size, and hyperparameters, as detailed in the major comments; therefore the headline conclusion is not yet established.","major_comments":[{"comment":"The central claim that the encoder-decoder architecture with multi-tokens decoding is the best of the three backbones is not established because the comparison confounds architecture with parameter count and pretraining. EncDec.MultiTokensFlanT5-base (Flan-T5-base, 248M parameters, span-corruption pretraining) is compared to DistilSplade-max (DistilBERT-base, 66M parameters, masked-LM distillation) and Dec.MultiTokensOPT-350M (350M parameters, causal-LM pretraining); no row in Table 2 or Table 4 isolates architecture while holding scale and pretraining fixed. Within the same T5 family, Enc.FlanT5-base uses only the encoder stack, Dec.FlanT5-base only the decoder stack, and EncDec.FlanT5-base both stacks, so parameter counts differ as well. The paper should either report controlled comparisons (e.g., matched-size T5 encoder-only and decoder-only models, or a DistilBERT-scale T5 variant) or substantially qualify the architecture ranking.","section":"Section 5, RQ2, Table 2"},{"comment":"The post hoc explanation that the weakness of Enc.FlanT5-base (NDCG@10 36.0) is due to 'incompatibility' between the MLM head and T5's encoder is not tested, and Appendix Table 6 shows that zero-shot EncDec.MultiTokensFlanT5-base produces an empty representation (RR@10 0, R@1k 0), which suggests the T5 decoder's LM head may also be miscalibrated for LSR. Because row 3 is used to argue that the encoder-only backbone is inferior, the explanation for its failure must be validated (e.g., by training the MLM head from scratch on the T5 encoder, or by reporting diagnostic sparsity statistics) before the architecture comparison can be interpreted.","section":"Section 5, RQ2, Table 2 row 3; Appendix Table 6"},{"comment":"The claimed 'same training configuration' is not actually identical across backbones: OPT uses a lower learning rate (1e-6) than Flan-T5 (5e-4), and the vocabulary sizes differ (DistilBERT 30k, T5 32k, OPT 50k), which can affect the FLOPs regularization and the sparse-representation search space. Moreover, no significance tests are reported for the Table 2 backbone comparisons; the daggers in Table 3 apply only to the head comparisons within EncDec.FlanT5-base. The paper should report paired significance tests or confidence intervals for the backbone differences, or explicitly state that these differences are not statistically significant.","section":"Section 3, Training Configuration; Table 2"}],"minor_comments":[{"comment":"The two 'Figure??' references in the descriptions of Decoder-only multi-tokens and Encoder-Decoder multi-tokens should be replaced with the correct figure number.","section":"Section 2.1"},{"comment":"In the definition of the MLP head, the function name 'importanceMLM' appears to be a typo for 'importanceMLP'; please also harmonize 'ReLu' and 'ReLU' across equations.","section":"Equation (7)"},{"comment":"The sentence 'the nDCG@10 of the decoder-only backbone increased to 60.9 on DL2020' contradicts Table 4 row 8, which reports 69.5; this appears to be a typo.","section":"Section 5, RQ4, Table 4"},{"comment":"The dagger and double-dagger footnote is repeated and ambiguous; for example, 'A † indicates paired t-test p < 0.05. († indicates test between MLM-MT and MLM-ST)' should be condensed into one clear statement.","section":"Table 3 caption"},{"comment":"The zero-shot EncDec.MultiTokensFlanT5-base row reports RR@10 0 and R@1k 0 without explanation; if this arises from an all-zero sparse vector, that should be stated and its cause discussed, as it bears on the zero-shot findings in RQ1.","section":"Appendix, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's cleanest contribution is the head comparison in Table 3 and the multi-tokens decoding recipe within the encoder-decoder family; the cross-backbone ranking in Table 2, which drives the headline claim, needs controlled experiments or a heavily qualified framing. The all-zero zero-shot row for EncDec.MultiTokensFlanT5-base in Table 6 is surprising and may indicate a technical issue worth verifying before the paper is finalized."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the LSR architecture paper. The short version: it's a solid empirical study with a real new adaptation, but the headline ranking isn't as clean as the authors claim.\n\nWhat's new: they take the MLM-MultiTokens head from SPLADE and attach it to decoder-only and encoder-decoder backbones, feeding the input to both encoder and decoder so the decoder produces a hidden state per token rather than a single start-token state. That's a sensible move, and the systematic three-architecture comparison with the same training recipe, same hard negatives, and reproduced baselines is genuinely useful. The code link helps. They also report zero-shot results and are honest about weaknesses and concurrent work.\n\nWhere I'd push back: the main conclusion--that EncDec.MultiTokens Flan-T5-base beats DistilSplade-max--rests on a 0.3-point NDCG@10 gap on MS MARCO dev and a 0.4-point gap on DL2019, with no significance test for that specific comparison. The daggers in Table 3 only address MLM-MT vs MLM-ST. More importantly, the comparison confounds architecture with parameter count (Flan-T5-base ~248M vs DistilBERT-base ~66M) and pretraining objective (span corruption vs masked LM). Their own same-pretraining encoder-only variant, Enc.FlanT5-base, is far worse (36.0), and the 'incompatibility' explanation is post hoc. So the architecture ranking is plausible but not established.\n\nAlso, the intro says decoder-only can match encoder-only when scaled to '3 billion parameters,' but the largest fine-tuned decoder in the tables is OPT-1.3B. The 2.7B/6.7B numbers are zero-shot only. That overstates the fine-tuned evidence.\n\nThese are addressable, not fatal. The paper deserves a serious referee. I'd ask for controlled experiments that hold parameter count or pretraining family fixed, paired significance tests on the architecture comparisons, and a more careful scaling claim. The conditional verdict from the reader is right.\n\nBring it to the reading group if you want a case study in how scale and pretraining sneak into architecture comparisons.\n\nRecommendation: send to peer review; expect revision.","headline":"Useful three-backbone LSR study with a real multi-token decoding adaptation, but the main architecture ranking is confounded by scale and pretraining and needs tighter controls.","tokens_in":15055,"tokens_out":2367,"would_cite":true,"duration_ms":22259,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-token decoding turns encoder-decoder transformers into the strongest learned sparse retrievers.","keywords":["learned sparse retrieval","sparse representations","transformer architectures","encoder-decoder models","multi-token decoding","term expansion","zero-shot retrieval","model scaling"],"falsifier":"Train an encoder-only masked-language model with about 248M parameters, the same teacher and hard negatives, the same FLOPs regularizer, and the same 600k steps as in Table 2; if it reaches or exceeds the 43.6 NDCG@10 of EncDec.MultiTokens FlanT5-base on MS MARCO dev, the paper's architecture ranking is confounded by scale or pretraining rather than by the encoder-decoder design.","tokens_in":13964,"feed_emoji":"🔍","tokens_out":8168,"duration_ms":72006,"temperature":0.7,"pith_summary":"This paper asks whether learned sparse retrieval — encoding queries and documents into weighted vocabulary vectors for inverted-index search — can move beyond the small encoder-only transformers that currently dominate it. The authors show that zero-shot large language models produce poor sparse vectors: they expand input text with noisy terms and lose recall. Under a shared fine-tuning recipe, an encoder-decoder backbone that feeds the full input to both encoder and decoder, then pools masked-language-model logits across decoder token positions, outperforms the leading encoder-only baseline (DistilSplade-max) on MS MARCO dev and TREC Deep Learning 2019, while using fewer FLOPs. Decoder-only models lag at small scale but catch up when scaled to over a billion parameters. The practical upshot is that the architecture and the way the sparse head sees the sequence matter more than raw model size.","feed_headline":"Multi-token decoding wins sparse retrieval","feed_subtitle":"Feeding the whole input to both encoder and decoder beats the best encoder-only baseline on MS MARCO with less compute.","key_machinery":"The load-bearing mechanism is multi-tokens decoding: instead of passing only a start token to the decoder — the single-token bottleneck of encoder-decoder sentence embeddings — the full input is copied into the decoder, so the model produces one hidden state per input token. An MLM head then turns each hidden state into a vocabulary-size logit vector and a row-wise max pooling collapses them into a single sparse vector. The decoder's causal attention over the copied input lets the representation aggregate lexical evidence from the entire sequence, which the paper argues is what lets encoder-decoder outperform encoder-only and decoder-only backbones under the same training regime.","core_discovery":"The paper's central claim is that the encoder-decoder transformer with multi-tokens decoding yields the most effective learned sparse representations among the three backbone families, under controlled fine-tuning. Concretely, the model takes the complete token sequence as encoder input, prepends the start token and feeds the same sequence to the decoder, and then applies an MLM head to every decoder hidden state, max-pooling the resulting vocabulary logits across positions to form one sparse vector per text. In the paper's experiments this design reaches 43.6 NDCG@10 on MS MARCO dev and 71.4 on TREC DL 2019, edging out the encoder-only DistilSplade-max (43.3 and 71.0) while spending fewer FLOPs (2.8 vs 4.0). The paper also reports that zero-shot LLMs generate noisy expansions and degraded recall, and that decoder-only models need roughly a billion parameters before they match encoder-only performance on the tested benchmarks.","pith_inferences":["A testable extension is to apply multi-tokens decoding to dense retrieval heads: feeding the full sequence to the decoder and pooling could similarly unify encoder-decoder backbones for both sparse and dense representations, potentially improving hybrid retrieval without separate encoders.","The paper's finding that a sharper teacher distribution hurts decoder-only students more than encoder-decoder students hints that distillation robustness, not just architecture, governs which backbone scales best; future work could measure how score-sharpness interacts with causal attention.","Because the paper does not match parameter count or pretraining objective across backbones, a direct implication is that the 1.3B decoder-only result may be driven by scale alone; the obvious next experiment is a same-size encoder comparison to isolate the architectural contribution.","The zero-shot expansion failure suggests that term expansion in LSR should be conditioned on retrieval objectives, not just language-model likelihood; a promising design is to restrict LLM expansions to a retrieval-specific vocabulary or to train a lightweight gating head on top of the MLM logits."],"forward_implications":["Encoder-decoder backbones can become the default starting point for learned sparse retrieval, since they combine bidirectional and causal attention without needing billion-scale parameters.","Decoding strategies matter as much as pretraining: a small change in how the decoder sees the input (full sequence vs. single token) shifts NDCG by several points, so future LSR designs should treat the sparse-representation head and decoding pattern as first-class design choices.","Zero-shot prompting of LLMs for sparse retrieval is not reliable as-is: expanding with all input-token logits injects noise, so retrieval-oriented fine-tuning or constrained decoding is necessary before LLMs help.","Scaling decoder-only models is a viable route to strong sparse retrieval: OPT-1.3B matches or exceeds the encoder-only baseline on DL2019 and DL2020, suggesting that causal LMs become effective sparsifiers only at larger scale."],"supporting_citations":[{"why":"Splade v2 defines the MLM head with max pooling and supplies the DistilSplade-max baseline that the encoder-decoder model beats.","marker":"[12]"},{"why":"Sentence-T5 establishes the single-token decoder encoding pattern that EncDec.SingleToken uses and that multi-token decoding is compared against.","marker":"[24]"},{"why":"PromptReps provides the zero-shot LLM prompting baseline whose expansion behavior the paper diagnoses as noisy.","marker":"[37]"},{"why":"Supplies the MarginMSE distillation loss and the hard-negative teacher setup used to train every student model.","marker":"[14]"},{"why":"RankLLaMA-13B is the larger teacher used in the teacher-scaling experiments.","marker":"[17]"},{"why":"MiniLM-L-6-v2 is the smaller teacher whose score distribution anchors the main distillation setting.","marker":"[31]"},{"why":"FLOPs regularization is the differentiable sparsity term in the training objective.","marker":"[27]"},{"why":"Splade-v3 provides the training data and hard negatives, and documents the MarginMSE sensitivity to teacher score sharpness.","marker":"[15]"}],"fun_headline_variants":["Encoder-decoder wins learned sparse retrieval","Multi-token decoding boosts sparse retrieval efficiency","Decoder-only scales to match encoder-only in sparse retrieval","Sparse retrieval: encoder-decoder beats encoder-only with fewer FLOPs","Zero-shot LLMs struggle at learned sparse retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The architecture ranking assumes that differences in parameter count and pretraining between the backbones (66M DistilBERT, 248M Flan-T5, 350M OPT) do not drive the results, since the paper controls training data, loss, and regularization but not model scale or pretraining objective.","fun_headline_variants_meta":{"raw":{"variants":["Encoder-decoder wins learned sparse retrieval","Multi-token decoding boosts sparse retrieval efficiency","Decoder-only scales to match encoder-only in sparse retrieval","Sparse retrieval: encoder-decoder beats encoder-only with fewer FLOPs","Zero-shot LLMs struggle at learned sparse retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1665,"prompt_tokens":889,"completion_tokens":776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":701}},"tokens_in":505,"tokens_out":776,"duration_ms":6715,"temperature":1.0,"reasoning_tokens":701,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:23:35.219584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an encoder-only masked-language model with about 248M parameters, the same teacher and hard negatives, the same FLOPs regularizer, and the same 600k steps as in Table 2; if it reaches or exceeds the 43.6 NDCG@10 of EncDec.MultiTokens FlanT5-base on MS MARCO dev, the paper's architecture ranking is confounded by scale or pretraining rather than by the encoder-decoder design.","supporting_citations":[{"cited_title":"In: Findings of the Association for Computational Linguistics: ACL 2022","cited_arxiv_id":null,"evidence_quote":"Sentence-T5 establishes the single-token decoder encoding pattern that EncDec.SingleToken uses and that multi-token decoding is compared against."},{"cited_title":"In: Al-Onaizan, Y., Bansal, M., Chen, Y.N","cited_arxiv_id":null,"evidence_quote":"PromptReps provides the zero-shot LLM prompting baseline whose expansion behavior the paper diagnoses as noisy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MarginMSE distillation loss and the hard-negative teacher setup used to train every student model."},{"cited_title":"Advances in neural information processing systems33, 5776–5788 (2020)","cited_arxiv_id":null,"evidence_quote":"MiniLM-L-6-v2 is the smaller teacher whose score distribution anchors the main distillation setting."},{"cited_title":"In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=SygpC6Ntvr","cited_arxiv_id":null,"evidence_quote":"FLOPs regularization is the differentiable sparsity term in the training objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Splade-v3 provides the training data and hard negatives, and documents the MarginMSE sensitivity to teacher score sharpness."}],"review_version":1}