{"id":"2094e5c3-7318-4109-99c4-3a3d88f57d24","arxiv_id":"2505.03676","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Rational Retrieval Acts applies Rational Speech Acts to sparse IR models, reweighting token-document associations by corpus context and improving retrieval accuracy on BEIR datasets.","lead":"The paper adapts the Rational Speech Acts framework, a model of efficient human communication, to reweight term-document weights in sparse retrieval models such as SPLADE and BM25. The resulting method, Rational Retrieval Acts, reports consistent nDCG@10 gains on BEIR benchmark datasets without adding query-time cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-dataset alpha tuning with LLAMA3-8B synthetic queries is load-bearing; without evidence that it transfers to real queries, the zero-shot SOTA claim is not supported.","rationale":"The reader identified the same load-bearing assumption: alpha is tuned per target collection using synthetic queries, and the paper assumes this transfers to real user queries while preserving a zero-shot evaluation. My stress-test pass agrees and sharpens the concern. The mathematical core of RRA is a plausible reweighting of sparse representations, and the released code is a positive sign, but the empirical headline rests on the alpha-selection protocol. Section 3.1 explicitly states that alpha is selected by maximizing nDCG@10 on synthetic query-document pairs per dataset, and Figure 2 demonstrates high sensitivity to alpha. Since baselines in Tables 1 and 2 are not given a comparable target-adaptation step, the comparison conflates the RSA contribution with the benefit of unsupervised target-corpus hyperparameter selection. Other issues, such as typos in equation 7 and the lack of efficiency measurements, are real but secondary; they do not directly threaten the central effectiveness claim as much as the unverified transfer of the per-dataset alpha selection. The proposed test, using a fixed alpha or a source-collection-chosen alpha, would settle whether the claimed consistent improvement and SOTA status survive without target-specific tuning. Therefore the reader's CONDITIONAL verdict remains appropriate, with the condition being evidence that the synthetic-query alpha selection transfers to real queries.","tokens_in":12216,"tokens_out":10641,"duration_ms":112939,"concrete_test":"Re-run the S-RRA experiments with a single alpha chosen once on MS MARCO (or set alpha=1, the standard RSA value) and applied unchanged to all BEIR datasets, reporting per-dataset and mean nDCG@10 against the same baselines. If the mean gain over SPLADE drops below roughly 0.5 points or several datasets regress, the reported improvements are contingent on per-dataset synthetic validation and the zero-shot SOTA claim fails. As a secondary diagnostic, compare alpha*_synth with alpha*_real (oracle grid search on the actual test queries) for each dataset; large discrepancies would directly confirm the transfer problem.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RRA consistently improves sparse retrieval and achieves state-of-the-art performance on out-of-domain BEIR datasets depends on the alpha-selection procedure in Section 3.1. For each dataset, alpha is chosen by maximizing nDCG@10 on 500 synthetic query-document pairs generated from documents in that target dataset using LLAMA3-8B. The reported results in Tables 1 and 2 then evaluate on the real test queries with that dataset-specific alpha. This creates an asymmetric comparison: the baselines are fixed, zero-shot systems, while S-RRA receives target-collection information (the documents and the LLM-generated queries) to select alpha. Figure 2 shows that alpha can change nDCG@10 by up to 10 points on TREC-COVID, so the choice materially affects outcomes. If synthetic queries do not match real user query behavior, the selected alpha may be far from optimal, and the reported mean gains (+1.0 for SPLADE, +4.2 for SPARTA) could shrink or disappear. The paper provides no evidence that alpha values selected on synthetic queries correlate with alpha values optimal on real queries. Thus the load-bearing assumption is unverified, and the state-of-the-art claim is not a fair zero-shot comparison as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts the Rational Speech Acts (RSA) framework to sparse neural information retrieval. It treats tokens as utterances and documents as meanings, defines a literal lexicon from sparse-model weights, and applies one RSA speaker/listener iteration to obtain a ``pragmatic'' document representation. A sparse factorization (Eqs. 5-9) reduces the memory cost from |T| x |D| to |T| + |D| plus the nonzero sparse weights. Experiments on subsets of the BEIR benchmark compare five base models (SPLADE, SPARTA, BM25, UniCOIL, DeepImpact) with their RRA-enhanced versions, report consistent nDCG@10 gains, and compare S-RRA with several strong baselines. The hyperparameter alpha is selected per target dataset using 500 LLAMA3-8B-generated synthetic queries per dataset, and the pre-transformation function f is selected on MS-Marco.","tokens_in":12517,"tokens_out":5508,"duration_ms":51393,"significance":"If the claims hold, the paper offers a conceptually novel and potentially useful way to make sparse retrieval representations collection-sensitive without changing the query-time scoring archetype. The sparse factorization is a genuine practical contribution, and the demonstration of consistent gains across five sparse models is encouraging. The authors also release code and include an interpretable case study. However, the state-of-the-art claim depends on a per-dataset alpha-selection protocol that has not been shown to transfer to real queries, and the evaluation covers only part of BEIR, so the empirical contribution is currently weaker than the abstract and discussion suggest.","major_comments":[{"comment":"The zero-shot, out-of-domain claim is not supported as presented because the RSA hyperparameter alpha is tuned separately for each target dataset using 500 synthetic queries generated from that dataset's own documents, and the reported test scores are then obtained on the same dataset with the selected alpha. This makes the comparison asymmetric: the baselines have no per-dataset hyperparameter selection, while S-RRA is given target-collection information. Figure 2 shows that alpha can change nDCG@10 by up to 10 points on TREC-COVID, so the choice is materially load-bearing. Please provide evidence that alpha values selected on synthetic queries correlate with alpha values optimal on real queries, or report results with a fixed alpha (e.g., alpha=1 or alpha chosen on MS-Marco only). Without such evidence, the mean gains in Tables 1 and 2 are not a fair zero-shot comparison.","section":"Section 3.1, Tables 1-2"},{"comment":"The printed equations are inconsistent with the definitions above them. In Eq. (7), the first summand is written as exp(alpha L0(d|t)), but Eq. (3) defines S1(t|d) = exp(alpha log L0(d|t)) / Z_d, so the exponent should be alpha log L0(d|t). In Eq. (9), the first sum is written as sum_{d' in D_t} S1(d|t), which has the arguments reversed; it should be S1(t|d'). Also, in the second sum of Eq. (9), s_d^1 should be s_{d'}^1 since the factor depends on the document d'. These typos sit in the core derivation of the method and need to be corrected before the factorization can be checked.","section":"Equations (7) and (9)"},{"comment":"The text says the method is evaluated on the BEIR benchmark, but Table 1 covers 9 datasets and Table 2 covers 13 datasets, while BEIR has 18 datasets. The reported mean nDCG@10 and the comparison with prior state-of-the-art results are therefore computed on a subset, and the subset differs between the two tables. To support the ``state-of-the-art on BEIR'' claim, the authors should either report all BEIR datasets or explicitly state that the comparison is on a subset and justify the selection. The missing datasets (e.g., BioASQ, Signal-1M, Robust04, TREC-News) could alter both the mean and the ranking of methods.","section":"Section 3.2, Tables 1-2"},{"comment":"The paper claims that RRA enhances effectiveness ``without compromising efficiency,'' and the contributions state that gains are achieved ``without increasing inference costs.'' However, no efficiency measurements are reported: there are no latency, throughput, index-size, or memory figures. The sparse factorization is a plausible basis for the claim, but the actual retrieval-time cost of Eq. (10) and the offline cost of computing l_t^1 and l_d^1 should be measured and reported before the efficiency claim can be accepted.","section":"Section 4, Contributions"}],"minor_comments":[{"comment":"The caption begins with ``Table 1: Table 1:''; please remove the duplicate label.","section":"Table 1 caption"},{"comment":"The notation l_d^0 is defined as the constant 1 in Eq. (5), but the subsequent factorization would be clearer if l_d^0 were defined explicitly for all d and if s_d^1 in Eq. (6) were written with its full dependence on Z_d^(1).","section":"Equations (5)-(9)"},{"comment":"The caption states that underlined values are significant with a paired t-test (p<0.05), but the manuscript gives no details about the unit of analysis, the number of queries or runs, or which values are underlined. Please provide these details.","section":"Table 1 caption"},{"comment":"The selection of f among the six listed mappings is described as ``preliminary experiments on MS-Marco,'' but no scores or protocol are reported. Since f is a component of the method, a short table of these preliminary results in an appendix would improve reproducibility.","section":"Section 3.1, Pre-transformation function"},{"comment":"Figure 2 shows the sensitivity of nDCG@10 to alpha for S-RRA only. Adding similar curves for at least one other base model would strengthen the claim that alpha selection is robust across models.","section":"Section 3.1, Figure 2"},{"comment":"The sentence about functions with f(0) != 0 says they ``avoid contrasting too much the document weights L1(t|d),'' but the notation should be L1(d|t); please correct this.","section":"Section 3.3, Case study"},{"comment":"Reference [4] for Bunt's computational pragmatics appears to have a URL that points to an unrelated journal article; please provide the correct bibliographic entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is original and the gains across five base models are promising, but the alpha-selection protocol is the central technical risk. I would be willing to accept a revised version that either demonstrates transfer of the synthetic-query alpha choice to real queries or reframes the claims to avoid the zero-shot SOTA statement. The paper is within the scope of SIGIR and the short-paper format."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time. It is the first to apply Rational Speech Acts to sparse retrieval, and it solves a real scalability problem with a clean factorized decomposition of the RSA equations, shrinking the naive |T|×|D| matrices to something linear in the vocabulary and corpus size plus the sparse tensor. That technical contribution stands on its own. The experiments are also broader than many: five sparse models, consistent gains on every reported dataset, and a nice case study showing stopword down-weighting in SPARTA. The writing is clear, and the method is honestly motivated by the known \"wacky weights\" problem.\n\nThe main soft spot is the alpha selection. They tune alpha per dataset by maximizing nDCG@10 on 500 synthetic queries generated by LLAMA3-8B from the target documents, then evaluate on real test queries. This gives the method target-collection information that the fixed baselines do not get. Figure 2 shows alpha can shift nDCG@10 by up to 10 points on TREC-COVID, so the choice is consequential. The paper provides no evidence that alpha selected on synthetic queries correlates with what would be optimal on real queries. So the \"zero-shot out-of-domain state-of-the-art\" claim is not fully supported. I would not call it a fatal flaw—the gains are monotone across datasets and the method clearly does something sensible—but it needs to be qualified or the tuning procedure needs a transfer experiment.\n\nSmaller issues: Eq. (7) is missing a log in the first sum, and Eq. (9) has an index/species error (S1(d|t) vs S1(t|d)). These are typos but they make the math harder to trust. They also report only a subset of BEIR (9 datasets in Table 1, 13 in Table 2), and there are no error bars, though they mention paired t-tests without showing which comparisons are significant. The pre-transformation f is chosen on MS-Marco, which is defensible but adds another tuned component.\n\nWho should read this: anyone working on sparse retrieval, collection-dependent term weighting, or pragmatic models for IR. It deserves a serious referee; the method is novel and the evaluation, while imperfect, is a reasonable first pass. If you are reviewing a revision, push for evidence on alpha transfer or a sensitivity analysis with fixed alpha, and ask for the missing logs in the equations. In your own work, cite it for the method and the decomposition, not for a clean zero-shot result.","headline":"A genuinely new RSA-for-sparse-retrieval method with a neat sparsity decomposition, but the zero-shot SOTA claim is softer than it looks because alpha is tuned per target collection on LLM-generated synthetic queries.","tokens_in":12962,"tokens_out":3570,"would_cite":true,"duration_ms":34802,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adapting Rational Speech Acts to sparse retrieval, treating tokens as utterances and documents as meanings, consistently improves retrieval models on out-of-domain data at no extra query-time cost.","keywords":["Neural Information Retrieval","Sparse Retrieval","Rational Speech Acts","Pragmatic Reasoning","Linguistic Pragmatics for Retrieval","BEIR benchmark","zero-shot retrieval"],"falsifier":"On a dataset with high alpha sensitivity (the paper shows TREC-COVID varies by up to 10 nDCG@10 points with alpha), generate several independent 500-query synthetic sets, tune alpha on each, and evaluate on the real queries. If the best alpha from synthetic sets yields real-query scores no better than baseline SPLADE, or if the tuned alphas disagree wildly, then the alpha-selection step—not RSA itself—is the fragile link.","tokens_in":12075,"feed_emoji":"🔍","tokens_out":7395,"duration_ms":67096,"temperature":0.7,"pith_summary":"This paper argues that sparse retrieval models represent each document in isolation, ignoring the rest of the collection, and that borrowing a pragmatic-communication model from linguistics fixes that blind spot. It adapts Rational Speech Acts, a framework that explains how a speaker picks words to single out an object among alternatives, into a post-processing layer that reweights token–document associations against the whole collection. Applying this layer to BM25 and four learned sparse models improves nDCG@10 on out-of-domain datasets, and the strongest configuration sets a new state of the art on the BEIR benchmark. Because the reweighting happens offline, the gain costs nothing at query time.","feed_headline":"Pragmatic reasoning lifts sparse retrieval past prior bests","feed_subtitle":"Adapting Rational Speech Acts to sparse encoders lifts out-of-domain ranking, with no added query-time cost.","key_machinery":"The load-bearing object is the pragmatic listener L1(d|t), obtained by one RSA speaker–listener iteration from a sparse model's weights. The enabling identity is the factorization of the zero-weight entries: when w_{t,d} = 0, the literal-listener contribution L0(d|t) factors as a token-only factor l0_t times a document-only factor l0_d, and the pragmatic speaker and listener inherit this factored form (equations 5–8). This reduces the RSA update to computing a handful of per-token and per-document vectors instead of a dense |T| x |D| matrix, which is what lets the method scale to entire collections and score queries at retrieval time with inverted indices.","core_discovery":"The paper introduces Rational Retrieval Acts (RRA), an adaptation of Rational Speech Acts (RSA) to sparse information retrieval. RSA models communication as a speaker choosing an utterance to identify a meaning, and a listener inferring the meaning from the utterance. RRA maps tokens to utterances and documents to meanings, then computes a literal listener from the sparse model's raw token weights, a pragmatic speaker via an exponentiated softmax controlled by a temperature alpha, and finally a pragmatic listener that becomes the document's representation. The central technical contribution is a sparsity-aware factorization: for every token–document pair with zero weight, the RSA update factors into a token-only term and a document-only term, reducing the storage footprint from |T| x |D| to |T| + |D| plus the sparse nonzeros. This makes running RSA over an entire document collection feasible, and the resulting scores are still evaluated through an inverted index.","pith_inferences":["The paper frames RSA as a one-shot correction, but the same factored update can be iterated; the claim that alpha and depth trade off suggests iterating with a smaller alpha could yield further gains at the cost of another offline pass.","The observation that f(0) != 0 helps, preferring f(x)=1+x over log(1+x) or identity, suggests the zero weights are informative as 'this token is absent,' a signal most sparse models discard; a learned transformation might exploit this further.","The synthetic-query alpha tuning is a template for zero-shot hyperparameter selection in retrieval more broadly: if LLM-generated queries can pick alpha, they might also pick model variants or fusion weights.","The approach might transfer to dense retrieval by contrasting a query or document against a set of distractors sampled from the collection, though the exact factorization that makes RRA cheap would not carry over."],"forward_implications":["Any sparse retriever, lexical or neural, can be wrapped with RRA as an offline post-processing step, preserving its efficiency profile while improving accuracy.","Relative gains are largest for models that do no query-side weighting, so RRA can substitute for missing contextual query term weighting.","The mean nDCG@10 of 53.0 on the 13 BEIR datasets gives S-RRA the top position among the compared sparse and dense baselines.","The dependence on alpha, tuned on synthetic queries, makes the method applicable to new domains without human relevance judgments.","The factorization generalizes to any sparse scoring function whose nonzero entries can be separated from zero entries, so other sparse families could benefit."],"supporting_citations":[{"why":"Defines Rational Speech Acts, the pragmatic speaker–listener framework that RRA adapts to information retrieval.","marker":"[13]"},{"why":"Introduces SPLADE, the main sparse neural retrieval model that RRA is applied to and improved.","marker":"[11]"},{"why":"Documents the limited discriminative power of learned sparse term weights, motivating collection-aware reweighting.","marker":"[21]"},{"why":"Provides the synthetic-query generation method used to tune the alpha parameter per dataset.","marker":"[3]"},{"why":"Supplies the BEIR benchmark used for zero-shot out-of-domain evaluation.","marker":"[29]"},{"why":"Models users as rational agents who minimize query effort, motivating the pragmatic-speaker side of RSA.","marker":"[2]"},{"why":"Shows prior applications of RSA to image captioning and machine translation, establishing the recipe RRA follows.","marker":"[6, 7]"}],"fun_headline_variants":["Pragmatic reasoning gives sparse retrieval a context boost","RSA factorized: pragmatic boosts for sparse retrieval","Sparse retrieval gets pragmatic: efficient RSA for whole collections","Pragmatic sparse retrieval: factoring RSA down to collection scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rely on alpha being chosen per collection from a few hundred synthetic LLM-generated queries, and on that choice transferring to real user queries; if synthetic queries do not reflect real behavior, the improvements could shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Pragmatic reasoning gives sparse retrieval a context boost","RSA factorized: pragmatic boosts for sparse retrieval","Sparse retrieval gets pragmatic: efficient RSA for whole collections","Pragmatic sparse retrieval: factoring RSA down to collection scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001239,"raw_usage":{"total_tokens":5050,"prompt_tokens":875,"completion_tokens":4175,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":4109}},"tokens_in":491,"tokens_out":4175,"duration_ms":26146,"temperature":1.0,"reasoning_tokens":4109,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:45:12.844251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset with high alpha sensitivity (the paper shows TREC-COVID varies by up to 10 nDCG@10 points with alpha), generate several independent 500-query synthetic sets, tune alpha on each, and evaluate on the real queries. If the best alpha from synthetic sets yields real-query scores no better than baseline SPLADE, or if the tuned alphas disagree wildly, then the alpha-selection step—not RSA itself—is the fragile link.","supporting_citations":[{"cited_title":"Frank and Noah D","cited_arxiv_id":null,"evidence_quote":"Defines Rational Speech Acts, the pragmatic speaker–listener framework that RRA adapts to information retrieval."},{"cited_title":"Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation,","cited_arxiv_id":null,"evidence_quote":"Documents the limited discriminative power of learned sparse term weights, motivating collection-aware reweighting."}],"review_version":1}