{"id":"545bc2d8-10e3-49cd-9011-981ebc78c1ae","arxiv_id":"2608.12569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TTT-Embed distills reranker feedback into a reusable vector added to frozen query embeddings, improving retrieval by up to 8.36 nDCG@10 and generalizing to unseen queries and tasks.","lead":"A new method, TTT-Embed, learns a small correction vector that is added to search query embeddings, so feedback from a reranker can be reused across many queries without touching the embedding model or its index. In tests across 15 retrieval benchmarks it improves ranking accuracy, with gains of up to 8.36 points, and transfers to queries and tasks it never saw during learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-corpus gains rest on an unproven extrapolation from distillation over the base retriever's top-K candidates to a fresh ANN search over the full corpus; a direct outside-candidate-set evaluation is needed.","rationale":"The reader's weakest assumption correctly identifies the gap between optimizing a softmax over C_q and deploying via an ANN search over the whole collection. The paper's own Appendix 9 acknowledges that the ranking-equivalence proposition does not extend to documents outside the scored list. The held-out and LOTO experiments are the right kind of evidence, but they are presented as single averages over 15 tasks and five models with no variance estimates, and the LOTO per-model margins are small. The proposed filtered-index test directly measures the quantity the central claim is about: whether the learned vector improves ranking of documents that were never in a rewarded candidate set. If it does not, the contribution reduces to query-specific or scope-specific re-ranking and the 'reusable state' claim is weaker. This does not change the verdict: CONDITIONAL remains appropriate, with the added condition that the authors report the outside-candidate-set analysis.","tokens_in":19531,"tokens_out":8664,"duration_ms":98269,"concrete_test":"For each query in the MTEB evaluation, remove from the ANN index every document contained in that query's rewarded candidate set C_q, and re-run the deployment search with x_q + alpha_g v_g and with the raw x_q on the filtered corpus. If the adapted query does not beat raw retrieval in nDCG@10 on this filtered corpus, or does not maintain the reported gain, the claim that TTT-Embed improves ranking of documents outside the optimization set is unsupported. Additionally, report the fraction and nDCG contribution of final top-10 hits that lie outside C_q; this separates candidate-reordering effects from genuine full-corpus improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The objective in Eq. (4) fits v_g to the reranker's distribution over C_q only, while deployment re-searches the entire index with the adapted query. Proposition 2 proves only that the adapted query's ranking is equivalent to the additive score x_q^T z_d + alpha_g v_g^T z_d; it does not constrain or improve the scores of documents outside C_q. The held-out-query and leave-one-task-out experiments are relevant transfer evidence, but they are aggregates reported without error bars, and the per-model LOTO margins are small (0.85-4.71, mean 3.09). The main budget curves in Figure 2 and Table 1 do not separate gains from reordering C_q from gains on documents never seen during optimization. If most of the nDCG@10 improvement comes from documents inside C_q, the central 'corpus-wide retrieval' claim is not established; the method would then be mainly a re-ranking scheme with a learned query-side bias, not a demonstration that ranking rewards produce reusable state for unseen documents.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TTT-Embed, a method that learns a lightweight residual vector v_g in the frozen output-embedding space of a dense retriever by distilling scalar ranking scores from an external reranker over the retriever's top-K candidate documents. The vector is added to query embeddings at deployment as x_q + alpha_g v_g, with a scope parameter controlling whether the vector is shared globally, per task, or per query, and with a closed-form evidence-adaptive magnitude alpha_g = n_g/(n_g+1). The authors evaluate on 15 MTEB retrieval tasks across five embedding models (three open-weight and two API-only Gemini models), reporting gains up to +8.36 nDCG@10 over raw retrieval, positive transfer to held-out queries and leave-one-task-out settings, and recovery of general MTEB capability after domain-specific fine-tuning, all without weight access or index modification.","tokens_in":19770,"tokens_out":4991,"duration_ms":49342,"significance":"If the results hold, TTT-Embed offers a practical and conceptually clean interface for reusing ranking rewards across queries under a fixed reward budget, with a principled scope trade-off and a parameter-free shrinkage rule. The paper's strengths include a simple and correct ranking-equivalence proposition, a careful nested-budget evaluation protocol that gives every method the same rewarded query-document pairs, evaluation on closed-source embedding APIs, and explicit held-out-query and leave-one-task-out transfer experiments. The main weaknesses are the absence of empirical comparisons to the closest prior test-time query-embedding methods (TOUR, ReFIT, GQR), the lack of any error bars or significance analysis, and the absence of a direct decomposition separating gains from reordering the training candidate set from gains on documents never seen during optimization.","major_comments":[{"comment":"The objective in Eq. (4) fits v_g only to the teacher distribution over each rewarded query's candidate set C_q, while deployment performs a fresh ANN search over the full corpus. The held-out-query and leave-one-task-out experiments in Table 2 and Figure 5 already provide aggregate evidence that the learned vector helps on queries whose candidate sets were never optimized, but the main fixed-population budget curves do not separate gains from reordering documents inside C_q from gains on documents outside C_q. Please add a direct analysis, such as nDCG@10 computed only over documents not present in any training C_q for rewarded queries, or the overlap between the original top-10 and the adapted top-10. This would substantiate the claim in §4.1 of 'corpus-wide retrieval improvements' and directly address whether the method is primarily a learned reranking bias or a genuinely reusable state for unseen documents.","section":"§2.2, §4.1, Table 2, deployment paragraph"},{"comment":"The paper does not empirically compare against TOUR (Sung et al., 2023), ReFIT (Reddy et al., 2023), or GQR (Uzan et al., 2025), which are the closest prior methods that optimize a test-query embedding from reranker or complementary-retriever scores. In particular, the query-wise variant of TTT-Embed appears to be nearly identical in setting to ReFIT's query-level distillation before a second retrieval pass. Please add matched-budget comparisons to these methods, or explicitly justify why they are not applicable baselines. Without such comparisons, the claimed advantage over existing test-time query-embedding optimization is not established.","section":"Related Work §8, Tables 1 and 2"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere in the paper. This matters because some of the key margins are small, notably the leave-one-task-out gains of 1.36 for Gemini Embedding 1 and 0.85 for Gemini Embedding 2 in Table 2, and because the query-selection schedule depends on a fixed seed. Please report variance across at least the query-allocation seed, and across sampling seeds for the stochastic LLM judges if they are used in any main result, or state clearly that the main experiments are deterministic and explain why the single fixed seed is representative.","section":"Tables 1, 2, 3 and Figures 2, 5"},{"comment":"The claim that TTT-Embed 'resolves catastrophic forgetting' is based on a single fine-tuned model (SKILLRET-SFT-0.6B) and a single specialization task. Catastrophic forgetting generally refers to sequential or continual learning, and one experiment is thin support for the phrase 'resolves'. Please soften the claim to 'mitigates' or 'recovers from domain-specialization degradation' and, if possible, add a second specialization scenario or explicitly discuss the scope of the evidence.","section":"§4.3, Figure 3"}],"minor_comments":[{"comment":"The phrase 'provides a elegant solution' contains a grammatical error; it should be 'provides an elegant solution.'","section":"§4.3"},{"comment":"The caption is difficult to read because of the long parenthetical clause and inconsistent punctuation (e.g., 'paired base on shrunken UB' is not defined before that point). Please rewrite the caption to state clearly that the held-out-query curves use a raw-query baseline on the same shrinking evaluation set.","section":"Figure 5 caption"},{"comment":"The phrase 'modifications to index' in the abstract should read 'modifications to the index', and the comma after 'This vector is optimized purely from scalar ranking scores' is misplaced; consider rephrasing for clarity.","section":"Abstract and §1"},{"comment":"The notation in Eq. (1) defines B as the total budget and b as the normalized budget, but the text later refers to 'budget b = B/N' and also uses 'budget b' for a reward budget in §5.1. Please introduce 'normalized budget' consistently to avoid confusion with the total budget B.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about extrapolation from the top-K candidate set to full-corpus retrieval is partially addressed by the held-out-query experiments in Table 2, since those queries' evaluation results come from a fresh ANN search and their candidate sets were never optimized. However, a direct outside-candidate-set decomposition for the main curves would settle the question. The more serious issue for the journal is the missing empirical comparison with ReFIT and TOUR, which are essentially the query-wise instantiation of the proposed framework; without that comparison, the novelty and practical advantage are not fully demonstrated. The manuscript is otherwise well-organized and the evaluation protocol is careful and reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a solid empirical paper, not a breakthrough. The vector-addition trick already exists in TOUR, ReFIT, and GQR. What is new is sharing that vector across queries at global/task/query scope and showing that the optimal scope shifts as the reward budget grows. That part is well done and practically useful, especially since it works with closed API embedding models.\n\nThe experiments are broad: five embedding models, 15 MTEB tasks, consistent gains across all of them and across scopes. The held-out-query and leave-one-task-out results are the strongest evidence — if a vector learned on other queries/tasks improves unseen ones, the state is genuinely reusable and not just memorizing the rewarded candidates. The budget-controlled allocation is careful, and the alpha shrinkage rule is justified with a normal-means model rather than tuned on labels. That is healthy practice.\n\nNow the soft spots. First, no error bars or significance tests anywhere. Given that the LOTO margins are as small as 0.85 nDCG points for one closed model, this is not optional. Second, the paper never compares to the closest prior methods (TOUR, ReFIT, GQR). The query-wise variant is essentially ReFIT with a different objective; at minimum they should benchmark against it at high budget. Third, the claim that TTT-Embed \"resolves catastrophic forgetting\" is overblown: it is test-time recovery with external rewards, not forgetting resolution. Minor, but worth softening in the next version.\n\nOn the stress-test concern: I do not think it sinks the paper. The held-out query and task transfer experiments are direct evidence that gains are not just reordering the training candidates — those queries never saw a single reward during optimization, so any improvement must come from documents outside the training C_q. The paper also issues a new full-index search at evaluation, so the protocol is honest. What is missing is a decomposition of the main budget curves into within- vs outside-candidate gains, and significance testing of the transfer numbers. Both are addressable in revision.\n\nOne more thing: no code or data. For a paper whose contribution is mostly empirical, that limits confidence and independent verification.\n\nBottom line: worth sending to reviewers. The core extension is real, the experiments are broad, and the gaps are fixable. I would ask for a comparison to ReFIT/TOUR, error bars, and a decomposition analysis before accepting. Who should read it: anyone working on test-time adaptation for retrieval or RAG systems.\n\nBest,\n[Name]","headline":"Solid, useful extension of test-time query-vector distillation with a budget-aware scope analysis; needs error bars and comparisons to TOUR/ReFIT/GQR, but the core claims are plausible and the transfer results hold up.","tokens_in":20291,"tokens_out":3314,"would_cite":true,"duration_ms":31691,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned vector lifts dense retrieval by up to 8.36 nDCG@10","keywords":["test-time optimization","dense retrieval","query embeddings","knowledge distillation","ranking rewards","embedding space adaptation","catastrophic forgetting","reward budget"],"falsifier":"Train a task-wise vector on rewarded queries, then evaluate retrieval on a held-out document pool that excludes every candidate that appeared in the training top-$K$ lists (or on documents that never ranked in any rewarded query's top-$K$). If the gain over raw retrieval does not persist on those unseen documents, the claim that the vector reorders the corpus beyond the candidate set fails.","tokens_in":19337,"feed_emoji":"🔍","tokens_out":7634,"duration_ms":65051,"temperature":0.7,"pith_summary":"Test-time ranking scores from a reranker or LLM judge are usually used once and then discarded. This paper proposes TTT-Embed, which distills those scalar scores into a small residual vector added to the frozen query embedding, so the same rewards improve later retrievals. A single scope parameter decides whether the vector is shared by all queries, by one task, or kept private to a query, and the best scope shifts from global to task to query as the reward budget grows. Across five embedding models and fifteen retrieval tasks, the method reports gains up to +8.36 nDCG@10, transfers to unseen queries and tasks, and recovers general capability lost to fine-tuning while leaving model weights untouched. If the claim holds, reranker feedback becomes a reusable test-time state for any embedding system, including closed API models.","feed_headline":"A learned vector lifts dense retrieval up to 8.36 nDCG@10","feed_subtitle":"Frozen embeddings plus a scoped residual vector transfer gains to unseen queries and tasks, with no weight access.","key_machinery":"The central object is the scoped residual vector $\\boldsymbol{v}_g$ living in the frozen embedding space, with the adapted query $\\tilde{\\boldsymbol{x}}_q = \\operatorname{unit}(\\boldsymbol{x}_q + \\alpha_g \\boldsymbol{v}_g)$. The learning rule is a listwise knowledge-distillation objective in which the teacher is the reward model's softmax over the candidate set and the student is the softmax of embedding similarities; the only free design choice is which rewarded queries share a group $g$. A scope parameter selects global, task, or query-wise sharing, and a parameter-free shrinkage $\\alpha_g = n_g/(n_g+1)$, justified as a Bayes estimator, sets how strongly the vector is applied. This object carries the argument because all the information from the reward model is compressed into one additive vector that can be cached and reused against the unchanged ANN index.","core_discovery":"The paper's central claim is that the scalar relevance scores a reranker assigns to the top $K$ candidates of a dense retriever can be turned into a durable, reusable state: a single residual vector $\\boldsymbol{v}_g$ that is added to the query embedding before a fresh search over the unchanged document index. The vector is learned by minimizing a listwise KL divergence between the softmax of the reward scores (teacher) and the softmax of embedding similarities (student) over the retrieved candidates, with ridge regularization. Because normalization is a positive constant per query, the adapted score orders documents exactly like the base score plus the additive term $\\alpha_g \\boldsymbol{v}_g^\\top \\boldsymbol{z}_d$; the paper proves this ranking equivalence and uses it to argue that the correction applies corpus-wide, not only to the re-ranked candidate list. The authors report that with a budget of ten rewards per query the shared state lifts nDCG@10 by 6.36 points on average over raw retrieval and beats direct reranking by 3.15 points, and that gains persist on queries and tasks that never supplied rewards. They further claim that a frozen fine-tuned model can recover general capability lost to specialization (up to +8.00 nDCG@10) without any weight update.","pith_inferences":["Editorial inference: the reported transfer suggests the learned vector captures task-level regularities rather than query memorization; a direct test would be to train on one task family and evaluate on a disjoint domain with no shared vocabulary.","Editorial inference: because the method only needs scalar relevance scores and vector outputs, the same residual-state recipe should transfer to other embedding-based search problems such as code search, recommendation, or multimodal retrieval.","Editorial inference: the shrinkage rule treats each rewarded query as an independent unit of evidence; if reward quality varies per query, an evidence-weighted or confidence-weighted version of $\\alpha_g$ is a natural extension the paper does not explore.","Editorial inference: the finding that general-purpose judges produce usable rewards suggests a cheap-judge, high-depth variant could approximate the specialized reranker results at lower API cost."],"forward_implications":["With shared scopes, a reward budget spent on a subset of queries improves retrieval for all queries in that scope, so the system can spend scarce reranker calls where they help most.","The optimal sharing scope depends on budget: global reuse wins at very low budgets, task-wise at intermediate budgets, and query-private vectors at high budgets, giving a concrete deployment rule.","Because no weights or index entries change, the method works with closed embedding APIs and with precomputed document indexes, and the learned state is small enough to store.","A fine-tuned model can keep its specialization while regaining broad benchmark performance at test time, since the correction is applied outside the model.","The additive structure makes the corrected search equivalent to base score plus a per-document bias $\\boldsymbol{v}_g^\\top \\boldsymbol{z}_d$, so retrieved documents are not limited to the original candidate list."],"supporting_citations":[{"why":"It establishes the dense-retrieval setting of frozen encoders and inner-product ranking that this method extends.","marker":"Karpukhin et al., 2020"},{"why":"It supplies the listwise learning-to-rank formulation that the distillation objective instantiates.","marker":"Cao et al., 2007"},{"why":"It provides the soft-target knowledge-distillation mechanism used to match teacher and student distributions.","marker":"Hinton et al., 2015"},{"why":"It shows that cross-encoder rerankers supply the stronger relevance signal distilled at test time.","marker":"Nogueira and Cho, 2019"},{"why":"It is the per-query predecessor that distills reranker scores into a query representation, which this work makes reusable across queries.","marker":"Reddy et al., 2023"},{"why":"It is the large-scale validation that query-vector distillation from reranker feedback works, motivating the scoped-sharing extension.","marker":"Gangi Reddy et al., 2025"},{"why":"It provides the 15-task retrieval benchmark used to measure gains and transfer.","marker":"Muennighoff et al., 2022"},{"why":"It supplies the specialization corpus used in the forgetting-recovery experiment.","marker":"Cho et al., 2026"}],"fun_headline_variants":["Scoped residual vector lifts dense retrieval by up to 8.36 nDCG@10","One vector adapts frozen embeddings: up to +8.36 nDCG on retrieval","Reusable test-time reward vector improves retrieval up to 8.36 nDCG","Frozen retriever, no weight access: learned vector gains 8.36 nDCG","Test-time embedding tuning via ranking rewards: +8.36 nDCG@10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The learned vector is optimized to match the reranker on the base retriever's top-$K$ candidates, but at deployment it is used to search the whole corpus; if matching the teacher on those candidates does not also improve the ranking of unseen documents, including documents outside the candidate sets, the reported gains will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Scoped residual vector lifts dense retrieval by up to 8.36 nDCG@10","One vector adapts frozen embeddings: up to +8.36 nDCG on retrieval","Reusable test-time reward vector improves retrieval up to 8.36 nDCG","Frozen retriever, no weight access: learned vector gains 8.36 nDCG","Test-time embedding tuning via ranking rewards: +8.36 nDCG@10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1827,"prompt_tokens":1148,"completion_tokens":679,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":764,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":764,"tokens_out":679,"duration_ms":5925,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:05:46.669101+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a task-wise vector on rewarded queries, then evaluate retrieval on a held-out document pool that excludes every candidate that appeared in the training top-$K$ lists (or on documents that never ranked in any rewarded query's top-$K$). If the gain over raw retrieval does not persist on those unseen documents, the claim that the vector reorders the corpus beyond the candidate set fails.","supporting_citations":[],"review_version":1}