{"id":"18900c2f-27bd-4551-8785-d7bdfda21697","arxiv_id":"2511.13726","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RT improves embedding-model retrieval on some reasoning-heavy tasks by iteratively refining the query embedding at test time, but reported gains are mixed and the key implementation detail is unspecified.","lead":"The paper introduces RT (Refine Thought), a test-time procedure that re-encodes a query repeatedly, feeding its pooled embedding back into the model and using the last embedding for retrieval. It reports small gains on BRIGHT and larger gains on two algorithm-related job-matching tasks, while staying flat on C-MTEB, but releases no code or data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Listing 1 Step 2 defines `concatenate(x_q, states)` without specifying how pooled vector states become model input; RT is therefore not reproducible, and the reported gains may be input augmentation rather than temporal unrolling.","rationale":"The reader identified the same weakest assumption: the undefined `concatenate(x_q, states)` operation in Listing 1. I agree this is the most load-bearing concern because the entire method and its theoretical framing ('temporal unrolling in hidden space') depend on how previous pooled states are fed back into the encoder. Without this, the experiments cannot be interpreted: the reported gains (BRIGHT +0.2 average, PJBenchmark Algorithm improvements) could equally result from adding extra tokens to the input, which is a well-known prompt-sensitivity effect rather than multi-step reasoning. The paper provides no code, no tensor-shape details, and no control for added tokens, so the central claim is not currently verifiable. I do not escalate to REJECT because the idea may be salvageable with a concrete implementation and appropriate controls; the paper's own limitations about T-sensitivity and trajectory drift are secondary to this foundational ambiguity. A conditional acceptance requiring the implementation and control experiment is the appropriate verdict.","tokens_in":5921,"tokens_out":2810,"duration_ms":27024,"concrete_test":"Ask the authors to release the exact implementation of `concatenate(x_q, states)` (tokenization of vectors vs. hidden-state injection). Then run a control on PJBenchmark JD2CV/CV2CV and BRIGHT: for T=2, replace the fed-back state h_0 with either (a) a fixed random vector of the same dimension or (b) a repeated copy of the query text, keeping all other settings identical. If either control reproduces the reported gains, the effect is not temporal unrolling of the embedding model but input augmentation. Additionally, re-run BRIGHT with T selected on a validation split rather than post hoc to verify the +0.2 average.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that iteratively feeding pooled embedding states back into a decoder-only encoder improves reasoning via temporal unrolling—rests entirely on Step 2 of Listing 1: `concatenate(x_q, states)`. This operation is typed inconsistently: `x_q` is a token sequence, while each `states[i]` is a pooled embedding vector (e.g., a dense 4096-dim vector for Qwen3-Embedding-8B). The paper never explains how these are combined. If the states are serialized into text tokens, the method reduces to appending extra text to the query, and the observed gains could be a prompt-length/content effect rather than iterative hidden-state refinement. If the states are injected into hidden layers, this requires a nonstandard modification of the encoder's forward pass that is neither described nor supported by the cited Qwen3-Embedding interface. This ambiguity breaks the causal story in §3.2 and undermines the comparison across T: any benefit could come from adding tokens, not from temporal depth. The reported experiments do not control for this—BRIGHT improves by only 0.2 points on average (22.9→23.1, Table 2), while PJBenchmark gains are concentrated in two Algorithm subtasks (JD2CV +19%, CV2CV +21%) with a Finance decline, making it especially easy for an uncontrolled input-augmentation artifact to flip the conclusion. The paper's own limitations (§4.3) concede sensitivity to T and state-aggregation scheme, but do not address the underlying representation mismatch.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RT (Refine Thought), a test-time method for text embedding models that iteratively runs forward passes on the query, feeding intermediate pooled representations back into the model, and uses the final state as the query embedding. The method is evaluated on three benchmarks: BRIGHT (semantic reasoning retrieval), PJBenchmark (person-job matching), and C-MTEB STS (general semantic similarity). The authors report a small average gain on BRIGHT (22.9 to 23.1), large gains on two PJBenchmark Algorithm tasks (JD2CV +19%, CV2CV +21%), and stable performance on C-MTEB STS. They attribute these effects to temporal unrolling that 'activates' semantic reasoning abilities learned during pretraining in decoder-only models, and argue decoder-only architectures benefit more than encoder-only ones.","tokens_in":6310,"tokens_out":5528,"duration_ms":47482,"significance":"If the method is correctly specified and the reported gains are reproducible, the idea is noteworthy: it offers a training-free way to trade inference compute for improved embedding-model reasoning, with a plausible connection to looped transformers and chain-of-thought literature. The paper also addresses a real gap in embedding-model reasoning and includes a pseudocode description, a decoder-only versus encoder-only comparison, and an explicit discussion of limitations. However, the central implementation step is underspecified, the experimental evidence is too thin to support the strong causal and comparative claims, and the per-task selection of hyperparameter T raises concerns about test-set fitting. The paper's own limitations sections concede sensitivity to T and state-aggregation, and acknowledge that the architecture/task/data attribution is not established, which directly tempers the abstract's conclusions.","major_comments":[{"comment":"Step 2, 'concatenate(x_q, states)', mixes a token sequence with a list of pooled dense vectors. The paper never explains how the pooled states are converted into model input. If they are serialized into text, the method reduces to prompt augmentation and the gains may be caused by added tokens rather than temporal unrolling; if they are injected into hidden layers, the implementation is nonstandard and incompatible with the cited Qwen3-Embedding interface. This ambiguity affects every reported result. Please specify the exact tensor-level operation and add a control condition (e.g., appending random or fixed vectors of the same shape, or concatenating the same state repeatedly) to isolate the temporal-unrolling effect from simple input modification.","section":"§3.2, Listing 1"},{"comment":"The abstract and conclusion call the improvements 'significant', but the reported numbers do not support this. BRIGHT average improves from 22.9 to 23.1, while several subtasks decline (e.g., StackExchange 21.3→21.1, Rob 18.8→18.0, Sus 18.3→18.1). PJBenchmark gains are concentrated in two Algorithm tasks (JD2CV +19%, CV2CV +21%) while Finance JD2CV declines from 62.66 to 61.30 (-2%). No error bars, confidence intervals, significance tests, or variance estimates are provided. The strong wording is disproportionate to the effect size. Please add per-task variability measures, a multiple-comparison-aware significance test, and qualify the claim to 'numerically positive on average' unless statistical support is provided.","section":"§4.2, Tables 2–3"},{"comment":"T is selected per task after scanning values 1–10, and the paper states that 'more complex tasks require larger T' without a prespecified complexity measure. This constitutes ex post hyperparameter fitting on the test benchmarks. Because the final representation is h_T, choosing T based on benchmark results can inflate the reported gains. Please report the chosen T for every benchmark/task, show the full T sweep for all tasks (Figure 2 currently gives only an illustrative sketch), and propose an a priori rule for selecting T (e.g., based on query length or a separate complexity model) for practical use.","section":"§4.1, §4.3 and Figure 2"},{"comment":"The causal claim that RT 'activates the semantic reasoning ability learned during pretraining' is not directly tested. The paper's own limitations concede sensitivity to T and state-aggregation, drift on long inputs, lack of explicit credit assignment, and state in §5 that 'further studies are required to disentangle whether the observed improvements arise from differences in architecture, task type, or training data.' These concessions are in tension with the abstract and conclusion. Please soften the causal attribution to a phenomenological observation, or provide an experiment that isolates the pretraining-reasoning-trajectory mechanism (e.g., ablation on a decoder-only model without reasoning pretraining data).","section":"Abstract and §4.3/§5"},{"comment":"The conclusion that 'decoder-only architectures are inherently more effective for semantic reasoning' is based on a single decoder-only model (Qwen3-Embedding-8B) and a single encoder-only model (bge-large-zh-v1.5). This is an extremely narrow basis for a general architectural claim, especially given the differing model scales and training data. Please test additional decoder-only and encoder-only models of comparable scale, or explicitly limit the conclusion to the evaluated models.","section":"§4.2, Table 4 and §5"}],"minor_comments":[{"comment":"The pseudocode contains a typo ('previus') and the definition of T is inconsistent: h_0 is computed before the loop, but the text says 'T=1 is the single-step baseline.' Clarify whether the baseline is h_0 (no refinement) or h_1 (one refinement) and make the pseudocode's loop range match.","section":"Listing 1 and §4.1"},{"comment":"The table formatting is degraded: Table 2's header merges column labels awkwardly, and Table 3 has 'JD2CV' appearing under both Algorithmic and Finance. Please also specify the evaluation metric for each benchmark (e.g., nDCG@k, accuracy, Spearman correlation) and report dataset sizes.","section":"Tables 1–3"},{"comment":"Figure 1 is referenced but not described in detail, and Figure 2 lacks axis labels and a legend. Please ensure both figures are self-contained and that the T sensitivity curves are legible.","section":"Figures 1–2"},{"comment":"The phrase 'RT remains stable on general-purpose semantic understanding tasks' is accurate for C-MTEB STS, but 'achieves consistent gains on semantic reasoning tasks' is contradicted by the BRIGHT subtask declines and the PJBenchmark Finance decline. Rephrase to reflect the actual distribution of results.","section":"§4.2, paragraph 1"},{"comment":"The claim that RT operates 'implicitly in the hidden space' cites only a survey [21]; please cite the original looped-transformer or latent-thought works, such as [14], more directly. Also clarify how the method differs from simply adding more tokens to the query when states are serialized.","section":"§2 and §3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core idea is interesting but the current version is not verifiable: the central concatenation operation in Listing 1 is ill-defined, and the experimental support for 'significant' improvements is weak. The limitations sections are honest, which is a positive sign, but the abstract and conclusions overstate the causal mechanism. If the authors can specify the implementation concretely, provide code or reproducible tensor-level details, add basic significance testing, and temper the claims, a revised version could be suitable. If they cannot clarify the state-injection mechanism, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RT is a legitimate extension of looped-transformer and latent-reasoning work, applied to frozen decoder-only embedding models at test time. The paper is honest that the temporal-unrolling idea is borrowed, and the specific combination—feeding pooled query embeddings back through the same encoder for a few extra forward passes—is not something I have seen done for retrieval. That is worth a careful look. The benchmark coverage is reasonable for a short paper: BRIGHT, C-MTEB STS, and PJBenchmark. The authors also report the Finance decline in PJBenchmark rather than hiding it, and their limitations section concedes sensitivity to T, aggregation, and drift. That is more honest than many submissions.\n\nThe problems are real, though. The headline numbers are thin: BRIGHT goes from 22.9 to 23.1 with several subtask drops, and the PJBenchmark gains are concentrated in two algorithm tasks. There are no error bars or significance tests, so “significant improvements” is doing too much work. Bigger than that is the stress-test point, and I think it lands: Listing 1 Step 2 defines `concatenate(x_q, states)` without saying how pooled embedding vectors become model input. Qwen3-Embedding-8B takes text tokens, not 4096-dim vectors. If the states are serialized into text, the gains could be a prompt-length or token-content effect rather than temporal unrolling. If they are injected into hidden layers, that requires a nonstandard forward pass that is neither described nor supported by the cited model interface. Either way, the comparison across T is confounded, and the central causal story—extra iterations activate pretrained reasoning in hidden space—is not established.\n\nAlso, T is chosen per task after scanning 1–10. That is post hoc selection, and the paper does not report results with held-out T. The authors should either prespecify a rule based on task complexity or show that the choice does not drive the results.\n\nWho is this for? Someone working on embedding models or RAG who is curious whether iterative refinement can help complex retrieval. The idea is cheap to test and the paper gives a plausible direction. But as written, it is not reproducible and the evidence is mixed. I would not desk reject it—better to send it to peer review and ask for code, a precise state-injection mechanism, and controlled baselines that add comparable input tokens without iteration. If the implementation turns out to be input augmentation, the paper mostly collapses; if it is true hidden-state reuse, it becomes a useful incremental contribution. Worth a referee cycle, not worth citing yet.","headline":"RT is a plausible test-time refinement idea, but Listing 1 leaves the core state-injection mechanism unspecified, so the reported gains are not yet interpretable.","tokens_in":6750,"tokens_out":1709,"would_cite":false,"duration_ms":15971,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Test-time iteration over a query embedding—feeding each pooled state back through the same decoder-only model—improves semantic reasoning in retrieval, with gains up to 21% and no retraining.","keywords":["test-time inference","text embeddings","semantic reasoning","temporal unrolling","retrieval","decoder-only models","chain-of-thought analogue","embedding model reasoning"],"falsifier":"Fix T=2 and compare RT against a control that appends the same number of fixed, query-independent tokens at each iteration. If accuracy stays at the same level, token addition is doing the work; if it drops to the T=1 baseline, temporal unrolling is the cause. A second check: implement state injection into a specific hidden layer and compare with a text-serialized version on the same tasks.","tokens_in":5822,"feed_emoji":"🔁","tokens_out":7284,"duration_ms":58031,"temperature":0.7,"pith_summary":"RT (Refine Thought) is a test-time method that improves the semantic reasoning of text embedding models by running several forward passes on the query. At each pass the model's own pooled embedding state is fed back in, and the final state is used as the query representation. The paper reports clear gains on reasoning-heavy retrieval benchmarks (BRIGHT, PJBenchmark) while leaving general similarity benchmarks like C-MTEB essentially unchanged, and the gains grow with task complexity and peak at two to three iterations. The authors explain this as temporal unrolling: the extra steps activate reasoning ability that decoder-only embedding models acquired during pretraining, so retrieval systems can reason better without retraining.","feed_headline":"Looping the query through the encoder lifts complex retrieval by 21%","feed_subtitle":"No retraining, no index rebuild: extra forward passes sharpen difficult matching tasks.","key_machinery":"The central mechanism is the RT loop: after each forward pass, all token representations before the [EOS] token are mean-pooled into a state h_t; the next pass encodes the concatenation of the original query and all accumulated states; after T steps, the last pooled state becomes the query embedding. This is temporal unrolling 'implicitly in the hidden space'—the embedding-model analogue of chain-of-thought, but without generating any text. The recurrence is h_t = f(x, h_1, ..., h_{t-1}), and the key assumption is that decoder-only autoregressive models can integrate these recycled states to refine their representation.","core_discovery":"On its own terms, RT's central claim is that semantic reasoning is a multi-step computation and a single forward pass of an embedding model lacks the computational depth to carry it out. RT turns the query encoder into a recurrent system: for T steps it re-encodes the query together with the mean-pooled hidden states of all previous steps, and uses the last pooled state, h_T, as the query embedding. Across BRIGHT subtasks the overall average rises from 22.9 to 23.1; on PJBenchmark, JD2CV jumps from 62.77 to 74.33 and CV2CV from 52.14 to 63.26; C-MTEB STS scores stay flat. The authors attribute the effect to 'activating the semantic reasoning ability learned during pretraining' in decoder-onl","pith_inferences":["The paper leaves unspecified how the high-dimensional pooled states are fed into the decoder. If they are serialized as text, the gains could come from added tokens rather than recurrence; a control with appended random or fixed tokens would separate the two.","If the mechanism is genuine hidden-state recurrence, RT should also improve arithmetic and symbolic embedding tasks, which the paper does not test.","The paper's own limitation discussion in Section 4.3 highlights sensitivity to step count and state aggregation; a practical extension would couple RT with an early-stopping or drift signal to control cost.","The large gap between decoder-only and encoder-only models suggests a testable prediction: giving an encoder-only model an explicit recurrent state-injection module should recover part of the gap, but only if state feedback is implemented in the hidden space rather than as text."],"forward_implications":["Reasoning-heavy retrieval can be improved by spending a few extra forward passes on the query alone; document embeddings and indexes remain unchanged.","The gain saturates at T=2-3, so the latency cost is small and predictable, with diminishing returns beyond.","Decoder-only embedding backbones become the preferred choice for tasks that require compositional or multi-constraint matching.","Since simple similarity tasks are unaffected, the method can be applied selectively to hard queries without degrading general retrieval.","The results give empirical support to the theoretical view that temporal unrolling trades time steps for effective computational depth."],"fun_headline_variants":["Loop the query, lift reasoning: up to 21% on hard tasks","Test-time loops unlock reasoning in embedding models","No retraining, just repeat forward passes for better matching","Recurrent query encoding sharpens complex retrieval","Extra encoding steps improve reasoning, not general tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that `concatenate(x_q, states)` in Listing 1 is a well-defined way to feed previous pooled embedding states back into the decoder; the paper never specifies how high-dimensional state vectors become model input tokens, and its own limitation section concedes sensitivity to step count and state aggregation.","fun_headline_variants_meta":{"raw":{"variants":["Loop the query, lift reasoning: up to 21% on hard tasks","Test-time loops unlock reasoning in embedding models","No retraining, just repeat forward passes for better matching","Recurrent query encoding sharpens complex retrieval","Extra encoding steps improve reasoning, not general tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2064,"prompt_tokens":655,"completion_tokens":1409,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":1332}},"tokens_in":399,"tokens_out":1409,"duration_ms":12052,"temperature":1.0,"reasoning_tokens":1332,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T09:55:51.168560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix T=2 and compare RT against a control that appends the same number of fixed, query-independent tokens at each iteration. If accuracy stays at the same level, token addition is doing the work; if it drops to the T=1 baseline, temporal unrolling is the cause. A second check: implement state injection into a specific hidden layer and compare with a text-serialized version on the same tasks.","supporting_citations":[],"review_version":1}