{"id":"37a9d50a-6541-43e4-a4b6-3ee431caae04","arxiv_id":"2608.12049","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TCA, a token-level credit assignment reinforcement learning framework, improves R@1 and MRR@10 over sequence-level RL baselines in generative document retrieval on MS MARCO and Natural Questions.","lead":"This paper trains generative document retrieval models with a reward for every decoded identifier token instead of one reward for the whole identifier, and reports improved top-ranking accuracy on MS MARCO and Natural Questions. It matters because the systems are built token by token but evaluated document by document, and per-token feedback is meant to close that gap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The token-level reward in Eq. (10) is hidden-state similarity to the gold trajectory, not a measured change in expected retrieval quality; the reported gains may reflect reference-model distillation rather than token-level credit assignment.","rationale":"Both the reader and I identify the same soft spot: Eq. (10) is where the paper's central claim depends on an unvalidated proxy. The paper's abstract says token rewards are estimated by measuring how each token decision changes expected retrieval quality, but Eq. (10) computes cosine similarity to a frozen reference hidden state. That is a substantive gap, not a wording issue: if the cosine is merely a hidden-state matching signal, the RL stage is a form of distillation or regularization, and the experiments do not test token-level credit assignment for retrieval relevance. The proposed control ablation separates three explanations: exact-match bonus (R_hit), relevance content of the true gold hidden states, and generic hidden-state matching (shuffled h*). The reader's CONDITIONAL verdict remains appropriate because the empirical improvements are plausible and the method is coherent as a training objective, but the paper should either revise its mechanistic claims or supply the control. The lack of code and seeds is a separate reproducibility limitation that further supports conditionality rather than rejection.","tokens_in":19493,"tokens_out":8061,"duration_ms":81296,"concrete_test":"Run a two-arm ablation on NQ-PQ and MS-TU with the default hyperparameters from Section 5.4: (1) set R_hit=0 in Eq. (12) and retrain TCA-GRPO; (2) retrain with R_hit unchanged but replace the cached h*_t for each query with the cached hidden states of a randomly selected different training query under a fixed shuffle, so the target carries no relevance information for the current query. Report R@1 and MRR@10 for all four conditions plus the TCA-GRPOseq baseline. If arm (1) loses most of the Table 2/4 gains, the exact-match bonus rather than the token-level reward is responsible. If arm (2) retains the gains, any fixed hidden-state target suffices and Eq. (10) is not measuring token-level relevance. Either failure would invalidate the credit-assignment interpretation of the headline result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 defines the token-level reward as r_traj = 1[t<=T] cos(h_i,t, h*_t), a cosine similarity between the current policy's hidden state and the frozen SFT reference's hidden state at the same absolute decoding position. This is not a difference in expected retrieval quality attributable to the token decision, although the abstract and Section 1 promise exactly that. Because the comparison is position-wise and uses the gold prefix as context, any candidate that deviates from the gold token at an early position receives a low cosine at every later position, even if the deviation leads to another relevant document; the reward therefore cannot distinguish helpful alternative token decisions from harmful ones. Since the reference is the SFT model and Eq. (19) adds KL regularization toward that same reference, the RL stage may largely be a hidden-state matching or distillation objective. The central claim that the gains are due to token-level credit assignment for retrieval relevance requires that this cosine be a faithful per-token relevance signal, but no validation against any per-token relevance measure is provided. In addition, R_hit in Eq. (12) and the construction of the sequence-level baseline in Table 4 are not specified, leaving open the possibility that the exact-match bonus, not the trajectory reward, drives the improvements. The mask in Eq. (14) also conflicts with Eqs. (10)-(11), zeroing positions t>=T where the reward is defined or the length penalty applies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TCA, a two-stage reinforcement-learning framework for generative document retrieval that assigns token-level rewards during DocID generation. After supervised fine-tuning, a frozen SFT reference model is used to cache the gold DocID hidden-state trajectory, and the policy's decoder hidden states at each generated position are scored by cosine similarity to the corresponding gold hidden state, augmented by an overflow length penalty and an exact-match bonus. These token rewards are then optimized with either GRPO or PPO. Experiments on MS MARCO and Natural Questions with TU and PQ DocIDs report consistent improvements in R@1 and MRR@10 over the DDRO baseline and over a sequence-level GRPO variant, together with ablations on constrained decoding, group size, KL coefficient, and a case study of gold-token probability shifts.","tokens_in":19821,"tokens_out":4048,"duration_ms":41050,"significance":"If the central claim holds, the paper makes a useful contribution: it provides a dense, reward-model-free training signal for generative retrieval and shows that the resulting gains are not tied to a single policy optimizer. The final evaluation is external retrieval quality, so the headline result is not circular, and the controlled comparison against sequence-level RL under the same optimizer is the right experimental design. The paper also gives credit for careful reporting of multiple benchmarks, two DocID types, statistical significance tests, and extensive ablations. However, the validity of the token-level reward itself is the load-bearing assumption: the reward is a closeness-to-gold-trajectory measure, and the manuscript does not yet establish that this measure is a faithful per-token relevance signal. The reported gains are consistent with hidden-state distillation, which would weaken the claimed connection between token-level credit assignment and retrieval quality.","major_comments":[{"comment":"The token-level reward is defined as the cosine similarity between the generated decoder hidden state and the gold-DocID hidden state at the same absolute position from the frozen SFT model. The abstract and Section 1 promise a reward that measures how each token decision changes expected retrieval quality, but Eq. (10) measures agreement with one fixed gold trajectory. Because the comparison is position-wise against the gold prefix, any generated token that deviates at an early position will tend to receive low cosine scores at all later positions even if the deviation leads to another relevant document; the reward cannot distinguish helpful alternative routing decisions from harmful ones. Since the reference is the SFT model and Eq. (19) adds KL regularization toward that same reference, the RL stage may largely be a self-distillation objective. This is load-bearing: the claim that the gains come from token-level credit assignment for retrieval relevance requires either an independent validation of the cosine measure against per-token relevance, or a substantially revised interpretation of what the reward measures.","section":"Section 4.4, Eqs. (10)-(12)"},{"comment":"The mask m_{i,t} is defined as 1 if position t is less than the length of the target DocID and 0 otherwise, which conflicts with Eq. (10), where r_traj is defined for t <= T, and with Eq. (11), where the overflow penalty r_len is applied for t > T. Under Eq. (14), the overflow positions and also the final gold position t=T are masked out from the reward and from the normalization denominators in Eq. (15), so the length penalty cannot influence training. The authors should clarify the intended definition of a valid non-padding position and ensure that Eqs. (10), (11), and (14) are mutually consistent; as written, the implementation is ambiguous.","section":"Section 4.4, Eq. (14)"},{"comment":"The exact-match bonus R_hit is introduced but never given a numeric value in Section 4.4 or in the implementation details of Section 5.4. Because this bonus is applied to every valid position of an exactly matched DocID, it is a strong sequence-level signal that could dominate the token-level trajectory reward, and the paper does not report the contribution of R_hit relative to r_traj. The authors should report the value of R_hit and provide an ablation that separates r_traj, r_len, and the exact-match bonus, otherwise it is possible that the headline gains are driven by the bonus rather than by the proposed token-level credit assignment.","section":"Section 4.4, Eq. (12)"},{"comment":"The sequence-level baseline TCA-GRPOseq is not specified anywhere in the paper. To support the Q3 claim that token-level credit assignment adds gains over sequence-level rewards while keeping the optimizer fixed to GRPO, the authors must define exactly which reward is used by TCA-GRPOseq (for example, document-level relevance, exact-match, or a terminal reward only), and state whether the group size, KL coefficient, and rollout procedure are identical to TCA-GRPO. Without this information, the comparison in Table 4 is not reproducible and the central ablation is incomplete.","section":"Section 6.4, Table 4"},{"comment":"The text states that the value head is trained with an MSE loss against GAE return targets R_{i,t}, but Eq. (23) writes the target as the raw token reward r_{i,t}. If the implementation follows the equation, the value baseline is trained on immediate rewards rather than returns, which changes the advantage estimates and the behavior of TCA-PPO. Please correct the equation or the surrounding text and report the actual objective used in the experiments.","section":"Section 4.6, Eq. (23)"}],"minor_comments":[{"comment":"The paired t-test is mentioned but the number of test queries, the variance across runs, and whether significance is computed on the full dev set are not reported; adding these details would strengthen the statistical claims.","section":"Section 5.2"},{"comment":"The sentence \"All variants are initialized from baseline-sft model\" is repeated in the table caption; the duplication should be removed.","section":"Table 4"},{"comment":"The GRPO advantage uses a position-wise normalization across candidates, but the paper does not explain how positions with only one valid candidate are handled; with G=8 the variance estimate at such a position is degenerate and the epsilon term is the only stabilizer.","section":"Section 4.5, Eq. (16)"},{"comment":"The KL reward shaping term uses log pi_theta_old - log pi_ref, while Eq. (19) uses a KL divergence; the relationship between the two KL formulations should be clarified.","section":"Section 4.6, Eq. (21)"},{"comment":"The caption and text describe a 10th-90th percentile range, but the figure appears to show only one shaded band; please confirm that the band corresponds to the stated percentile range.","section":"Section 6.5, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The main uncertainty is not the experimental tables but whether the proposed cosine-to-gold-trajectory reward is measuring what the paper claims. I would encourage the editor to ask for either a direct per-token validation (for example, correlation with token-level counterfactual retrieval quality) or a substantial reframing of the contribution as a hidden-state distillation method rather than a validated token-level relevance signal. The missing specification of R_hit and TCA-GRPOseq is also important because it directly affects the interpretation of the headline ablation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is a real empirical result: token-level trajectory rewards built from hidden-state cosine similarity to the gold DocID trajectory consistently beat sequence-level rewards in generative retrieval, on two benchmarks and under both GRPO and PPO. The gains on R@1 and MRR@10 over DDRO are in the 1.5–4% relative range and the ablation in Table 4 isolating reward granularity is the right experiment. If you work on generative retrieval training, this is worth your time. Second, the paper's framing overreaches. The abstract and Section 1 promise a reward that measures how each token changes expected retrieval quality, but Eq. (10) defines the reward as agreement with a frozen SFT reference model's hidden state at the same absolute position. That is not a difference in expected retrieval quality; it is a hidden-state imitation reward. Because the policy is also KL-regularized toward that same reference, the RL stage is largely self-distillation. That does not invalidate the empirical gains, but it means the paper is not actually demonstrating token-level credit assignment for relevance. The authors need to either reframe the contribution as a practical imitation reward or validate the cosine signal against a per-token relevance measure, which they do not currently provide. The other issues are smaller but still real: R_hit in Eq. (12) is never specified, so the exact-match bonus could be doing more work than the trajectory reward; the sequence-level baseline TCA-GRPOseq in Table 4 is under-defined; and Eq. (14) masks positions beyond the gold length, which contradicts the length penalty in Eq. (11). The case study is nice but only shows that the SFT and RL models rank the gold DocID differently, not why per-token credit is responsible. Credit where due: the main tables are consistent, the baseline reproduction effort is honest, and the conclusion acknowledges the extra training overhead and pipeline complexity. No code or seeds are released, which makes the open questions harder to resolve independently. The central empirical claim survives the stress test: token-level rewards do improve retrieval even if the explanation is imitation rather than credit assignment. This paper deserves a serious referee. Send it for review, but the reviewers should push for a clearer separation between the imitation-reward result and the credit-assignment claim, for the mask/R_hit definitions, and for code or at least detailed hyperparameter and seed reporting.","headline":"A solid, internally consistent empirical study showing that position-wise cosine rewards against a frozen SFT reference's gold hidden states improve generative retrieval over sequence-level RL baselines—but the paper sells this as measured token-level relevance when it is really an imitation signal, and a few under-specified equations need fixing before it is fully reproducible.","tokens_in":20354,"tokens_out":2161,"would_cite":true,"duration_ms":22788,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token-level credit assignment for each DocID decoding step improves generative retrieval by giving every token its own relevance reward.","keywords":["Generative retrieval","Token-level credit assignment","Reinforcement learning","DocID generation","Reward design","GRPO","PPO","Document ranking"],"falsifier":"On the same MS MARCO-TU and NQ-PQ setups, replace the gold hidden states in Eq. (10) with hidden states from a different document at the same token position, keeping the exact-match bonus and all hyperparameters fixed. If R@1 and MRR@10 still improve over the sequence-level baseline, the gains do not come from trajectory-specific credit assignment; if they disappear or reverse, the cosine-to-gold-trajectory reward is the operative mechanism.","tokens_in":19287,"feed_emoji":"🔍","tokens_out":10515,"duration_ms":93287,"temperature":0.7,"pith_summary":"Generative retrieval models produce documents by generating an identifier (DocID) token by token, yet they are judged only after the whole sequence is complete. This paper argues that a single document-level reward sent back to every token is too coarse: an early token can route decoding into a bad region of the DocID space, while later tokens may be locally plausible but lead to an irrelevant document. To fix this, it proposes a token-level credit assignment framework that rewards each decoding step by how close the generated hidden state stays to the hidden-state trajectory of the gold DocID from a frozen supervised reference model. Instantiating this with GRPO and PPO improves R@1 and MRR@10 over sequence-level reward baselines on MS MARCO and Natural Questions. The central claim is that fine-grained, position-wise rewards align token generation with document-level relevance better than whole-sequence rewards.","feed_headline":"Per-token rewards lift top-1 retrieval for generative search","feed_subtitle":"Giving each DocID decoding step its own relevance reward beats whole-sequence rewards on MS MARCO and Natural Questions.","key_machinery":"The load-bearing object is the token-level trajectory reward of Eq. (10), which compares the last-layer decoder hidden state before each generated token with the corresponding hidden state of the gold DocID, cached once under teacher forcing from a frozen SFT reference model, via cosine similarity. It converts a document-level target into a dense per-position signal, and is coupled with a length penalty and a global exact-match bonus. TCA-GRPO then normalizes these rewards position-wise across a group of candidates sampled for the same query to compute advantages, while TCA-PPO uses a value head on the prefix hidden state with generalized advantage estimation. This shared reward design is what lets the paper compare reward granularity independently of the optimizer.","core_discovery":"The paper's central claim is that the credit-assignment problem in generative retrieval is token-level, not sequence-level: each decoded token is a routing decision over the constrained DocID space, and the same final relevance score cannot distinguish which tokens helped or hurt. TCA therefore assigns a token-level reward $r_{i,t} = \\mathbb{1}[t\\le T]\\,\\cos(h_t^i, h_t^*)$ comparing the decoder hidden state of a generated token with the hidden state of the gold DocID at the same position, plus a length penalty for over-generation and an exact-match bonus for hitting the gold DocID. These rewards are turned into advantages either by position-wise normalization over a group of candidates (GRPO) or by a learned value function with generalized advantage estimation (PPO). On MS MARCO with title/URL DocIDs, TCA-GRPO raises R@1 from 38.24 to 39.60 and MRR@10 from 50.33 to 51.10 over the DDRO baseline, and on Natural Questions with PQ DocIDs it raises R@1 from 48.10 to 50.02 and MRR@10 from 54.32 to 56.10. The paper also reports that the effect is separable from RL alone: a sequence-level version of the same optimizer gains less than the token-level version. The authors close by acknowledging that the pipeline is more complex and adds rollout overhead.","pith_inferences":["An implication left implicit is that the method's ceiling is tied to the quality of the frozen SFT reference: if the reference model's hidden states are poor at representing relevance, the cosine reward may reward imitation rather than retrieval; updating or distilling the reference during training is a natural extension the paper does not test.","A testable consequence is that the position-wise group normalization matters most at early decoding positions, where the candidate space is large; ablating advantages by position should show larger gains early.","The same token-level reward design could be plugged into preference-based objectives such as token-wise DPO, not just GRPO and PPO, because it produces a per-token scalar signal.","The paper's stated cost—a supervised reference plus rollout overhead—invites a lighter single-stage variant, and whether it retains the gains is untested."],"forward_implications":["If token-level credit assignment is responsible for the gains, then retrieval quality is not limited by what sequence-level RL can achieve, and future generative retrievers can be aligned by position-wise relevance signals.","The same trajectory-reward design should transfer to other DocID types beyond title/URL and PQ codes, since it only requires hidden states of the gold trajectory.","The improvement is concentrated in R@1 and MRR@10, so token-level rewards mainly sharpen early precision; recall-oriented or listwise objectives remain complementary.","Both GRPO and PPO instantiations improve over SFT, so the token-level reward is optimizer-agnostic; however, group-relative normalization is more reliable than learned value estimation on discrete prefixed DocID states."],"supporting_citations":[{"why":"Provides the DDRO baseline, the reproduced SFT initialization, dataset preprocessing, and the TU/PQ DocID settings used in the main comparison.","marker":"[22]"},{"why":"Supplies GenRRL, the sequence-level reinforcement-learning-from-relevance-feedback baseline that token-level rewards are compared against.","marker":"[52]"},{"why":"Supplies group-relative policy optimization (GRPO), used to convert the token-level rewards into position-wise advantages in TCA-GRPO.","marker":"[30]"},{"why":"Supplies the clipped PPO actor-critic objective used in TCA-PPO.","marker":"[29]"},{"why":"Supplies generalized advantage estimation, used by TCA-PPO to compute token-level advantages from the value head.","marker":"[28]"},{"why":"Establishes the DSI generative-retrieval paradigm with DocIDs and serves as a generative baseline in the comparison tables.","marker":"[33]"}],"fun_headline_variants":["Per-token rewards beat sequence rewards for generative retrieval","Token-level credit assignment improves generative retrieval","Fine-grained token rewards sharpen document retrieval","Token-level rewards lift retrieval: per-token credit assignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward in Eq. (10) assumes that, at each decoding position, how close the model's internal state before predicting a token is to the internal state of the gold DocID at the same position measures how much that token decision contributes to document-level relevance; the paper does not validate this against any per-token relevance signal.","fun_headline_variants_meta":{"raw":{"variants":["Per-token rewards beat sequence rewards for generative retrieval","Token-level credit assignment improves generative retrieval","Fine-grained token rewards sharpen document retrieval","Token-level rewards lift retrieval: per-token credit assignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2998,"prompt_tokens":1070,"completion_tokens":1928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":1870}},"tokens_in":686,"tokens_out":1928,"duration_ms":16704,"temperature":1.0,"reasoning_tokens":1870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:18:25.453485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same MS MARCO-TU and NQ-PQ setups, replace the gold hidden states in Eq. (10) with hidden states from a different document at the same token position, keeping the exact-match bonus and all hyperparameters fixed. If R@1 and MRR@10 still improve over the sequence-level baseline, the gains do not come from trajectory-specific credit assignment; if they disappear or reverse, the cosine-to-gold-trajectory reward is the operative mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies GenRRL, the sequence-level reinforcement-learning-from-relevance-feedback baseline that token-level rewards are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the DSI generative-retrieval paradigm with DocIDs and serves as a generative baseline in the comparison tables."}],"review_version":1}