{"id":"c1ba2524-e970-4d4a-a68e-d3441958804d","arxiv_id":"2608.06128","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CIPO rewards search agents when retrieved evidence changes their next reasoning action, and this dense evidence-use signal improves F1 on QA benchmarks.","lead":"Search agents trained with a new reinforcement learning reward learn to base their reasoning steps on newly retrieved evidence rather than on prior guesses. The method adds a masked-evidence comparison to standard outcome rewards and improves question-answering scores on seven benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EALR's masked-condition likelihood is an OOD attention pattern; whether it is a valid no-evidence counterfactual is unverified and is the load-bearing assumption.","rationale":"The reader's weakest assumption correctly identifies the validity of the evidence-masked counterfactual as the load-bearing point. My stress-test agrees and sharpens it: the mask creates an OOD attention pattern with a position gap, so ℓmask may not represent a no-evidence condition. This is not merely a theoretical quibble; if the test fails, the EALR reward does not measure what the paper claims, and the central mechanism—dense credit for evidence-dependent reasoning—is unsupported even if the empirical F1 gains remain. The reader's other concerns (missing variance estimates, closed-source judge for the behavioral claim) are real but secondary, because they affect the strength of the empirical evidence rather than the identification of the reward signal. The paper has genuine independent support: released code, a reproducible retrieval environment, consistent gains over strong baselines, and an ablation (Table 9) showing that random-context masking does not reproduce the effect, which indicates the signal is not generic context perturbation. That ablation, however, does not validate the masked likelihood against a true no-evidence rollout; it only shows span-specificity. My proposed check fills exactly that gap. If the test passes, the central claim stands and the conditional acceptance is appropriate; if it fails, the verdict should move toward rejection of the mechanism claim. Since my read does not alter the reader's assessment, the verdict remains UNCHANGED (conditional acceptance pending the counterfactual test).","tokens_in":17525,"tokens_out":6203,"duration_ms":64950,"concrete_test":"On a held-out set of at least 500 post-retrieval transitions from the initial policy (before CIPO training), compute for each transition: (a) ℓvis as in Eq. 3; (b) ℓmask with the paper's attention mask as in Eq. 4; and (c) ℓremove, the teacher-forced log-likelihood of the same next-turn action with E_i,t simply deleted from the context, so target tokens start immediately after h_i,t with contiguous positions and no mask. Define EALR_mask = (ℓvis − ℓmask)/m and EALR_remove = (ℓvis − ℓremove)/m. Then compute the mean absolute difference and the Spearman rank correlation between EALR_mask and EALR_remove across transitions. If the mean absolute difference exceeds roughly 0.1 nats/token, or the rank correlation is below about 0.8, the masked condition is not a faithful no-evidence counterfactual, and the EALR reward's interpretation as evidence-use credit would be invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EALR (Eq. 5) quantifies whether a post-retrieval action genuinely depends on retrieved evidence. This requires the evidence-masked likelihood in Eq. 4 to approximate what the policy would assign to the same action if the evidence were absent. The mask is implemented by setting attention logits from target tokens to the immediately preceding information block E_i,t to −∞, while retaining token identities, sequence positions, and all other context. This produces an attention pattern—a causal mask with a contiguous block of −∞ over previous positions—that the model has almost certainly never seen during pretraining or any RL phase. Because softmax renormalizes over the remaining keys, the resulting likelihood may be arbitrarily distorted; EALR could then measure sensitivity to an OOD attention pattern rather than to evidence content. Moreover, target-token positional encodings still lie after the masked block, so the condition corresponds to 'evidence was retrieved but is unreadable,' not to 'no evidence was retrieved.' A faithful counterfactual would remove E_i,t from the sequence, shifting target positions back and leaving no position gap. The paper's ablations (Table 9, Appendix E) test which span is masked, not whether the masked condition is a valid no-evidence counterfactual. Since the entire method and the prior-driven-reduction claim rest on this identification, the unvalidated masking assumption is the most load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Contextual Information Policy Optimization (CIPO), a reinforcement learning framework for search agents that adds a dense, turn-level reward, the Evidence-Access Log-Likelihood Ratio (EALR), to a terminal outcome reward. EALR is computed by teacher-forcing the same next-turn action under evidence-visible and evidence-masked conditions through the frozen rollout policy and taking the token-averaged log-likelihood difference (Eq. 5). The paper shows that the expected value of the sequence-level ratio under the visible policy equals the KL divergence between visible and masked action distributions (Eq. 7). CIPO normalizes EALR rewards and outcome rewards within a rollout group, computes discounted advantages, and optimizes a clipped PPO objective with a KL penalty. Experiments on seven QA benchmarks (NQ, TQ, HotpotQA, 2Wiki, MuSiQue, Bamboogle, PopQA) with Qwen2.5-3B and 7B report consistent F1 improvements over outcome-only and process-supervised baselines, plus ablations on reward design and a counterfactual masking analysis. The paper also measures prior-driven vs evidence-driven reasoning with an LLM judge and reports reduced prior-driven rates.","tokens_in":17951,"tokens_out":7487,"duration_ms":76824,"significance":"Assuming the masked-likelihood identification is valid, CIPO is a clean and practical idea: it extracts a dense evidence-use signal from a single extra forward pass, requires no human process labels or an additional reward model, and is agnostic to the base RL algorithm. The KL derivation in Eq. (7) is mathematically correct, and the paper makes a good-faith effort to control confounds: all RL methods share the same retrieval environment, training data, rollout group size, and budget, and Appendix E explicitly tests whether the method merely rewards indiscriminate sensitivity to retrieved text. The reported gains are substantial (4.7-5.0 F1 points over the strongest baseline) and hold across two model scales and out-of-domain benchmarks; if reproducible, this is a meaningful advance for training grounded search agents. Code is released, which helps reproducibility.","major_comments":[{"comment":"The claim that EALR measures the dependence of the agent's action on retrieved evidence rests on interpreting πθold(a | h, M(E)) as the likelihood of the same action in a counterfactual 'no-evidence' condition. This identification is not validated. The attention mask that blocks the immediately preceding information block while keeping token identities, sequence positions, and all other context creates an attention pattern that the model has almost certainly never seen during pretraining or RL; softmax renormalization over the remaining keys means the masked likelihood can change for reasons unrelated to evidence content (e.g., due to the unusual position gap left by the masked block). The ablations in Table 9 vary which span is masked, but no experiment compares the block-mask condition with an actual deletion of E (with positions realigned) or with a different operationalization of evidence dependence (e.g., replacing E with a matched-length irrelevant passage). I request such a validation; if the masked-likelihood signal diverges from the deletion-based signal, the central conclusion that CIPO reduces prior-driven reasoning would be undermined.","section":"Section 3.1, Eq. (4)"},{"comment":"All experimental tables report a single run per configuration without error bars or significance testing. The headline improvements—4.7 F1 points average over IGPO/GiGPO in Table 1 and 5.0 points over GSPO in Table 3—could be within run-to-run variance for online RL with 16 rollouts per prompt. Please report at least three independent training runs per condition (mean±std) and a paired test across the seven benchmarks (e.g., Wilcoxon signed-rank) to support the claim of consistent superiority.","section":"Section 4, Tables 1-3"},{"comment":"The main behavioral evidence for the paper's thesis—that CIPO reduces prior-driven reasoning—comes from an LLM judge (GPT-5.5) that classifies trajectories as prior- or evidence-driven based on whether retrieved evidence influences reasoning. The EALR training reward is also a direct measure of the same construct (action-likelihood sensitivity to retrieved evidence). While the F1 improvements are independent of this metric, the behavioral claim is at risk of circularity because the evaluation metric is aligned with the optimized objective. The paper should include a human-evaluated subset (with inter-annotator agreement) and/or an observational measure (e.g., token-level attribution of answer spans to retrieved passages) to verify that the decrease in prior-driven rate is not an artifact of the judge being sensitive to the same surface features that EALR rewards.","section":"Section 4.4, Appendix F"},{"comment":"EALR credits only the immediate next-turn action (the next <think> and <search> blocks). Evidence can change a later reasoning step or the final answer without changing the immediate next action, and in that case the dense evidence-use signal contributes nothing at the turn where retrieval occurred. Appendix D compares with a full-trajectory variant, but that variant is itself confounded by later retrieved evidence and apparently performs worse; the paper does not quantify how often evidence-induced revisions occur only after more than one turn. Please report this frequency on the validation set and clarify whether the outcome reward alone is expected to cover such delayed evidence use.","section":"Section 3.1, Eq. (5), Appendix D"}],"minor_comments":[{"comment":"In the evidence bubble for the prior-driven example, 'V on Braun' should be 'von Braun' (the same typo appears in the figure text).","section":"Figure 1"},{"comment":"'With this manner' is not idiomatic; use 'In this way'.","section":"Abstract and Section 1"},{"comment":"The text says training is on the 'training splits of the in-domain datasets'—please specify whether the training data is the union of the four training splits or a subset, and how this matches the baselines' training data.","section":"Section 4.1"},{"comment":"γ is set to 1.0 in all experiments; no sensitivity analysis is reported for γ, β, ε, or the rollout group size G, so the robustness of the method to these free parameters is not demonstrated.","section":"Section 3.2, Eq. (9)"},{"comment":"The retrieval environment is said to be 'consistent with the Search-R1 environment architecture', but the search API, number of results returned, and snippet length are not fully specified; please provide these details to make the environment reproducible.","section":"Appendix C"},{"comment":"Sheng et al. (2025) is cited in the text as the 'verl framework' but the reference title given is 'Hybridflow: A flexible and efficient rlhff framework'; the relationship should be clarified.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear scope fit and the code release is commendable. My main worry for the editor is the gap between the conceptual claim (masked likelihood equals no-evidence counterfactual) and its validation; this needs to be resolved before publication. The absence of error bars is also a barrier to assessing the stability of the reported gains. No concerns about novelty disclosure or citation behavior."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing you should know: this is a real idea, not a repackaged baseline sweep. The EALR reward—token-averaged log-likelihood ratio between evidence-visible and evidence-masked teacher-forced scoring of the same next-turn action—is new as far as the cited literature goes, and the paper gives it a clean KL-divergence interpretation (Eq. 7). If it works, it gives the search-agent RL community a dense, reward-model-free credit signal for evidence grounding.\n\nWhat the paper does well: the ablations are genuinely informative. EALR alone raises supportive-evidence utilization from ~20% to ~45% while also raising irrelevant-evidence utilization; adding the outcome reward cuts irrelevant utilization and boosts F1. The counterfactual masking variants in Table 9 (random span, supportive-only, non-supportive-only) are a good-faith attempt to show the signal isn't just 'respond to any text'. The efficiency analysis is honest: one extra masked scoring pass, modest overhead.\n\nNow the soft spots. The biggest one is exactly the load-bearing assumption flagged in the stress-test note: attention-masking the previous information block during scoring is an OOD intervention. The model has never been trained with a contiguous -inf block over those positions, so the masked likelihood is an arbitrary renormalization over the remaining keys. The condition isn't 'no evidence was retrieved'—it's 'evidence is present but unreadable', with positions and token identities intact. Table 9 tests which span to mask, not whether masking is a valid no-evidence counterfactual. That's a real gap, and it matters because the paper's central claim—that EALR measures whether actions 'genuinely depend' on retrieved evidence—rests on that identification.\n\nSecond, all tables report single runs without error bars. Gains are a few F1 points; without variance estimates, we can't tell if the 4.7-point margin over IGPO is within noise. Third, the prior-driven/evidence-driven evaluation relies on GPT-5.5 and DeepSeek-V4-Flash judges with manual checking. The protocol is detailed and honest, but the headline behavioral claim is only as reliable as those closed-source judges.\n\nThe derivation itself is correct, and the circularity concern is mild: the training reward and evaluation metric both measure evidence dependence, but the eval is on held-out trajectories with an external judge, and the masked-likelihood signal isn't fitted to the judge labels.\n\nWho should use this: anyone working on RL for RAG or search agents. The EALR idea is worth building on, and the masking-counterfactual validity question is a good research problem in its own right. It deserves a serious referee, and I'd accept it with major revision: the counterfactual needs a validation study (e.g., compare masking to sequence deletion with re-normalized positions, or train with such masks), and the empirical claims need variance estimates.\n\nRecommendation: send it to review. With the masking assumption addressed, this would be a solid contribution.","headline":"CIPO's EALR reward is a genuinely new dense signal for evidence grounding, but the masked-condition counterfactual is an unvalidated OOD attention pattern; the paper deserves a serious referee but needs a validity check on that assumption before the claim is solid.","tokens_in":18334,"tokens_out":2862,"would_cite":true,"duration_ms":28595,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that rewarding search agents specifically for letting retrieved evidence influence their next reasoning action, rather than only for answer correctness, reduces confirmation bias and improves question-answering across…","keywords":["reinforcement learning","search agents","retrieval-augmented generation","evidence grounding","prior-driven reasoning","credit assignment","likelihood ratio","question answering"],"falsifier":"A decisive test is to remove the information block at inference time for a frozen CIPO-trained policy and measure whether the immediate next action changes; if EALR does not predict which transitions actually reverse or alter when evidence is absent, the masked-likelihood counterfactual does not estimate no-evidence behavior, and the training signal is not measuring evidence grounding.","tokens_in":17357,"feed_emoji":"🔎","tokens_out":7564,"duration_ms":70410,"temperature":0.7,"pith_summary":"Search agents retrieve evidence into context, but standard reinforcement learning rewards final answer correctness or coarse progress, so an agent can earn full credit for reasoning that merely confirms what it already believed from memory. CIPO measures, at each post-retrieval turn, how much the observed next thinking-and-search action depends on the just-retrieved information, by comparing its likelihood with and without attention access to that information. This Evidence-Access Log-Likelihood Ratio (EALR) becomes a dense turn-level reward, combined with a normalized outcome reward to preserve accuracy. The paper reports that this reduces measured prior-driven reasoning and improves F1 across seven question-answering benchmarks, without human process annotations or a separate reward model.","feed_headline":"Train search agents to use, not just retrieve, evidence","feed_subtitle":"A likelihood-ratio reward cuts prior-driven reasoning and lifts QA scores across seven benchmarks.","key_machinery":"The Evidence-Access Log-Likelihood Ratio (EALR) is the central object: for each turn $t$, EALR is the token-averaged log of the ratio of the likelihood of the observed next-turn action when the previous information block $E_{i,t}$ is visible to its likelihood when attention to $E_{i,t}$ is masked during teacher-forced scoring (Eq.\\ 5). The mask changes only the attention edges from the scored action tokens to that preceding block; positions, token identities, and all other context stay fixed, so the ratio isolates the influence of the new evidence. EALR is normalized within a rollout group and accumulated into a discounted turn-level advantage together with the outcome reward before the clipped policy update, which is what actually reshapes the policy. The paper also notes that the expectation of the unnormalized log-ratio under the evidence-visible policy equals the KL divergence between the visible and masked action distributions, giving the signal a policy-level interpretation.","core_discovery":"The central claim is that prior-driven reasoning in search agents can be directly penalized by a local counterfactual: for each transition from a retrieved information block to the next thinking-and-search action, compute the token-averaged log-likelihood of that action with normal attention to the evidence minus the same action's log-likelihood when attention to the evidence block is masked. A positive difference means seeing the evidence made the actual action more probable; a difference near zero means the action was equally likely without the evidence. The paper asserts that using this difference as a dense reward, together with a global outcome reward, trains the policy to let retrieved facts guide or revise subsequent reasoning instead of merely confirming parametric guesses. In the paper's experiments this reduces prior-driven behavior and improves performance on most of the seven evaluated benchmarks.","pith_inferences":["An implicit consequence is that EALR could serve as a diagnostic grounding meter for any agentic system: by comparing rollout actions with and without attention access to a context block, one could audit whether a deployed agent actually conditions on retrieved or tool-provided information.","The next-turn scoring choice means evidence whose effect shows up only several actions later, such as changing the final answer without changing the immediate next search, receives no direct credit; a variant that scores a window of future actions could test whether longer-horizon grounding credit is needed.","A possible failure mode the paper does not fully rule out is reward hacking of the contrast: a policy could inflate EALR by generating verbose thought blocks that make the masked-visible likelihood gap large without genuinely changing its conclusions, so an independent check comparing EALR against human judgments of evidence use would be a natural next experiment."],"forward_implications":["After training with CIPO, a search agent should be more likely to revise or abandon an initial hypothesis when later search results contradict it, because continuing to act as if the evidence were absent lowers the discounted advantage.","EALR adds one masked scoring pass per post-retrieval turn during training and no extra scoring at inference; the paper's efficiency numbers show the relative F1 gain exceeds the relative wall-clock increase for both tested model sizes.","Because the signal is computed from a tagged information block followed by policy-generated tokens, it transfers to other settings where tool outputs or database results are inserted into context, without re-annotating process rewards.","Combining EALR with outcome supervision raises supportive-evidence utilization while lowering irrelevant-evidence utilization, indicating the policy becomes more selective about which retrieved content it depends on.","Held-out benchmarks improve even though training used only in-domain data, suggesting the learned evidence-dependence behavior generalizes beyond the training distribution."],"supporting_citations":[{"why":"Search-R1, an outcome-only RL search agent, defines the setup CIPO improves on and serves as the main outcome-only baseline.","marker":"Jin et al. 2025"},{"why":"IGPO, the process-supervised baseline measuring turn-wise progress, is the closest comparison and the source of the prior-driven analysis.","marker":"Wang et al. 2026b"},{"why":"GiGPO, which estimates group-relative step advantages, is another process-supervised method CIPO must beat.","marker":"Feng et al. 2026"},{"why":"DeepResearcher, an outcome-only deep-research RL baseline, is compared against in the experiments.","marker":"Zheng et al. 2025"},{"why":"Documents when parametric memory fails on long-tail facts, motivating the prior-driven problem, and provides the PopQA out-of-domain benchmark.","marker":"Mallen et al. 2023"},{"why":"Retrieval-augmented generation, the foundation of the RAG paradigm and the CoT+RAG baseline.","marker":"Lewis et al. 2020"},{"why":"PPO, the clipped policy objective that CIPO's update adapts.","marker":"Schulman et al. 2017"}],"fun_headline_variants":["Counterfactual reward grounds search agents in evidence","Search agents trained to be grounded in evidence, not priors","CIPO: reward actions that truly use evidence, not just retrieve","Evidence-grounded counterfactual reward improves search QA","Evidence reward reduces prior-driven reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that attention-masking the previous search-result block while scoring the same action faithfully estimates what the agent would have done without that evidence, even though the model was never trained with such masks.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual reward grounds search agents in evidence","Search agents trained to be grounded in evidence, not priors","CIPO: reward actions that truly use evidence, not just retrieve","Evidence-grounded counterfactual reward improves search QA","Evidence reward reduces prior-driven reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001261,"raw_usage":{"total_tokens":5150,"prompt_tokens":914,"completion_tokens":4236,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":4160}},"tokens_in":530,"tokens_out":4236,"duration_ms":32683,"temperature":1.0,"reasoning_tokens":4160,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:49:07.160040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to remove the information block at inference time for a frozen CIPO-trained policy and measure whether the immediate next action changes; if EALR does not predict which transitions actually reverse or alter when evidence is absent, the masked-likelihood counterfactual does not estimate no-evidence behavior, and the training signal is not measuring evidence grounding.","supporting_citations":[],"review_version":2}