{"id":"1e159399-37d2-4490-ad95-9c8b7c967860","arxiv_id":"2603.12109","paper_version":2,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Outcome-based RL induces information self-locking in active-reasoning LLM agents via an Action Selection–Belief Tracking bottleneck; AREW reweights advantages with directional critiques to mitigate it.","lead":"Outcome-based RL can trap LLM agents in “information self-locking”: they neither seek useful feedback nor absorb what they get. The authors propose AREW, a credit-reweighting fix that they report yields large gains on agentic tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Abstract-only review leaves the independence of AREW's directional critiques unverified; that independence is the single load-bearing condition for the SeL diagnosis and the claimed non-circular mitigation.","rationale":"Only the abstract is present, so no theorems, ablations, critique protocols, or tables can be inspected. The Reader correctly isolated the circularity risk around “easy-to-obtain directional critiques” as the weakest assumption supporting both the SeL diagnosis and AREW’s claimed mitigation. That remains the single most load-bearing concern: if the critiques already encode reliable BT, the method does not break the bidirectional bottleneck it diagnoses. No stronger or different concern can be raised without the full text. The Reader’s CONDITIONAL verdict with LOW confidence is therefore appropriate and should stay unchanged; the concrete test above is the minimal check that would settle whether the concern lands once the paper is readable.","tokens_in":2114,"tokens_out":481,"duration_ms":5375,"concrete_test":"Once the paper is available, extract the exact critique generation procedure (prompt, model, human, or rule) and the independence claim. Re-run the main AREW experiments after replacing the critiques with (a) random directional labels and (b) critiques generated by a deliberately weak BT model that cannot solve the task. If performance collapses to near-baseline under (a) or (b), the critiques are not independent of BT and the non-circularity claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that outcome-based RL induces SeL via a bidirectional AS–BT bottleneck, and that AREW mitigates it by reweighting advantages with “easy-to-obtain directional critiques.” For this to be a genuine fix rather than an external oracle that already solves BT, the critiques must (1) correctly label informative vs. uninformative steps and (2) be obtainable without reliable belief tracking or task understanding—the very capability SeL asserts is weak. The abstract asserts both properties but supplies no protocol, source, or independence argument. Without that, the 60-point gains and the SeL diagnosis itself remain circular: success may simply reflect smuggled BT signal rather than a reweighting that breaks the bottleneck. This is the same soft spot the Reader flagged; full text is required to check it.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript argues that outcome-based RL for LLM agents in active reasoning induces information self-locking (SeL): agents neither elicit informative observations nor internalize evidence once obtained. The authors attribute SeL to a bidirectional bottleneck between Action Selection (AS), which shapes the observation stream, and Belief Tracking (BT), which updates internal task understanding—weak BT blurs credit for informative actions, while weak AS starves BT of useful evidence. They propose AREW, an advantage-reweighting method that reallocates credit within trajectories using easy-to-obtain directional critiques, and report that it substantially mitigates SeL with gains of up to 60 points across 9 agentic tasks of varying complexity. Code is stated to be available.","tokens_in":2284,"tokens_out":901,"duration_ms":14485,"significance":"If the SeL diagnosis and the independence of AREW’s critiques hold under full scrutiny, the work would offer a useful conceptual account of why pure outcome RL can stall in interactive agent settings, plus a simple, practical mitigation. The AS–BT coupling framing is potentially transferable beyond the specific method. Significance is conditional on (i) a non-circular critique signal that does not already solve BT, and (ii) rigorous empirical support (baselines, ablations, variance) for the large reported gains. On the abstract alone those conditions cannot be confirmed; if they are met in the full paper, the contribution would be of clear interest to the LLM-agent RL community.","major_comments":[{"comment":"Load-bearing independence of directional critiques (Abstract: “uses easy-to-obtain directional critiques to reallocate credit within trajectories”). For AREW to break the claimed AS–BT bottleneck rather than import an external BT oracle, critiques must correctly distinguish informative vs. uninformative steps without already requiring reliable belief tracking or task success. The abstract asserts this but supplies no generation protocol, accuracy characterization, or independence argument. This is the single condition on which both the SeL diagnosis and the non-circularity of the fix rest; it must be demonstrated explicitly (protocol, ablations with noisy/random critiques, comparison to oracle BT).","section":null},{"comment":"Empirical claim of “up to 60-point gains … across 9 agentic tasks” cannot be assessed from the abstract. Without task definitions, baselines (including strong outcome-RL and process-supervision controls), ablations isolating reweighting vs. critique content, error bars, and training curves, the magnitude and generality of the result remain unverified. These results are load-bearing for the claim that AREW “significantly mitigates SeL.”","section":null},{"comment":"The bidirectional AS–BT bottleneck is presented as established by “theoretical and empirical analyses,” but no theorem statements, proof sketches, or measurement definitions appear in the available text. The causal story (weak BT → obscured credit → weak AS → starved BT) is central; it needs a concrete formalization and a falsifiable empirical signature that distinguishes SeL from generic credit-assignment difficulty or exploration failure.","section":null}],"minor_comments":[{"comment":"Abstract-only review: notation for AS, BT, SeL, and AREW is introduced without formal definitions; the full manuscript should define these early and consistently.","section":null},{"comment":"The phrase “easy-to-obtain” for directional critiques should be operationalized (source model, prompt template, cost relative to outcome reward) so readers can judge deployability.","section":null},{"comment":"Code link is given; the camera-ready version should pin commit/hash and document how to reproduce the 9-task suite and the critique pipeline.","section":null}],"recommendation":"uncertain","confidential_remarks":"This is an abstract-only review; full text was not available. I cannot responsibly recommend accept/revise/reject without the theory, critique protocol, and experimental tables. The circularity risk around directional critiques is the primary item I would press in a full review. If the full paper cleanly shows critiques that remain useful under controlled degradation and that do not presuppose BT, the contribution looks promising; if not, the 60-point gains may largely reflect smuggled supervision. Please provide the full manuscript for a definitive recommendation."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know: this abstract claims that outcome-based RL for active-reasoning LLM agents systematically produces information self-locking—agents neither ask for informative feedback nor absorb what they get—because Action Selection and Belief Tracking form a bidirectional bottleneck, and that a simple advantage reweighting (AREW) with directional critiques breaks it with up to 60-point gains on nine tasks plus public code.\n\nWhat looks new and useful if it holds is the framing. Sparse reward and credit assignment are old, but coupling AS and BT as a mutual lock that weakens both learning signals is a clean diagnosis for multi-step tool-using agents. Naming SeL and shipping a cheap reweighting method is the kind of concrete contribution the agentic-RL subfield can actually use. Code is promised; that is real credit if the repo matches the claims.\n\nThe soft spot is exactly the one the stress-test flags, and it is load-bearing rather than minor. AREW’s critiques must correctly mark informative vs uninformative steps without already requiring reliable belief tracking—the capability SeL says is weak. The abstract calls them “easy-to-obtain” but gives no protocol, source, or independence argument. If the critiques smuggle in BT signal or task success, the gains are partly an external oracle, not a pure reallocation that breaks the bottleneck. Free parameters (reweighting schedule, critique policy) and invented entities (SeL, AREW) are fine as long as they are measured, not just narrated. We cannot check theorems, ablations, baselines, error bars, or task definitions from the abstract alone, so soundness is provisional.\n\nWho it is for: people training multi-turn tool-using LLM agents with outcome RL. A serious referee should see the full paper—especially critique independence, SeL measurement, and whether the 60-point numbers survive proper controls. I would not desk-reject on the abstract; the claim shape is coherent and the problem is real enough to deserve referee time. Bring it to reading group only after the full text and code are in hand; until then treat the numbers as unconfirmed.","headline":"Abstract-only package names a real-looking agentic-RL pathology (SeL) and a simple reweighting fix with big claimed gains; the load-bearing risk is whether “easy” directional critiques are independent of the BT skill SeL says is missing.","tokens_in":2969,"tokens_out":553,"would_cite":false,"duration_ms":4592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Outcome-based RL for LLM agents can self-lock information gathering; reweighting advantages with directional critiques unlocks large gains.","keywords":["information self-locking","active reasoning","LLM agents","reinforcement learning","action selection","belief tracking","advantage reweighting","AREW"],"falsifier":"Train identical agents with and without AREW on a controlled active-reasoning task whose ground-truth informative actions and belief states are known; if the AREW agent still fails to select those actions or to update its internal state correctly at rates comparable to the baseline, SeL is not the operative failure mode or AREW does not break it.","tokens_in":2945,"feed_emoji":"🔓","tokens_out":626,"duration_ms":5281,"temperature":0.7,"pith_summary":"This paper argues that when LLM agents must actively seek observations to solve multi-step tasks, ordinary outcome-based reinforcement learning systematically fails: the agent neither asks for the right evidence nor absorbs the evidence it does receive. The authors call this failure information self-locking (SeL). They decompose the problem into two tightly coupled skills—Action Selection (choosing what to query or do next) and Belief Tracking (updating an internal picture of the task from the returned observations)—and show that each skill’s weakness starves the other of a usable learning signal. The result is a bidirectional bottleneck that collapses exploration and internalization. To break the lock, they introduce AREW, a lightweight advantage-reweighting scheme that uses simple directional critiques of trajectory steps to redistribute credit, so that informative actions and correct belief updates receive clearer gradients. Across nine agentic benchmarks the method yields gains as large as sixty absolute points, suggesting that SeL is both common and fixable without redesigning the entire RL pipeline.","feed_headline":"RL locks LLM agents out of their own evidence; simple reweighting unlocks them","feed_subtitle":"Directional critiques break the Action–Belief bottleneck and recover up to 60 points on nine tasks","key_machinery":"Information self-locking (SeL) is the named failure mode, realized by the bidirectional bottleneck between Action Selection (AS) and Belief Tracking (BT). AREW is the proposed remedy: a credit-reallocation rule that multiplies ordinary advantages by directional critique scores so that informative AS steps and correct BT updates receive stronger learning signals.","core_discovery":"Outcome-based RL induces information self-locking (SeL) in active-reasoning agents: because Action Selection and Belief Tracking form a bidirectional bottleneck, the agent fails both to elicit informative feedback and to internalize the feedback it obtains; reweighting advantages with easy directional critiques (AREW) breaks the lock and recovers large performance gains.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Outcome RL locks agents out of evidence via Action–Belief bottleneck","SeL: RL agents fail to elicit or internalize feedback they need","AREW reweights advantages to break Action–Belief self-lock","Directional critiques unlock 60-point gains from SeL on nine tasks","Bidirectional AS–BT bottleneck freezes credit for informative acts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The directional critiques used for reweighting are assumed to be both easy to obtain and independently reliable, without already requiring the very belief-tracking skill that SeL says is weak.","fun_headline_variants_meta":{"raw":{"variants":["Outcome RL locks agents out of evidence via Action–Belief bottleneck","SeL: RL agents fail to elicit or internalize feedback they need","AREW reweights advantages to break Action–Belief self-lock","Directional critiques unlock 60-point gains from SeL on nine tasks","Bidirectional AS–BT bottleneck freezes credit for informative acts"]},"model":"grok-4.5","effort":"low","cost_usd":0.002124,"raw_usage":{"total_tokens":876,"prompt_tokens":783,"num_sources_used":0,"completion_tokens":93,"cost_in_usd_ticks":21240000,"prompt_tokens_details":{"text_tokens":783,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":783,"tokens_out":93,"duration_ms":1077,"temperature":1.0,"reasoning_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T22:26:37.591994+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train identical agents with and without AREW on a controlled active-reasoning task whose ground-truth informative actions and belief states are known; if the AREW agent still fails to select those actions or to update its internal state correctly at rates comparable to the baseline, SeL is not the operative failure mode or AREW does not break it.","supporting_citations":[],"review_version":1}