{"id":"965be598-7e06-4103-8a43-17f862288dde","arxiv_id":"2505.07596","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A reinforcement-learned question-answering agent, IKEA, uses a reward favoring correct answers with fewer searches and a balanced easy/hard training set to cut retrieval frequency while keeping or improving accuracy.","lead":"This paper trains an AI search agent to answer questions from memory first, and to look things up online only when its memory is not enough. It reports better accuracy with fewer web searches than standard retrieval-augmented models on several question-answering benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe-defined easy/hard splits in Section 3.2 and Appendix B self-select the test subsets, so the claimed accuracy-per-search gains over Search-R1 may be artifacts of that selection.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the easy/hard probing procedure in Section 3.2 is used both to construct training data and to build the test subsets in Section 4.1 and Appendix B, so the reported EM and retrieval reductions may be artifacts of the selection. My reading of the paper confirms this: Appendix B states 'We use the Qwen2.5-3B-Instruct as the sampling model' and 'There are 512 examples in each subset', and Section 4.1 says test sets were 'constructed like the training set'. The probe labels a question easy if the model answers correctly at least once in N sampled trials, which is a weak and noisy criterion. The central behavioral claim, that the trained agent can delineate its own knowledge boundary and exploit parametric knowledge, is then evaluated only on questions whose boundary is defined by a different, fixed probe model. The paper's own limitations section lists reliance on dataset construction and probing, grid-searched reward parameters, and computational expense; these are not hidden. The checkpoint issues in Table 1 (e.g., Search-R1-Zero-3B marked over-optimized with missing RT, and ablation Table 2 showing a 2Wiki-Hard EM of 0.21 for w/o r_kb- that looks like a value off by a factor of ~100) further weaken the empirical support, but the probe-selection issue is the most load-bearing because it directly conditions the headline claim. A full-benchmark or held-out-probe evaluation would settle whether the effect is real. The verdict should remain CONDITIONAL: the method is coherent and ablations support the reward design, but the central claim is not yet rigorously supported.","tokens_in":20969,"tokens_out":2662,"duration_ms":20049,"concrete_test":"Retrain or re-evaluate IKEA on the full original test sets without the easy/hard filtering, or on a held-out split constructed with a different probe model (e.g., Qwen2.5-7B-Instruct or a stricter majority-vote self-consistency probe). If the EM improvement and RT reduction over Search-R1 persist on the unfiltered benchmarks, the probe-selection artifact concern is resolved. Additionally, report per-subset standard errors or bootstrap confidence intervals over at least 5 seeds.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that IKEA learns to answer from parametric knowledge when possible and retrieves externally only when needed, improving accuracy-per-search over Search-R1. The mechanism is trained and evaluated using a knowledge-boundary probe: a question is labeled Qeasy if Qwen2.5-3B-Instruct answers correctly at least once in N sampled CoT trials, and Qhard otherwise. Appendix B reports that each test subset contains only 512 examples per dataset. Several load-bearing assumptions are insecure here. First, the probe is a single model with a single prompting scheme, and 'correct at least once' is a weak criterion: a question on which the probe succeeds once but fails often is still labeled easy, and the probe is never recalibrated on the trained IKEA agent. Second, the same probe that defines the training signal is also used to create the test subsets, so the reported EM gains and RT reductions are measured on a selection that is aligned with the training construction. Claims about generalization to 'hard' questions then rest on the assumption that probe difficulty transfers to trained agents of different sizes (3B and 7B). Third, Section 7 explicitly concedes that the reward parameters (r_kb+ = 0.6, r_kb- = 0.05, RT_max = 3) require grid searching, and the paper acknowledges high computational cost; the hyperparameter sensitivity is therefore not established. These concerns do not invalidate the approach, but they mean the abstract's 'significantly outperforms' claim is not yet supported beyond the specific probe-defined splits.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IKEA, a reinforcement-learning-based search agent that aims to decide when to answer from parametric knowledge and when to retrieve external knowledge. The method has three components: an agent prompt template that encourages internal-knowledge recall before searching; a knowledge-boundary aware reward function (Eqs. 3-4) that rewards correct answers, penalizes unnecessary retrievals on easy questions, and rewards retrieval on hard questions; and a training dataset balanced 1:1 between probe-defined easy and hard questions. Experiments on NQ, PopQA, HotpotQA, and 2Wiki, split into easy and hard subsets, report exact match and retrieval counts against direct, RAG, adaptive-RAG, and RL search baselines. The central claim is that IKEA significantly outperforms baselines, especially Search-R1, while substantially reducing retrieval frequency and generalizing out-of-distribution.","tokens_in":21203,"tokens_out":4891,"duration_ms":46219,"significance":"If the empirical claims hold, IKEA is a practically useful training recipe for RL search agents: it is simple, it is released with code, and the ablations in Section 5 give informative evidence that both the reward shape and the easy/hard data balance matter. The training curves in Figure 2 are consistent with the intended mechanism of initially exploring retrieval and later pruning it. However, the evaluation currently has a load-bearing circularity: the same probing procedure that labels training data also constructs the test subsets, and the probe is run on a model from the same family as the trained agents. In addition, no error bars or significance tests are reported, while the abstract and Section 4.2 claim 'significantly' better performance. The contribution is therefore promising but not yet verified at the level claimed; the gap can be closed by full-benchmark evaluation and uncertainty quantification.","major_comments":[{"comment":"The test sets are constructed with the same probing procedure used to label the training data (Section 3.2), and the probe is Qwen2.5-3B-Instruct, a model from the same family as the trained agents. Table 1 therefore reports EM and RT averaged over subsets selected to align with the training signal; the headline gains over Search-R1 (e.g., +2.51 EM and -34.76% RT for IKEA-3B, +5.05 EM and -50.81% RT for IKEA-7B) may be artifacts of that selection rather than properties of the trained policy on the underlying benchmarks. Please report results on the full NQ, PopQA, HotpotQA, and 2Wiki test sets, and if the easy/hard split is retained, construct it with an independent probe and verify label stability across sampling seeds and model checkpoints.","section":"§4.1 and Appendix B"},{"comment":"The definition of Qeasy as 'the correct answer is obtained at least once' over N sampled trials is a weak and noisy criterion: a question answered correctly in only one of, say, eight trials is still treated as internal knowledge for reward shaping and later aggregated into the Easy test subset. The probe is also never recalibrated on the trained IKEA agent, and it is used to label questions for 7B and base models even though it is run on 3B-Instruct. Please report the probe's success-rate distribution, show how the easy/hard labels depend on N, and test whether the trained agents' retrieval behavior actually correlates with probe difficulty rather than with surface-level properties of the questions.","section":"§3.2, Eq. (4), and Table 1"},{"comment":"No error bars, confidence intervals, or significance tests are reported, and the authors explicitly state that computational cost prevented repeated runs. On 512-example subsets, EM differences of 2-5 points are plausibly within sampling noise, yet the abstract and Section 4.2 repeatedly use 'significantly outperforms' and 'significantly reduces' without statistical support. Report bootstrap confidence intervals on the EM/RT differences or full-benchmark results, and temper the language to match the evidence actually provided.","section":"Checklist item 7 and Table 1"},{"comment":"The reward parameters (rkb+ = 0.6, rkb- = 0.05, RTmax = 3) are acknowledged in Section 7 to potentially require grid searching, but no sensitivity analysis is provided. The ablation in Table 2 removes rkb or rkb- entirely but never varies their magnitudes or RTmax. Since the paper's central claim is that the knowledge-boundary aware reward is responsible for the retrieval reduction, the reader cannot tell whether the effect is robust across the plausible parameter range. Please add a parameter sweep or at least a small robustness study over rkb+, rkb-, and RTmax.","section":"§7 and Appendix D"}],"minor_comments":[{"comment":"There are several typos that should be fixed: 'indentify' in the abstract, 'trainign' in the captions of Figures 2-4, 'IEKA' in the Section 3.2 heading, and 'w/orkb' in Table 2.","section":"Abstract and throughout"},{"comment":"The Search-R1-Zero row reports no RT and uses a '***' footnote saying the checkpoint might be over-optimized; this should be explained directly in the table caption or main text so the reader knows why that baseline is excluded from the RT comparison.","section":"Table 1"},{"comment":"The definition of 'valid searches' (RT) is not precise. Please specify whether malformed search actions, searches that return no documents, and repeated searches for the same query count toward RT, and state whether the reported numbers include only successful environment interactions.","section":"§4.1 and Appendix D"},{"comment":"The corpus is described only as 'wikipedia2018' and the retriever as 'e5-base'; for reproducibility, please give the exact corpus version, the exact retriever checkpoint (e.g., e5-base-v2), and any preprocessing or filtering applied to retrieved documents.","section":"Appendix D"},{"comment":"The paper states that data and code are provided in supplementary materials and gives a GitHub URL in the header, but no license is mentioned. Please add a license for the released code and data.","section":"Checklist and GitHub repository"},{"comment":"Reference [46] contains a typo ('facutality'), and some references are missing venue or year information (e.g., [3]); a final proofreading pass of the reference list is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the authors' checklist answers are unusually candid, particularly item 7 ('No' to statistical significance), but the main text's use of 'significantly' conflicts with that answer. The strongest risk is circularity between the probe used to build training data and the probe used to build the test subsets; reporting full-benchmark results would largely resolve it. The paper also appears to be a preprint with an author-provided GitHub URL, so please verify that the journal's submission policy permits the NeurIPS checklist format to remain in the published version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: IKEA is a plausible and well-motivated training recipe for RL search agents: a GRPO objective with an exact-match reward, a retrieval-count penalty that shrinks as searches drop, and a small reward for retrieving when the answer is wrong. The authors also build a training set as a 1:1 mix of questions a probe model can and cannot answer, which is a reasonable way to encourage both parametric and external knowledge use. The ablations in Section 5 are the strongest part — they actually show that removing the retrieval penalty or the negative reward component shifts behavior in the expected direction, and that the mixed easy/hard data matters.\n\nWhat's genuinely new is the knowledge-boundary-aware reward plus balanced data construction. I don't think prior RL search agents (Search-R1, R1-Searcher, DeepRAG) combine these particular incentives. The paper also ships code, gives full training details, and reports on both 3B and 7B models, including two out-of-distribution datasets. That's real work and worth taking seriously.\n\nNow the soft spots. The evaluation is built on easy and hard subsets created by the same probing procedure that labels the training data. Each subset is only 512 examples, and the probe is a fixed Qwen2.5-3B-Instruct prompt that is never recalibrated on the trained IKEA agents. That means the reported EM and retrieval reductions are measured on splits aligned with the training construction. Comparisons against Search-R1 on those splits are internally fair, but they don't support the abstract's \"significantly outperforms\" claim as a general property. You'd need at least a held-out sample from the full benchmarks, or error bars across seeds, to make that case. The paper itself concedes on statistical significance in the checklist and says the reward parameters require grid searching.\n\nThere are also some implausible numbers in the ablation tables — for example, 2Wiki Hard EM of 0.21 in Table 2 and PopQA Hard EM of 0.35 in Table 3. Those look like formatting errors from percentages, and they should be corrected before anything else.\n\nNone of this kills the central idea. The mechanism is coherent, the ablations support it internally, and the retrieval reduction is directionally large and consistent across model sizes. But as submitted, the headline claim is stronger than the evidence.\n\nWho this is for: anyone working on RL for adaptive retrieval or search agents. It deserves a serious referee. I'd send it out, but ask the authors for full-benchmark or representative evaluation, multiple seeds or confidence intervals, and a fix to the ablation tables.\n\nRecommendation: engage with it, conditional on those revisions.","headline":"A sensible RL recipe for teaching search agents when to skip retrieval, with solid ablations, but the headline gains are measured on probe-defined subsets and need full-benchmark validation.","tokens_in":21838,"tokens_out":3443,"would_cite":true,"duration_ms":33173,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language-model search agent can learn to know what it knows: answer from memory first and search only when memory is insufficient, cutting searches by a third to a half while improving accuracy.","keywords":["retrieval-augmented generation","reinforcement learning","knowledge boundary","adaptive retrieval","search agent","knowledge-intensive question answering","internal knowledge","external knowledge"],"falsifier":"Run the fully trained IKEA agent and the retrieval-always reinforcement-learning baseline on the complete, unfiltered versions of the four evaluation benchmarks, without any easy/hard splitting, and compare exact-match accuracy and mean retrieval counts over the whole test set. If the retrieval reduction approaches zero or accuracy falls below the baseline on the unfiltered sets, the reported efficiency gain is an artifact of the probe-built subsets rather than evidence of a learned knowledge boundary.","tokens_in":20708,"feed_emoji":"🔍","tokens_out":12042,"duration_ms":97891,"temperature":0.7,"pith_summary":"The paper sets out to show that reinforcement learning can teach a language-model search agent where its own knowledge stops, so the agent answers from memory when it can and calls the search engine only when it cannot. The proposed agent, IKEA, is trained with a reward that credits a correct answer, gives a bonus for answering with no retrieval, and adds a small penalty when a wrong answer was not checked against external sources. Training data is deliberately half questions the probing model can already answer and half it cannot, keeping both behaviors visible during exploration. Across single-hop and multi-hop knowledge QA benchmarks, IKEA reports better exact-match accuracy than retrieval-always and retrieval-never baselines while using roughly a third to a half fewer searches. If the results hold, adaptive retrieval timing does not need external classifiers or hand-designed policies: a reward signal alone can make the agent internalize the boundary between internal and external knowledge.","feed_headline":"Search agent that knows its own gaps searches up to 50% less","feed_subtitle":"A reward for using memory first cuts search-agent overhead while raising accuracy.","key_machinery":"The load-bearing mechanism is the knowledge-boundary aware reward function, $R = r_{ans} + r_{kb}$ with a -1 format penalty: $r_{ans}$ is 1 for an exact match and 0 otherwise, while $r_{kb}$ is $r_{kb}^{+}(1 - RT/RT_{max})$ when the answer is correct and retrieval count $RT$ is reduced, 0 for a wrong answer with no search, and $r_{kb}^{-}$ for a wrong answer that did search. Paired with it is the 1:1 easy/hard training set, where easy questions are those a same-family model answers correctly in at least one of several sampled chain-of-thought rollouts and hard questions are the rest. Together they give the group-relative policy optimization process a reward landscape in which internal knowledge is the cheap path to the correct answer and external search is the useful path only when internal knowledge fails; the ablations show that removing either the positive bonus or the negative penalty collapses the trained behavior into always-retrieve or never-retrieve.","core_discovery":"The central claim is that retrieval timing in a knowledge-intensive agent is a learnable policy, not a separate module: the model can discover its own knowledge boundary from reward alone. IKEA's reward is the exact-match answer reward plus a knowledge-boundary reward that decreases linearly with retrieval count when the answer is correct, is zero for a wrong answer with no retrieval, and is a small negative value for a wrong answer that did involve retrieval, with a total penalty of -1 for format violations. The training set is built by probing the base model with chain-of-thought exemplars, labeling a question easy if any sampled rollout yields the correct answer and hard otherwise, and mixing the two classes 1:1. The authors report that the trained agent uses about one search per easy question and slightly more on hard questions, beats the always-retrieve and never-retrieve reinforcement-learning baselines on exact match, and transfers to two out-of-distribution datasets. They interpret this as the model learning three behaviors at once: dividing queries into known and unknown, recalling internal knowledge inside the boundary, and searching outside it.","pith_inferences":["The same reward shape could be transferred to other tool-use settings where the model can answer from memory, such as code generation with API lookups or tabular QA with database calls; testing it there would show whether the learned 'search only when needed' policy is a general control principle or specific to text retrieval.","Because the reported retrieval counts are measured on probe-labeled easy and hard subsets, a natural extension is to evaluate on unfiltered benchmarks; the paper's own setup does not show how the agent's learned threshold behaves on questions the probe never classified.","The probe uses a model from the same family to build training labels, so an informative robustness check would be to rebuild the easy/hard dataset with a weaker and a stronger probe model and see whether the agent's boundary shifts accordingly.","The reward's success suggests that explicit know/unknown classifiers may be unnecessary for adaptive retrieval; this connects to the broader question of whether calibration of self-knowledge can be shaped purely through reinforcement learning."],"forward_implications":["On the paper's easy/hard test splits, IKEA beats the always-retrieve search-agent baseline by 2.5-5.5 exact-match points while using 35-50 percent fewer searches, consistently across 3B and 7B models.","The learned retrieval-timing behavior transfers to two out-of-distribution datasets, so the agent appears to acquire a general policy rather than memorizing when each benchmark wants a search.","Ablating the positive knowledge-boundary bonus makes the model over-retrieve, and ablating the negative penalty makes it under-retrieve; both components are needed for the balanced behavior.","Training on an all-easy dataset collapses retrieval and harms hard questions, while an all-hard dataset inflates retrieval and harms easy questions; the 1:1 mix is required for synergistic internal-external use.","Base and instruction-tuned models converge to similar reward and retrieval profiles, indicating the behavior can be learned without an instruction-tuned cold start."],"supporting_citations":[{"why":"the retrieval-always reinforcement-learning search-agent baseline that defines the efficiency comparison.","marker":"[16]"},{"why":"a concurrent RL search-agent baseline in the same always-retrieve family.","marker":"[33]"},{"why":"the internal-knowledge-only RL baseline that shows reasoning alone can activate parametric knowledge but fails on hard questions.","marker":"[5]"},{"why":"the source of the group-relative policy-optimization algorithm used to train IKEA.","marker":"[31]"},{"why":"the adaptive-retrieval baseline that frames retrieval timing as an MDP, contrasted with the direct reward approach.","marker":"[10]"},{"why":"the chain-of-thought-triggered retrieval baseline that hand-designed timing must beat.","marker":"[35]"},{"why":"the token-confidence retrieval-trigger baseline whose poor performance motivates reward-based timing.","marker":"[15]"},{"why":"the survey that supplies the knowledge-boundary concept used to define internal versus external knowledge.","marker":"[20]"},{"why":"Natural Questions, an in-distribution dataset used to build the easy/hard training and test subsets.","marker":"[19]"},{"why":"HotpotQA, the in-distribution multi-hop dataset used for training and easy/hard evaluation.","marker":"[42]"}],"fun_headline_variants":["Agent learns when not to search, cuts retrieval by half","Reward teaches AI to use memory first, search less","Search agent that knows its gaps retrieves 50% less","IKEA: RL agent that searches only when it must","LLM search agent learns its knowledge boundary, searches less"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the probing procedure that labels a question 'easy' when a same-family model happens to answer it correctly in at least one of several chain-of-thought samples reliably marks the true boundary of what the trained agent knows; if that probe is noisy, the accuracy gains and retrieval reductions measured on the probe-built easy/hard subsets may be an artifact of subset selection rather than a property of the trained agent.","fun_headline_variants_meta":{"raw":{"variants":["Agent learns when not to search, cuts retrieval by half","Reward teaches AI to use memory first, search less","Search agent that knows its gaps retrieves 50% less","IKEA: RL agent that searches only when it must","LLM search agent learns its knowledge boundary, searches less"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1322,"prompt_tokens":978,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":261}},"tokens_in":594,"tokens_out":344,"duration_ms":3455,"temperature":1.0,"reasoning_tokens":261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:12:51.217286+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fully trained IKEA agent and the retrieval-always reinforcement-learning baseline on the complete, unfiltered versions of the four evaluation benchmarks, without any easy/hard splitting, and compare exact-match accuracy and mean retrieval counts over the whole test set. If the retrieval reduction approaches zero or accuracy falls below the baseline on the unfiltered sets, the reported efficiency gain is an artifact of the probe-built subsets rather than evidence of a learned knowledge boundary.","supporting_citations":[{"cited_title":"Perception of knowledge boundary for large language models through semi- open-ended question answering","cited_arxiv_id":null,"evidence_quote":"HotpotQA, the in-distribution multi-hop dataset used for training and easy/hard evaluation."},{"cited_title":"Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy","cited_arxiv_id":null,"evidence_quote":"a concurrent RL search-agent baseline in the same always-retrieve family."},{"cited_title":"Deeprag: Thinking to retrieval step by step for large language models, 2025","cited_arxiv_id":null,"evidence_quote":"the adaptive-retrieval baseline that frames retrieval timing as an MDP, contrasted with the direct reward approach."},{"cited_title":"Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions","cited_arxiv_id":null,"evidence_quote":"the chain-of-thought-triggered retrieval baseline that hand-designed timing must beat."}],"review_version":1}