{"id":"8256a14f-d46c-44d7-8fb6-28cd4b050398","arxiv_id":"2412.11536","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A judge-trained 'I Know' classifier, fed the first 32 tokens of a model's answer, can skip retrieval for over half of queries on several QA datasets without lowering judge-scored answer quality.","lead":"The paper trains a small classifier to predict when a large language model already knows an answer, using an LLM judge to create training labels. This lets question-answering systems skip expensive retrieval for many queries, which matters because retrieval and reranking are the main cost in RAG pipelines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on SOLAR serving as both label source and final metric; without an independent QA metric, the reported savings may reflect learning the judge's biases rather than true parametric knowledge.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: both the training labels and the final metric come from the same SOLAR judge. I agree that this is the point on which the central claim hinges. The paper's empirical results are internally consistent and the method is plausible as a way to compress an LLM judge into a fast classifier, but the conclusion claims more: that the classifier detects whether an answer is stored in parametric memory. That interpretation requires the judge to be a reliable proxy for factual knowledge, and the paper provides no independent check. The manuscript itself flags the issue in Section 5 and the incomplete GPT-4o teacher row further underlines that validation is unfinished. I do not see an internal inconsistency in the experiments; the concern is external validity. If an independent QA metric confirms the benefit, the claim holds and the paper is a solid conditional contribution. If it does not, the retrieval savings are judge-specific and the central claim collapses to 'we can predict this particular judge,' which is a much weaker result. Because the reader already set CONDITIONAL with moderate confidence, this stress-test does not move the verdict; it sharpens the specific experiment needed to convert the condition into acceptance or rejection.","tokens_in":10299,"tokens_out":5855,"duration_ms":56522,"concrete_test":"Re-run the final QA evaluation for the Mistral-32 and SOLAR-32 rows of Table 1 using an independent correctness metric—exact match or token-level F1 against gold, or a different judge such as GPT-4o or Llama-3.1-70B—instead of LLMEval(SOLAR). Compare three policies on the same queries: full RAG, no RAG, and IK-threshold selection with thresholds fixed at 0.5 and at the reported best thresholds. If IK selection's independent-metric score is materially lower than full RAG (e.g., more than 2–3 points) while LLMEval stays flat, the retrieval savings are an artifact of matching the SOLAR judge. Additionally, compute the correlation between SOLAR's labels and independent correctness on a sample of 200–500 NQ queries; low correlation would directly invalidate the 'I Know' interpretation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the IK classifier detects whether an answer is stored in parametric memory and that skipping RAG is beneficial—depends on the judge being a valid measure of factual correctness. In Section 3.1, silver labels are produced by SOLAR as an LLM-as-judge; in Section 3.3, the final QA quality metric is LLMEval, also computed by SOLAR. The IK model is trained to reproduce SOLAR's Yes/No decisions, and the policy is then scored by the same judge. Consequently, the reported 80% accuracy and the retrieval savings measure how well the classifier predicts this one judge, not whether the underlying answer is factually correct. Section 4.4 states that 'SOLAR appears to be a more lenient judge,' and Section 5 itself asks 'Are we not just learning how to increase the judge score instead of detecting a lack of knowledge of the LLM?' but answers only with qualitative analysis. If SOLAR is lenient or biased—for example, toward its own model family—a plausible but wrong no-RAG answer can receive a high IK score and be routed away from retrieval, preserving LLMEval while degrading actual answer quality. The load-bearing assumption is that SOLAR's judgment correlates with true knowledge; this is not tested anywhere in the paper. The 'best point' thresholds are also selected on the evaluation set, which inflates the apparent benefit, but the more fundamental issue is the shared judge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains an 'I Know' (IK) classifier that predicts whether an LLM can answer a question without retrieval. Silver labels are produced by applying a judge LLM (SOLAR) to the model's answer, and the classifier is fine-tuned with adapters to output a Yes/No score, optionally taking the first 32 tokens of the generated answer as additional input. The IK score is then used to decide when to trigger RAG within the BERGEN framework. Experiments on six QA datasets report IK accuracy around 80%, retrieval reductions of more than 50% on several datasets, and a finding that about 20K training samples suffice when generated tokens are used.","tokens_in":10512,"tokens_out":7326,"duration_ms":65239,"significance":"The main strength of the paper is its concise demonstration that a lightweight classifier, trained on judge-produced labels and a few generated answer tokens, can route queries away from RAG while roughly maintaining the LLMEval score. The paper is transparent about single-run variance, points to public code, and includes useful analyses of training set size and answer-token length. If the circularity concern regarding the shared judge is adequately addressed, the method could be practically valuable for reducing RAG cost. As presented, however, the evidence does not yet separate 'learning the judge's preferences' from 'detecting parametric knowledge,' and the oracle-based threshold selection overstates the expected benefit. The significance is therefore conditional on additional validation.","major_comments":[{"comment":"The silver labels for IK training are produced by the SOLAR LLM-as-judge, and the final QA quality metric (LLMEval) is also computed by SOLAR. Because the IK classifier is trained to reproduce SOLAR's judgments and the retrieval policy is scored by that same judge, the reported accuracy and downstream gains mainly measure alignment with SOLAR's preferences rather than factual correctness. The Discussion (Section 5) acknowledges this concern but only offers qualitative analysis; an independent metric (e.g., exact match or F1 on NQ/TriviaQA, or a judge from a different model family) should be added to validate the central claim.","section":"Sections 3.1 and 3.3; Tables 1-2"},{"comment":"The best-point threshold is selected by maximizing LLMEval on the evaluation split itself. This is an oracle threshold selection and likely overstates the achievable benefit, since in practice the threshold must be fixed before seeing the test set. Please report a threshold-selection procedure on a held-out validation set and then evaluate the chosen threshold on a separate test set, or at least quantify the sensitivity by also presenting results at a predetermined threshold such as 0.5.","section":"Tables 1-5, 'Best Point' columns"},{"comment":"The paper states that 'SOLAR appears to be a more lenient judge' and argues that this leniency is 'necessary' for NQ because of multi-item answers. This observation directly bears on the validity of the learned labels. The paper should quantify agreement between SOLAR and the other judges (e.g., Llama-3.1-70B, Mixtral) on a sample, and show how label disagreements propagate to the IK classifier's decisions and final QA scores. Without such analysis, the claim that the IK score detects 'whether an answer is stored in parametric memory' is not established.","section":"Section 4.4 and Table 4"},{"comment":"The claim that 'the gain is mostly due to cases where non relevant documents were retrieved' is not supported by any quantitative evidence in the manuscript. Please include a breakdown of the cases where the IK policy chooses no-RAG but the RAG answer is better, and vice versa, along with judge scores or independent quality indicators for those cases. This would also help the reader evaluate whether the retrieval savings are obtained at the expense of correctness.","section":"Section 5"}],"minor_comments":[{"comment":"In the row for 0 tokens, the NORAG LLMEval is shown as 0.42, while all other rows show 0.62 for the same column; this appears to be a typo and should be corrected.","section":"Table 3"},{"comment":"The captions of Figures 2 and 3 appear to be swapped relative to their references in Section 4.2: Section 4.2 first refers to Figure 3 for the distribution of IK scores, whereas Figure 2 has the histogram caption, and the later reference to 'Figure 3 shows the strange behaviour of the SCIQ dataset' matches the evaluation curves. Please renumber the figures or fix the cross-references.","section":"Figures 2 and 3"},{"comment":"The row for '10K 32' appears to have misaligned or merged columns: '0.78 0.7 50' does not clearly separate the Best Point LLMEval, %Retr, and IK theta values. Please realign the table entries.","section":"Table 5"},{"comment":"The last row of Table 4 contains the incomplete placeholder 'idem GPT-4o as Judge to do to do'; this should be completed or removed before publication.","section":"Table 4"},{"comment":"The related work section would benefit from a short comparison with reference [19], which also trains a judgment model using generated answers; making the differences explicit would help the reader position the contribution.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short empirical study that reads like an extended abstract. The central idea is reasonable and the cost-related findings could be useful, but the evaluation with a single judge, the oracle threshold selection, and the absence of concrete baselines from the adaptive-retrieval literature would need substantial strengthening for a journal venue. If the journal's scope includes concise systems papers, the contribution is borderline and the requested revisions are essential."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper for the token-length and data-efficiency numbers, not for the headline 'we skip half of retrievals' claim, because that claim rests on using the same judge to create the training labels and to score the final answers.\n\nWhat's genuinely new here is the breadth of the empirical mapping. The core recipe—distill an LLM-as-judge into a binary 'I know' classifier and feed it a few generated tokens—is already in Self-RAG and the Slim Proxy Models work, and the paper cites both. But Déjean tests it across four model families, six datasets, several teacher models, token lengths from 0 to 128, and training set sizes from 5K to 83K. The result that 32 tokens is enough, that 20K samples suffice when tokens are included, and that string-based teachers fail while larger LLM teachers mostly transfer are useful, concrete data points for anyone building a RAG system.\n\nThe paper is admirably transparent about its setup: single runs, variance under 0.005, the unfinished GPT-4o teacher row ('to do'), and no code released. That last bit is a real gap—an efficiency paper should ship the classifier—but it's not a fatal flaw.\n\nThe soft spot is the one the paper itself voices in Section 5: SOLAR generates the silver labels and also produces the LLMEval score used to measure final QA quality. The IK classifier is trained to predict SOLAR's 'Yes/No', and the policy is then scored by the same SOLAR. If SOLAR is lenient or biased, the classifier may simply be learning to match judge preferences, and the reported retrieval savings could be an artifact. The teacher ablation helps a little—Llama-70B as teacher gives similar QA results—but the evaluation metric is still SOLAR in those runs, so the circularity isn't broken. The 'best point' thresholds are also selected on the evaluation set, which inflates the apparent benefit.\n\nThat said, the central empirical findings are plausibly robust. The token-length plateau at 32 and the 20K training-set saturation are the kind of results that survive a change of metric. The paper just needs an independent QA metric (exact match, F1, or a different judge) and a fixed threshold policy to make the efficiency claim credible.\n\nWho's this for? Engineers working on retrieval pipelines who want a lightweight decision rule; they'll get practical guidance and a few implementation details. It deserves a serious referee, though my recommendation would be major revision rather than acceptance as is.","headline":"Useful empirical mapping of when-to-skip retrieval, but the shared judge for labels and evaluation makes the headline retrieval-reduction claim unproven.","tokens_in":11092,"tokens_out":2792,"would_cite":true,"duration_ms":25834,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training an LLM to output an 'I Know' score—distilled from an LLM-as-judge and fed the first 32 tokens of the model's own answer—lets a RAG pipeline skip retrieval on more than half of queries on some datasets while matching or beating…","keywords":["I Know score","LLM-as-a-judge","retrieval-augmented generation","when to retrieve","parametric memory","distillation","question answering","confidence estimation"],"falsifier":"Train the IK classifier with SOLAR as judge, then score the final answers on the NQ test set with a different, stricter judge (or human raters); if the retrieval-skipping threshold that worked under SOLAR now lowers answer quality, the classifier was optimizing for judge leniency rather than for genuine parametric knowledge. Additionally, test on recent-fact questions outside the model's knowledge cutoff; if the classifier marks many as 'I know', it is not actually detecting stored knowledge.","tokens_in":10034,"feed_emoji":"🧠","tokens_out":12375,"duration_ms":95091,"temperature":0.7,"pith_summary":"This paper asks whether a large language model can be trained to know what it knows: specifically, whether it can answer a question correctly without retrieving external documents. The author's answer is yes—an LLM fine-tuned to predict an 'I Know' (IK) score, with labels produced by an LLM-as-judge, reaches around 80% accuracy on the detection task. When the first 32 tokens of the model's own answer are fed in, the classifier becomes accurate enough that a retrieval-augmented generation (RAG) pipeline can skip retrieval for more than half of queries on some datasets while keeping answer quality equal or slightly better. The claim matters because RAG's main costs—retrieval, reranking, and long prompts—are only worth paying when the parametric memory actually needs the help.","feed_headline":"32 tokens let an LLM skip retrieval on half of queries","feed_subtitle":"A distilled 'I know' score predicts when the model already knows the answer, cutting retrieval cost with no quality loss.","key_machinery":"The IK (I Know) score is the central object: after fine-tuning on judge-generated Yes/No labels, the score is the softmax probability of the 'Yes' token taken over only the Yes and No logits. The second mechanism is the input format—the question alone versus the question plus the first $k$ tokens of the model's own generated answer. The $k=32$ prefix is the operational sweet spot: it lifts IK accuracy by 4–5 points over question-only input and cuts the required training data to about 20K samples. The score does the work by acting as a gate: when it exceeds a chosen threshold, the retriever and reranker are skipped, so the claim stands or falls on whether this gate reliably separates queries the model can answer from those it cannot.","core_discovery":"The paper's central discovery is that the 'when to retrieve' decision for a RAG system can be distilled into a small classifier: take the LLM, fine-tune it to output a binary 'I Know' label using silver labels produced by an LLM-as-a-judge, and feed it the first 32 tokens of the model's own generated answer. This yields an IK classifier with 80–83% accuracy and AUC up to 0.92 on KILT NQ. With a suitable threshold, the classifier lets a RAG pipeline skip retrieval on 29–50% of queries (depending on model and threshold) while achieving LLMEval scores equal to or up to 2 points above the full-RAG baseline. The author also finds the IK score can characterize datasets—TriviaQA and SCIQ are mostly answered from parametric memory, PopQA and HotpotQA mostly require retrieval, and NQ/ASQA are mixed—and that roughly 20,000 training samples suffice when the 32-token prefix is included.","pith_inferences":["The evaluation pipeline uses the same model (SOLAR) as both the label generator and the final answer-scorer, so a natural next step would be to re-run the experiments with an independent, stricter judge for the final LLMEval score; if the retrieval savings shrink, the IK classifier was partly matching the judge's taste rather than detecting knowledge.","The 32-token prefix finding suggests a streaming policy: generate a short prefix, compute IK, then decide whether to continue without retrieval or to restart with retrieved documents; this would remove the extra pass and could be tested directly.","The IK score's dataset-characterization behavior could be used to pre-screen corpora or query streams for whether RAG is worth deploying at all, or to set per-query retrieval budgets instead of a single threshold.","Because the best threshold varies by dataset (0.5–0.8), a practical extension would be a calibration procedure that maps the IK score to a desired retrieval rate, rather than fixing one threshold."],"forward_implications":["For datasets where the model is largely confident (e.g., TriviaQA, SCIQ), the IK classifier can cut retrieval to 7–27% of queries with no quality loss.","Adding the first 32 generated answer tokens improves IK accuracy by 4–5 points over question-only input, and reduces the required training data from roughly 40K to 20K samples.","The distillation works across model families (Llama, Mistral, Gemma, SOLAR), but only when the teacher is an LLM judge; string-based teachers like recall or substring match produce classifiers too weak to be useful.","The IK score also serves as a dataset characterization tool, revealing a U-shaped distribution for NQ and ASQA, high confidence for TriviaQA/SCIQ, and low confidence for PopQA/HotpotQA.","Skipping RAG based on IK reduces per-query latency substantially (up to 80% of RAG processing time, ignoring retrieval/reranking), since the classifier itself adds only a few milliseconds."],"supporting_citations":[{"why":"Establishes the teacher-distillation paradigm for the IK task and is cited as the first work training a model to decide when to retrieve.","marker":"[1]"},{"why":"Supplies the evaluation framework used for all QA results, including the LLMEval metric, the retriever, the reranker, and the judge.","marker":"[16]"},{"why":"Provides the NQ dataset used for training the IK classifier and for the main evaluation.","marker":"[17]"},{"why":"Prior work that showed including generated answer tokens helps a judgment model; the paper extends this to its IK classifier.","marker":"[20]"},{"why":"The fine-tuning-with-adapters approach the author follows for training the IK classifier.","marker":"[10]"},{"why":"Intermediate-layer probing baseline that the paper tests and finds underperforms its full fine-tuning setup.","marker":"[2]"}],"fun_headline_variants":["A 32-token peek tells if an LLM already knows the answer","LLM 'I know' classifier cuts retrieval in half","32 tokens predict when to skip retrieval in RAG","Teaching LLMs to say 'I know' halves RAG retrieval load","I Know score: 32 tokens decide if retrieval is needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the assumption that the same judge model, SOLAR, can reliably label both whether the LLM knows an answer and whether a final answer is good; if that judge is lenient or biased, the classifier may be learning to match the judge's taste instead of detecting genuine knowledge.","fun_headline_variants_meta":{"raw":{"variants":["A 32-token peek tells if an LLM already knows the answer","LLM 'I know' classifier cuts retrieval in half","32 tokens predict when to skip retrieval in RAG","Teaching LLMs to say 'I know' halves RAG retrieval load","I Know score: 32 tokens decide if retrieval is needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3435,"prompt_tokens":948,"completion_tokens":2487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":2399}},"tokens_in":564,"tokens_out":2487,"duration_ms":15718,"temperature":1.0,"reasoning_tokens":2399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:49:28.441798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the IK classifier with SOLAR as judge, then score the final answers on the NQ test set with a different, stricter judge (or human raters); if the retrieval-skipping threshold that worked under SOLAR now lowers answer quality, the classifier was optimizing for judge leniency rather than for genuine parametric knowledge. Additionally, test on recent-fact questions outside the model's knowledge cutoff; if the classifier marks many as 'I know', it is not actually detecting stored knowledge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evaluation framework used for all QA results, including the LLMEval metric, the retriever, the reranker, and the judge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NQ dataset used for training the IK classifier and for the main evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work that showed including generated answer tokens helps a judgment model; the paper extends this to its IK classifier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The fine-tuning-with-adapters approach the author follows for training the IK classifier."}],"review_version":1}