{"id":"be823edd-0301-43c9-b51e-81d9714b7b31","arxiv_id":"2506.22852","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On MobileCS2, a GPT-2 finetuned with retrieved knowledge outperforms a prompted GPT-3.5 in both RAG and agent dialog systems, though the comparison is confounded.","lead":"This paper tests whether finetuning an LLM on customer-service dialogs together with retrieved knowledge beats prompting the LLM with the same knowledge, in both retrieval-augmented and agent-based systems. It reports gains on the MobileCS2 benchmark, but the comparison uses different base models, which weakens the headline claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KAFT vs prompting comparison conflates base model and finetuning status; no matched-base control.","rationale":"The reader's weakest_assumption correctly identifies the confound: the comparison conflates method with base model and finetuning status. This is the single load-bearing issue because the paper's headline and conclusion depend entirely on the contrast between KAFT(GPT-2) and prompting(GPT-3.5). The paper provides no evidence that the advantage would survive a matched-model comparison, and the discussion in Section IV-B explicitly frames the result as a small-model-beats-large-model claim, which changes the nature of the contribution. The ablations (Tables IV and V) are useful for the RAG pipeline and the dataset is a real contribution, but they do not address the confound. A concrete paired comparison would settle the question. Since the reader's verdict of REJECT is based on the same concern and remains appropriate given the missing control, the verdict is unchanged.","tokens_in":9957,"tokens_out":2097,"duration_ms":22449,"concrete_test":"Run the prompting baseline using GPT-2 with the same 5-shot prompts, retrieval/API pipeline, and evaluation protocol on MobileCS2. If GPT-2 prompting attains an Inform Rate near KAFT's 0.145/0.147, the claimed advantage is attributable to finetuning status rather than the KAFT construction. As an additional arm, apply KAFT to a finetunable GPT-3.5-class model and compare against prompted GPT-3.5; if the gap persists in both arms, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'KAFT substantially surpasses prompting' rests on a confounded comparison. Section IV-A states: 'For the KAFT method, we finetune the GPT2, while for the prompting method, we use the GPT3.5.' Thus every KAFT result is a finetuned small model and every prompting result is a larger, unfinetuned generalist model. The observed gap in Inform Rate (0.145/0.147 vs 0.063/0.082) could be caused by the choice of base model, the finetuning status, or the interaction between them, not by the KAFT method as such. The paper itself frames the result as 'a small model like GPT2 with KAFT can beat GPT3.5 with prompting' in Section IV-B, which is a model-and-method comparison, not a clean test of KAFT versus prompting. No experiments with a prompted GPT-2, a KAFT-tuned GPT-3.5, or any matched model are reported. Without such controls, the central claim is not supported as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes knowledge augmented finetuning (KAFT): instead of prompting a large language model with retrieved knowledge in RAG- or agent-based dialog systems, the authors finetune the generator (GPT-2) on domain-specific dialogs together with the knowledge pieces retrieved by the system's retriever/APIs. Experiments on the MobileCS2 customer-service dataset compare KAFT (finetuned GPT-2) against prompting (GPT-3.5 with 0- or 5-shot in-context examples) in both a RAG pipeline and an agent pipeline, with additional ablations on using retrieved vs. annotated knowledge in training and testing. The paper claims that KAFT substantially surpasses prompting, especially in factual accuracy measured by Inform Rate, and that this is the first solid empirical work on the KAFT idea.","tokens_in":10139,"tokens_out":2969,"duration_ms":33622,"significance":"If the central claim were cleanly established, the paper would provide a useful practical result: domain-specific finetuning with retrieved knowledge can improve factual accuracy of dialog systems. The study has some real strengths: it uses a realistic human-human customer-service dataset, follows the official evaluation scripts, and includes thoughtful ablations (retrieved vs. oracle knowledge in training and testing, RAG vs. agent retrieval quality). The ablations showing that training with retrieved knowledge improves test-time robustness are informative. However, as reported, the headline comparison is confounded and therefore cannot support the claim that KAFT as a method outperforms prompting. The paper also overstates its novelty given the cited work on agent finetuning and retriever-augmented finetuning.","major_comments":[{"comment":"The central comparison 'KAFT substantially surpasses prompting' is confounded. The paper states 'For the KAFT method, we finetune the GPT2, while for the prompting method, we use the GPT3.5.' Thus every KAFT result is a finetuned GPT-2 and every prompting result is an unfinetuned GPT-3.5. The observed gaps (e.g., Inform 0.145 vs. 0.063 in RAG, 0.147 vs. 0.082 in agent) could be due to the base model, the finetuning status, or their interaction, rather than to KAFT. No matched control is reported: there is no prompted GPT-2, no KAFT-finetuned (or LoRA-tuned) GPT-3.5, and no same-parameter-count prompt-based system. Without such controls, the abstract's and Section IV-B's claim that KAFT beats prompting is not supported as a statement about the method.","section":"§IV-A and Table I"},{"comment":"The results are reported as single point estimates with no variance, significance testing, or information about the number of runs and seeds. Table I reports one BLEU, one BERTScore, one Inform, and one Score per setting. For the main substantive claim about factual accuracy, the difference in Inform Rate should be accompanied by a significance test or at least confidence intervals. This is especially important because the prompting baselines use in-context examples selected 'randomly from the dataset' (Section III-B.2), so their variance may be substantial.","section":"§IV-A and §IV-B"},{"comment":"The claim that 'this paper represents the first solid empirical work to investigate the KAFT idea' is inconsistent with the paper's own citations. FireAct [11] investigates finetuning language agents for tool use, and RAFT [26] explicitly adapts language models to domain-specific RAG by finetuning on retrieved knowledge with distractors; the latter is described in Section III-B.1 as being similar to the paper's approach of using retrieved rather than oracle knowledge in training. The novelty claim should be substantially revised, and the relation to RAFT and FireAct should be discussed explicitly rather than asserted to leave a gap.","section":"§I and §II (related work)"},{"comment":"The sentence 'It is found in our experiments that a small model like GPT2 with KAFT can beat GPT3.5 with prompting' is a model-vs-method comparison, not a method-vs-method comparison. It demonstrates at most that finetuning a small model on a specific dataset can outperform a general-purpose API model in this task, which is not the same as showing that KAFT is superior to prompting for a fixed model. Section IV-A's statement that 'Using GPT-2 suffices to investigate the research question' is an assumption, not a demonstrated fact, and it is in tension with the stated goal of systematically comparing prompting and KAFT.","section":"§IV-B"}],"minor_comments":[{"comment":"Typo: 'Futuremore' should be 'Furthermore'.","section":"§I"},{"comment":"'close-book QA' should be 'closed-book QA'.","section":"§I"},{"comment":"The text reads 'We use the ICL) method'; the opening parenthesis is missing.","section":"§III-C.2"},{"comment":"The paper refers to 'the SLT 2024 FutureDial Challenge [17]' and 'MobileCS2 dataset, released from the SLT 2024 FutureDial Challenge', but reference [17] is titled 'The 2nd FutureDial challenge: Dialog systems with retrieval augmented generation (FutureDial-RAG)'. The terminology should be made consistent.","section":"§IV-A and references"},{"comment":"The combined score formula 'score = 0.5*(BLEU/100 + BERTScore) + Inform' is unusual because BERTScore and Inform can be on different scales; a brief explanation of why this formula is used would help readers interpret Table I.","section":"§IV-A"},{"comment":"Table IV shows that using oracle knowledge in testing yields a combined score of 0.992, leaving almost no room for variation; this near-ceiling effect should be noted when interpreting the retrieval-error analysis in the surrounding text.","section":"§IV-C, Table IV"}],"recommendation":"reject","confidential_remarks":"The confound in the main comparison is load-bearing and cannot be repaired by local edits; the authors would need to add matched-model baselines (e.g., prompted GPT-2, KAFT-tuned GPT-3.5 or another API model) and ideally variance estimates before the central claim can be evaluated. The paper also overclaims novelty in light of its own related-work citations. That said, the dataset and the retrieved-vs-oracle training ablation are useful; if the authors resubmit with matched controls and a scoped contribution claim, the work could become publishable. Given the current form, I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is Table V. Within the same finetuned GPT-2 model, training on retrieved knowledge instead of oracle knowledge improves test Inform Rate (0.145 vs 0.127) and BLEU (22.2 vs 14.1). That is a clean within-model control, and it supports the narrow claim that knowledge-augmented finetuning helps the model cope with noisy retrieval. The headline claim — 'KAFT substantially surpasses prompting' — is not supported as stated, because every KAFT condition uses a finetuned GPT-2 and every prompting condition uses a prompted GPT-3.5. The gap could come from the base model, the finetuning status, or the method; you cannot tell. The paper's own comment that 'a small model like GPT2 with KAFT can beat GPT3.5 with prompting' is honest, but it is a model-and-method comparison, not a test of KAFT versus prompting.\n\nWhat is new is modest but real. MobileCS2 is a genuine customer-service corpus, and the paper builds both RAG and agent versions, reporting decision-maker accuracy separately. The observation that agent search APIs retrieve better than the RAG retriever while decision accuracy is low, netting out to comparable end-to-end performance, is fair. The retriever and decision numbers are worth having.\n\nSoft spots, in order: the GPT-2 vs GPT-3.5 confound; no variance or significance reporting, so the Inform Rate differences are single numbers; and the 'first solid empirical work' claim is contradicted by the paper's own references to RAFT and FireAct, which finetune with retrieved knowledge. The dataset comes from overlapping authors, which is minor.\n\nDespite the flawed headline, the paper is worth engaging. A referee should demand matched-base controls: a prompted GPT-2, a finetuned GPT-3.5 (or a same-size model), and repeated runs. With those, the claim would either hold or collapse. As it stands, it is a revise-and-resubmit candidate, not a reject. I would cite the retrieval and decision-maker results, and the Table V ablation, but not the headline comparison.\n\nRegards,","headline":"KAFT paper's headline comparison is confounded (GPT-2 finetuned vs GPT-3.5 prompted), but the Table V within-model ablation is a genuine and useful result.","tokens_in":10669,"tokens_out":3362,"would_cite":true,"duration_ms":35211,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Finetuning a dialog model on retrieved knowledge beats prompting for factual accuracy in both RAG and agent systems.","keywords":["knowledge augmented finetuning","retrieval augmented generation","LLM agents","dialog systems","factual accuracy","hallucination","MobileCS2","customer service dialog"],"falsifier":"A controlled run on MobileCS2 that holds the base model fixed, for example prompting GPT-2 with the same five examples and instructions or finetuning GPT-3.5 with KAFT, would settle the claim: if the Inform Rate gap largely disappears or reverses, the advantage is due to the base model rather than the finetuning method.","tokens_in":9734,"feed_emoji":"💬","tokens_out":5312,"duration_ms":50376,"temperature":0.7,"pith_summary":"The paper argues that a language model serving in a retrieval-augmented (RAG) or tool-calling (agent) dialog system should be finetuned on domain dialogs together with the retrieved knowledge, rather than merely prompted with that knowledge. It calls this knowledge augmented finetuning (KAFT) and tests it on MobileCS2, a real customer-service dataset. On both architectures, KAFT with a small GPT-2 model beats GPT-3.5 prompting on fluency, semantic similarity, and especially Inform Rate, a measure of whether the response supplies the information the user asked for. If the result is right, it means the largest gains in knowledge-intensive dialog come from teaching the model to use external knowledge during training, not from scaling up prompts at inference time.","feed_headline":"Finetuned GPT-2 beats prompted GPT-3.5 on dialog factual accuracy","feed_subtitle":"Finetuning a small GPT-2 on retrieved knowledge doubles factual accuracy over prompting GPT-3.5 in both setups.","key_machinery":"The load-bearing mechanism is knowledge augmented finetuning itself: supervised finetuning of the generator (and, in the agent setting, the decision maker) on domain data where each training example includes the knowledge the system would actually retrieve at test time. For RAG, the generator is trained with the auto-regressive loss over $p_\\theta(r_t \\mid c_t, h_t)$ using retriever output $h_t$; for the agent, the same loss is applied with API search results, and a second finetuned module predicts the search decision $a_t$. The key design choice is that training uses retrieved knowledge instead of the clean annotated knowledge, which aligns training with test-time conditions and teaches the model to cope with imperfect retrieval.","core_discovery":"The central claim is that training the response generator on pairs of dialog context and retrieved knowledge pieces, which the paper calls knowledge augmented finetuning (KAFT), substantially improves knowledge-intensive dialog over prompting, with the largest effect on factual accuracy. In the RAG system, KAFT finetunes the generation model $p_\\theta(r_t \\mid c_t, h_t)$ on retrieved (rather than oracle) knowledge; in the agent system, it additionally finetunes the decision maker $p_\\theta(a_t \\mid c_t)$ that chooses which API to call. On the MobileCS2 test set, KAFT with GPT-2 reaches an Inform Rate of 0.145 in RAG and 0.147 in the agent system, versus 0.063 and 0.082 for GPT-3.5 prompting, and also raises BLEU and BERTScore. The paper also finds that training with retrieved knowledge outperforms training with annotated knowledge when the system is tested with retrieved knowledge, because the generator learns to judge whether retrieved pieces are correct.","pith_inferences":["Editorial inference: a natural extension, implied but not tested by the paper, is that the same KAFT recipe should transfer to larger open-weight models and other vertical domains, since the mechanism is a training-data construction choice rather than a GPT-2-specific trick.","Editorial inference: the large drop when the retriever's recall@1 is low for Product search suggests that improving retrieval accuracy would amplify KAFT's gains more than further prompting would; this is a bottleneck the paper identifies but does not quantify end-to-end.","Editorial inference: because the paper measures only one dataset and one evaluation script, the strongest test would be to re-run KAFT on a second knowledge-intensive dialog dataset with the same base model for prompting and finetuning."],"forward_implications":["A dialog system can reach higher factual accuracy by finetuning a relatively small generator on retrieved knowledge than by prompting a much larger model, implying that training data construction matters more than model scale for this task.","Agent systems benefit from finetuning not only the response generator but also the decision maker that chooses which knowledge API to call: KAFT raises decision accuracy on Personal, Product, and FAQ searches over both 0-shot and 5-shot prompting.","Training a RAG generator on the retriever's actual noisy output improves test-time performance, so retrieval noise should be simulated during finetuning rather than hidden behind oracle knowledge.","Because RAG and agent systems perform on par despite the agent's better retrieval, improving the agent's low decision-making accuracy is a concrete route to further gains."],"supporting_citations":[{"why":"Supplies the MobileCS2 dataset, the official evaluation scripts, and the baseline system design for the RAG dialog setting.","marker":"[17]"},{"why":"Provides the RAG formulation and the retrieval distribution $p_\\eta(z_i \\mid c_t)$ that the retriever and its training objective build on.","marker":"[4]"},{"why":"Is the GPT-3.5 model used as the prompting baseline in all experiments.","marker":"[20]"},{"why":"Is the GPT-2 model that KAFT finetunes for both the RAG and agent systems.","marker":"[30]"},{"why":"Supplies the BERT-based pretrained initialization of the knowledge-piece and context encoders in the dual-encoder retriever.","marker":"[29]"},{"why":"Motivates the noise-adding adaptation of training on retrieved rather than oracle knowledge.","marker":"[26]"},{"why":"Defines the in-context learning setup used for the prompting baselines.","marker":"[18]"}],"fun_headline_variants":["Finetuned GPT-2 outperforms GPT-3.5 prompting on dialog factual accuracy","Knowledge-augmented finetuning beats prompting in RAG and agent dialogs","KAFT: training on retrieved knowledge tops prompting for factual dialogs","Small finetuned model trumps large prompted model in knowledge-intensive chat","First empirical study: finetuning with domain knowledge beats prompting in RAG and agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the performance gap comes from KAFT and not from the fact that KAFT uses a finetuned GPT-2 while prompting uses GPT-3.5, since the two methods are never tested with the same base model.","fun_headline_variants_meta":{"raw":{"variants":["Finetuned GPT-2 outperforms GPT-3.5 prompting on dialog factual accuracy","Knowledge-augmented finetuning beats prompting in RAG and agent dialogs","KAFT: training on retrieved knowledge tops prompting for factual dialogs","Small finetuned model trumps large prompted model in knowledge-intensive chat","First empirical study: finetuning with domain knowledge beats prompting in RAG and agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001138,"raw_usage":{"total_tokens":4751,"prompt_tokens":998,"completion_tokens":3753,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":3648}},"tokens_in":614,"tokens_out":3753,"duration_ms":27423,"temperature":1.0,"reasoning_tokens":3648,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:56:37.799485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled run on MobileCS2 that holds the base model fixed, for example prompting GPT-2 with the same five examples and instructions or finetuning GPT-3.5 with KAFT, would settle the claim: if the Inform Rate gap largely disappears or reverses, the advantage is due to the base model rather than the finetuning method.","supporting_citations":[{"cited_title":"The 2nd FutureDial challenge: Dialog systems with retrieval aug- mented generation (FutureDial-RAG),","cited_arxiv_id":null,"evidence_quote":"Supplies the MobileCS2 dataset, the official evaluation scripts, and the baseline system design for the RAG dialog setting."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive NLP tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the RAG formulation and the retrieval distribution $p_\\eta(z_i \\mid c_t)$ that the retriever and its training objective build on."},{"cited_title":"Training language models to follow instructions with human feedback,","cited_arxiv_id":null,"evidence_quote":"Is the GPT-3.5 model used as the prompting baseline in all experiments."},{"cited_title":"Language models are unsupervised multitask learners,","cited_arxiv_id":null,"evidence_quote":"Is the GPT-2 model that KAFT finetunes for both the RAG and agent systems."},{"cited_title":"BERT: Pre-training of deep bidirectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT-based pretrained initialization of the knowledge-piece and context encoders in the dual-encoder retriever."},{"cited_title":"Language models are few-shot learners,","cited_arxiv_id":null,"evidence_quote":"Defines the in-context learning setup used for the prompting baselines."}],"review_version":1}