{"id":"b9c38611-1447-43eb-a46b-0431b960c87b","arxiv_id":"2412.03096","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A tool-calling framework that lets an LLM decide when to consult a commonsense knowledge base produces modestly better empathetic responses than fixed knowledge infusion on the EmpatheticDialogues benchmark.","lead":"This paper tests whether a chatbot replies more empathetically when it can choose to call a commonsense knowledge tool during a conversation, instead of adding that knowledge every turn. The authors built a new training set by inserting tool-use traces into an existing empathy dialogue dataset, fine-tuned two LLMs on it, and report small but consistent improvements on benchmark metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Golden-response lookahead in both Annotator and Reflector makes the reported gains over ED-only fine-tuning uninterpretable as evidence for tool calling.","rationale":"The reader's weakest_assumption identifies the Reflector's golden-response filtering as the key risk. My review agrees and broadens it: the Annotator prompt also includes the golden response, so even the 'noref' ablation retains lookahead at the tool-call decision level. This makes the concern more central because it threatens the causal interpretation of the main result, not just the dataset's cleanliness. The proposed test removes both sources of reference-conditioning while keeping all other training choices fixed, which would settle whether the benefit is due to tool use itself or to training on reference-filtered tool calls. I do not see grounds to reject the paper: the framework is plausible, the code is released, the plug-and-play swap between COMET and CICERO is a useful check, and the human and LLM evaluations provide some independent support. But the current experiments do not separate 'tool use helps' from 'training on reference-filtered tool calls helps.' That missing control is exactly what a conditional acceptance should require. Therefore I recommend keeping the reader's CONDITIONAL verdict unchanged; if the proposed test later fails, the verdict should move toward UNVERDICTED.","tokens_in":17217,"tokens_out":6016,"duration_ms":60084,"concrete_test":"Construct a lookahead-free version of TOOL-ED: mask the 'assistant_response' field in both the Annotator and Reflector prompts, so tool-call decisions and relevance filtering use only dialogue context, then fine-tune Qwen1.5-14B and Vicuna-7B with identical LoRA hyperparameters and re-run the Table 1/4 automatic metrics and the Table 2 human A/B test on the same 8:1:1 split. If the margin over qwen_lora/vicuna_lora falls within run-to-run noise or the human win rate drops below roughly 60%, the golden-response lookahead is the load-bearing component; if the gains persist unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison (qwen_tool_comet/vicuna_tool_comet vs qwen_lora/vicuna_lora, Tables 1-4) is meant to show that selective tool calling improves empathetic response generation. The load-bearing assumption is that TOOL-ED's tool-call supervision is representative of inference-time tool use. That assumption is insecure: Section 3.3 and Appendix A.1 show that the Annotator prompt includes the golden assistant response ('your response: {assistant_response}') and asks whether the tool would be used to produce that response, while the Reflector keeps only COMET outputs judged relevant to the same golden response. Thus every retained training example pairs a tool call with a reference that a 70B model judged useful for that exact reference. At inference there is no reference, and tool outputs are unfiltered; the model has not been trained on observations that are irrelevant or weakly relevant to the eventual response. The 'noref' ablation (Section 4.5.2) removes only the Reflector, not the Annotator's access to the golden response, so it cannot separate genuine tool-use benefit from reference-conditioned selection. The automatic improvements are also modest and inconsistent (e.g., Table 4: vicuna_noref_comet has BLEU-1 0.1943 vs vicuna_tool_comet 0.1859), so the claimed effect is not clearly robust once the lookahead is accounted for. If the gains are driven by the golden-response-filtered training distribution, the framework's central claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Emotional Knowledge Tool Calling (EKTC), a framework for empathetic response generation that wraps commonsense knowledge bases (COMET and CICERO) as callable tools and fine-tunes LLMs (Vicuna-7B, Qwen1.5-14B) with LoRA on a newly constructed dataset TOOL-ED. TOOL-ED is created from EmpatheticDialogues by using LLaMA3-70B as an Annotator to decide whether to call the tool and as a Reflector to filter tool outputs based on their relevance to the golden assistant response. The authors report automatic metrics (BLEU, BERTScore, ROUGE, Distinct) and human/LLM-based A/B evaluations showing improvements over models fine-tuned on the original ED dataset, and they run ablations including a 'noref' variant without the reflection step.","tokens_in":17546,"tokens_out":4580,"duration_ms":42195,"significance":"If the central claim holds—that selectively invoking an external commonsense knowledge base via tool calling improves empathetic response quality over always injecting knowledge or never using it—this would be a useful contribution to empathetic dialogue and tool-augmented LLMs. The paper provides a reproducible code repository, a plug-and-play tool design that works with two knowledge bases, and a new dataset. However, the experimental evidence for the central claim is weakened by a training-data confound and a lack of statistical rigor, as detailed in the major comments.","major_comments":[{"comment":"The training data construction uses the golden response in both the Annotator prompt ('your response: {assistant_response}' and asking whether the tool would be used to produce that response) and the Reflector (keeping only tool outputs judged relevant to that same golden response). This introduces a reference-conditioned selection bias: the model is trained on tool calls that are specifically beneficial for the ground-truth response, but at inference no reference is available and tool outputs are unfiltered. As a result, the reported gains of qwen_tool_comet and vicuna_tool_comet over qwen_lora and vicuna_lora (Tables 1, 2, 3) may stem from the curated training distribution rather than from the model's learned ability to call tools selectively. The 'noref' ablation in §4.5.2 removes only the Reflector, not the Annotator's access to the golden response, so it does not control for this confound. To support the central claim, the authors should either construct training data without reference lookahead (e.g., an Annotator that sees only dialogue context) or provide evidence that the inference-time tool-calling policy generalizes beyond the reference-filtered training distribution.","section":"§3.3, Appendix A.1"},{"comment":"No variance, confidence intervals, or statistical significance tests are reported for any automatic metric. Many differences are small in absolute terms (e.g., BERTScore 0.8765 vs. 0.8760 for qwen models; ROUGE-L 0.1793 vs. 0.1696), so without repeated runs or paired significance tests the observed improvements cannot be distinguished from noise. The human evaluation (§4.4) uses only 100 randomly selected conversations and three annotators, with no inter-annotator agreement reported, and the GPT-4-based evaluation gives no sample size or agreement analysis. Quantitative claims of improvement should be accompanied by significance levels or at least standard deviations across multiple runs.","section":"Tables 1-4"},{"comment":"The paper re-splits the ED dataset into training, validation, and test sets with a random 8:1:1 ratio. The original EmpatheticDialogues dataset has a predetermined split; using a different split makes the comparisons with prior published numeric results (e.g., CEM, KEMP, MIME in Table 1) invalid, because those baselines were evaluated on the original test partition. The authors should use the standard split or explicitly state that all baselines were re-evaluated on the same re-split; otherwise the claimed improvements over SOTA are not established.","section":"§4.3"}],"minor_comments":[{"comment":"In Table 4, qwen_noref_comet reports BLEU-4 = 0.3130 and qwen_noref_cicero reports 0.3190; these are likely typos for 0.0313 and 0.0319, since other BLEU-4 values are around 0.03. Please correct them.","section":"Table 4"},{"comment":"Table 1 reports qwen_oneshot BLEU-3 as 0.1780, which is likely a typo (should be 0.0178). Also, the Dist-1/Dist-2 values for qwen_base (0.2322/0.1720) are implausible as Dist-1 is usually lower than or comparable to Dist-2; please verify these numbers.","section":"Table 1"},{"comment":"In the human evaluation description, 'Informativity (Inf.) meatures' should be 'measures'. Additionally, the paper should state whether the 100 samples are drawn from the test set and whether the three annotators were blinded to the model identities.","section":"§4.4"},{"comment":"The abstract contains a formatting error: 'EMPATHETICMPATHETIC DIALOGUE' should be 'EMPATHETIC DIALOGUE'.","section":"Abstract"},{"comment":"The paper reports a 26.46% tool-call insertion rate but does not specify the resulting dataset size or the filtering rate of the Reflector. This information is useful for assessing the dataset quality and the potential selection bias.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading. First, this is the first paper I know of that packages commonsense knowledge bases (COMET, CICERO) as callable tools for empathetic response generation, and it ships a concrete artifact, the TOOL-ED dataset. Second, the central comparison in the paper is contaminated by golden-response lookahead. The stress-test note is right about this.\n\nThe contamination is not subtle. Section 3.3 and Appendix A.1 show that the Annotator prompt includes the assistant_response and asks whether the tool would be used to produce that response. The Reflector prompt does the same thing and keeps only COMET outputs judged relevant to that same golden response. Every retained training example therefore pairs a tool call with an observation that a 70B model has already certified useful for the exact reference. At inference there is no reference, and the observations the model gets are unfiltered. The model has never been trained on weak or irrelevant tool outputs, which is exactly what it will encounter. The noref ablation in Section 4.5.2 removes only the Reflector; the Annotator still sees the golden response, so it does not separate genuine tool-use ability from reference-conditioned selection. This does not prove the method is worthless, but it means the reported gains over ED-only fine-tuning do not establish that selective tool calling is what helps.\n\nThe automatic gains are also modest and uneven. vicuna_tool_comet has BLEU-1 0.1859 versus vicuna_lora 0.1935, so the tool version is actually worse on that metric; qwen_tool_comet improves BLEU-1 by about 0.7 points. There are no significance tests or error bars anywhere. Human evaluation uses 100 samples and three annotators, with no inter-annotator agreement reported. The fine-tuned models call the tool 82% (Qwen) and 65% (Vicuna) of the time, which sits awkwardly with the selective-invocation story. The paper also re-splits ED at 8:1:1 rather than the canonical split, so comparisons with published numbers need care.\n\nNone of this is a reason to ignore the paper. The framing is genuinely new, the COMET/CICERO plug-and-play check is a nice sanity test, the limitation statement about single-tool use is honest, and the dataset construction is documented well enough that the lookahead is visible, which is itself a sign of careful reporting. The related work is fine.\n\nWho gets value: people working on empathetic dialogue or tool-augmented generation should know this paper, mainly as a reminder that dataset construction can silently encode the answer. It deserves a serious referee: the idea is useful, the flaw is diagnosable and fixable, and major revision could turn it into a solid negative or positive result. I would not desk-reject it, but I would not accept the current evidence as establishing the central claim.","headline":"First tool-calling framing for empathetic knowledge infusion, with a useful new dataset, but the headline gains are undermined by golden-response lookahead in dataset construction.","tokens_in":18023,"tokens_out":4612,"would_cite":false,"duration_ms":45222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an LLM to selectively call a commonsense knowledge tool improves empathetic responses over always injecting the knowledge.","keywords":["empathetic response generation","tool learning","commonsense knowledge","COMET","TOOL-ED dataset","LLM fine-tuning","LoRA","dialogue generation"],"falsifier":"Force the fine-tuned tool-calling model at inference to always answer directly, or replace the tool observation with random text, and measure the same empathy metrics; if performance does not drop below the tool-calling version, the observed improvement is not caused by the tool itself.","tokens_in":17034,"feed_emoji":"💬","tokens_out":4432,"duration_ms":38969,"temperature":0.7,"pith_summary":"The paper argues that letting an LLM decide when to consult an external commonsense knowledge base, rather than always feeding it knowledge, yields better empathetic responses. It constructs TOOL-ED by adding tool-use traces to the Empathetic Dialogues dataset, using LLaMA3 as both annotator (when to call) and reflector (filtering calls whose results align with the golden response). Fine-tuning Vicuna-7B and Qwen1.5-14B on TOOL-ED with COMET or CICERO as the tool improves BLEU, ROUGE, BERTScore, and human/LLM judgments of empathy over the same models fine-tuned on the original ED dataset. The central comparison against an always-inject-knowledge ablation supports the claim that selective access reduces noise.","feed_headline":"Selective tool calls make LLM replies more empathetic","feed_subtitle":"Fine-tuning on TOOL-ED, where models call COMET only when needed, beats always injecting knowledge.","key_machinery":"The central mechanism is the EKTC tool-calling loop: the model outputs an \"Action: EmotionKnowledgeBase\" with an \"Action Input\" containing the dialogue context, receives five commonsense relations (xIntent, xNeed, xWant, xEffect, xReact from COMET, or the equivalent relations from CICERO) as an observation, and then generates the response. The dataset construction uses LLaMA3-70B as an Annotator to decide whether a tool call is appropriate and as a Reflector to filter calls whose results are judged highly relevant to the golden response, yielding a training set where only 26.46% of instances contain tool calls. The ReAct-style (action, observation) format carries the mechanism, and LoRA fine-tuning adapts the base models to the new task.","core_discovery":"The paper establishes that a dialogue model can be trained end-to-end to invoke a commonsense knowledge base as a tool, and that this active, selective invocation outperforms both no-knowledge baselines and unconditioned knowledge injection. The evidence is that Qwen1.5-14B and Vicuna-7B fine-tuned on TOOL-ED with COMET or CICERO as the tool score higher across BLEU-1/2/3/4, BERTScore, ROUGE-1/2/L, and in A/B human and GPT-4 evaluations against the same models fine-tuned on the original ED dataset. The framework, called EKTC, thus demonstrates that the timing of knowledge access is itself a learnable skill that improves response quality.","pith_inferences":["The approach could extend beyond knowledge bases to other auxiliary modules such as sentiment classifiers, persona retrievers, or safety filters, letting the model decide when to consult each one.","Because the Reflector filters training calls using the golden response, the learned policy at inference may be more conservative than optimal; a reward model or test-time filtering could close that gap.","The lower tool-calling ratio observed in the fine-tuned models suggests they learn to reserve tool calls for emotionally intense contexts, which might be even better exploited with an adjustable threshold.","The same selective-tool framework could apply to other generation tasks where external knowledge is sometimes helpful and sometimes noisy, such as counseling, negotiation, or open-domain QA."],"forward_implications":["If correct, LLMs can benefit from model-initiated, selective use of commonsense knowledge rather than unconditional knowledge injection, reducing noise from irrelevant or redundant information.","Knowledge bases become swappable tools: the same fine-tuned policy works with COMET or CICERO by replacing the tool API, suggesting generalizability across commonsense resources.","The TOOL-ED dataset provides training traces for tool-calling empathetic dialogue, enabling other models to be adapted to this paradigm via supervised fine-tuning.","The reflection-based filtering of tool calls against golden responses is a viable method for creating tool-use demonstrations from existing dialogue corpora without manual annotation."],"supporting_citations":[{"why":"COMET, the commonsense knowledge base used as the primary empathetic tool.","marker":"(Bosselut et al., 2019)"},{"why":"The Empathetic Dialogues dataset, from which TOOL-ED is constructed and on which the model is evaluated.","marker":"(Rashkin et al., 2018)"},{"why":"ReAct, whose action-observation format the paper adapts for tool-calling responses.","marker":"(Yao et al., 2022)"},{"why":"LoRA, the parameter-efficient fine-tuning method used to train Vicuna-7B and Qwen1.5-14B.","marker":"(Hu et al., 2021)"},{"why":"Earlier work using COMET to enhance empathetic response generation, which the paper extends through tool calling.","marker":"(Qian et al., 2023)"},{"why":"CICERO, the second knowledge base defined as a tool to demonstrate plug-and-play generalization.","marker":"(Shen et al., 2022)"},{"why":"LLaMA3-70B, used as the Annotator and Reflector during TOOL-ED dataset construction.","marker":"(AI@Meta, 2024)"},{"why":"CEM, a strong commonsense-aware baseline, and source of the human A/B evaluation methodology.","marker":"(Sabour et al., 2022)"},{"why":"Provides the COMET BART version trained on ATOMIC2020, which is the concrete tool implementation used.","marker":"(Hwang et al., 2021)"}],"fun_headline_variants":["LLMs learn when to call tools for empathic replies","Selective tool calling enhances response empathy","Tool-based knowledge access improves dialogue empathy","Calling COMET selectively improves empathetic generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training data only contains tool calls that a strong LLM judged, after seeing the golden response, to produce knowledge relevant to that response; the model must then learn to decide when to call the tool without ever seeing the golden response.","fun_headline_variants_meta":{"raw":{"variants":["LLMs learn when to call tools for empathic replies","Selective tool calling enhances response empathy","Tool-based knowledge access improves dialogue empathy","Calling COMET selectively improves empathetic generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000431,"raw_usage":{"total_tokens":2166,"prompt_tokens":878,"completion_tokens":1288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":1232}},"tokens_in":494,"tokens_out":1288,"duration_ms":12342,"temperature":1.0,"reasoning_tokens":1232,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:46:23.856532+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Force the fine-tuned tool-calling model at inference to always answer directly, or replace the tool observation with random text, and measure the same empathy metrics; if performance does not drop below the tool-calling version, the observed improvement is not caused by the tool itself.","supporting_citations":[],"review_version":1}