Pith. sign in

REVIEW 3 major objections 5 minor 15 references

TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Fine-tuning an LLM to selectively call a commonsense knowledge tool improves empathetic responses over always injecting the knowledge.

desk verdict First tool-calling framing for empathetic knowledge infusion, with a useful new dataset, but the headline gains are undermined by golden-response lookahead in dataset construction. read the letter →

arxiv 2412.03096 v2 pith:PFK3VZD3 submitted 2024-12-04 cs.CL

classification cs.CL
keywords empatheticresponsegenerationtoollearningcommonsenseknowledgeCOMETTOOL-EDdatasetLLMfine-tuningLoRAdialogue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that letting an LLM decide when to consult an external commonsense knowledge base, rather than always feeding it knowledge, yields better empathetic responses. It constructs TOOL-ED by adding tool-use traces to the Empathetic Dialogues dataset, using LLaMA3 as both annotator (when to call) and reflector (filtering calls whose results align with the golden response). Fine-tuning Vicuna-7B and Qwen1.5-14B on TOOL-ED with COMET or CICERO as the tool improves BLEU, ROUGE, BERTScore, and human/LLM judgments of empathy over the same models fine-tuned on the original ED dataset. The central comparison against an always-inject-knowledge ablation supports the claim that selective access reduces noise.

What carries the argument

The central mechanism is the EKTC tool-calling loop: the model outputs an "Action: EmotionKnowledgeBase" with an "Action Input" containing the dialogue context, receives five commonsense relations (xIntent, xNeed, xWant, xEffect, xReact from COMET, or the equivalent relations from CICERO) as an observation, and then generates the response. The dataset construction uses LLaMA3-70B as an Annotator to decide whether a tool call is appropriate and as a Reflector to filter calls whose results are judged highly relevant to the golden response, yielding a training set where only 26.46% of instances contain tool calls. The ReAct-style (action, observation) format carries the mechanism, and LoRA fine-tuning adapts the base models to the new task.

What would settle it

Force the fine-tuned tool-calling model at inference to always answer directly, or replace the tool observation with random text, and measure the same empathy metrics; if performance does not drop below the tool-calling version, the observed improvement is not caused by the tool itself.

Watch

Extended reading notes

Core claim

The paper establishes that a dialogue model can be trained end-to-end to invoke a commonsense knowledge base as a tool, and that this active, selective invocation outperforms both no-knowledge baselines and unconditioned knowledge injection. The evidence is that Qwen1.5-14B and Vicuna-7B fine-tuned on TOOL-ED with COMET or CICERO as the tool score higher across BLEU-1/2/3/4, BERTScore, ROUGE-1/2/L, and in A/B human and GPT-4 evaluations against the same models fine-tuned on the original ED dataset. The framework, called EKTC, thus demonstrates that the timing of knowledge access is itself a learnable skill that improves response quality.

Load-bearing premise

The training data only contains tool calls that a strong LLM judged, after seeing the golden response, to produce knowledge relevant to that response; the model must then learn to decide when to call the tool without ever seeing the golden response.

Editorial extensions

If this is right

  • If correct, LLMs can benefit from model-initiated, selective use of commonsense knowledge rather than unconditional knowledge injection, reducing noise from irrelevant or redundant information.
  • Knowledge bases become swappable tools: the same fine-tuned policy works with COMET or CICERO by replacing the tool API, suggesting generalizability across commonsense resources.
  • The TOOL-ED dataset provides training traces for tool-calling empathetic dialogue, enabling other models to be adapted to this paradigm via supervised fine-tuning.
  • The reflection-based filtering of tool calls against golden responses is a viable method for creating tool-use demonstrations from existing dialogue corpora without manual annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The approach could extend beyond knowledge bases to other auxiliary modules such as sentiment classifiers, persona retrievers, or safety filters, letting the model decide when to consult each one.
  • Because the Reflector filters training calls using the golden response, the learned policy at inference may be more conservative than optimal; a reward model or test-time filtering could close that gap.
  • The lower tool-calling ratio observed in the fine-tuned models suggests they learn to reserve tool calls for emotionally intense contexts, which might be even better exploited with an adjustable threshold.
  • The same selective-tool framework could apply to other generation tasks where external knowledge is sometimes helpful and sometimes noisy, such as counseling, negotiation, or open-domain QA.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Emotional Knowledge Tool Calling (EKTC), a framework for empathetic response generation that wraps commonsense knowledge bases (COMET and CICERO) as callable tools and fine-tunes LLMs (Vicuna-7B, Qwen1.5-14B) with LoRA on a newly constructed dataset TOOL-ED. TOOL-ED is created from EmpatheticDialogues by using LLaMA3-70B as an Annotator to decide whether to call the tool and as a Reflector to filter tool outputs based on their relevance to the golden assistant response. The authors report automatic metrics (BLEU, BERTScore, ROUGE, Distinct) and human/LLM-based A/B evaluations showing improvements over models fine-tuned on the original ED dataset, and they run ablations including a 'noref' variant without the reflection step.

Significance. If the central claim holds—that selectively invoking an external commonsense knowledge base via tool calling improves empathetic response quality over always injecting knowledge or never using it—this would be a useful contribution to empathetic dialogue and tool-augmented LLMs. The paper provides a reproducible code repository, a plug-and-play tool design that works with two knowledge bases, and a new dataset. However, the experimental evidence for the central claim is weakened by a training-data confound and a lack of statistical rigor, as detailed in the major comments.

major comments (3)
  1. [§3.3, Appendix A.1] The training data construction uses the golden response in both the Annotator prompt ('your response: {assistant_response}' and asking whether the tool would be used to produce that response) and the Reflector (keeping only tool outputs judged relevant to that same golden response). This introduces a reference-conditioned selection bias: the model is trained on tool calls that are specifically beneficial for the ground-truth response, but at inference no reference is available and tool outputs are unfiltered. As a result, the reported gains of qwen_tool_comet and vicuna_tool_comet over qwen_lora and vicuna_lora (Tables 1, 2, 3) may stem from the curated training distribution rather than from the model's learned ability to call tools selectively. The 'noref' ablation in §4.5.2 removes only the Reflector, not the Annotator's access to the golden response, so it does not control for this confound. To support the central claim, the authors should either construct training data without reference lookahead (e.g., an Annotator that sees only dialogue context) or provide evidence that the inference-time tool-calling policy generalizes beyond the reference-filtered training distribution.
  2. [Tables 1-4] No variance, confidence intervals, or statistical significance tests are reported for any automatic metric. Many differences are small in absolute terms (e.g., BERTScore 0.8765 vs. 0.8760 for qwen models; ROUGE-L 0.1793 vs. 0.1696), so without repeated runs or paired significance tests the observed improvements cannot be distinguished from noise. The human evaluation (§4.4) uses only 100 randomly selected conversations and three annotators, with no inter-annotator agreement reported, and the GPT-4-based evaluation gives no sample size or agreement analysis. Quantitative claims of improvement should be accompanied by significance levels or at least standard deviations across multiple runs.
  3. [§4.3] The paper re-splits the ED dataset into training, validation, and test sets with a random 8:1:1 ratio. The original EmpatheticDialogues dataset has a predetermined split; using a different split makes the comparisons with prior published numeric results (e.g., CEM, KEMP, MIME in Table 1) invalid, because those baselines were evaluated on the original test partition. The authors should use the standard split or explicitly state that all baselines were re-evaluated on the same re-split; otherwise the claimed improvements over SOTA are not established.
minor comments (5)
  1. [Table 4] In Table 4, qwen_noref_comet reports BLEU-4 = 0.3130 and qwen_noref_cicero reports 0.3190; these are likely typos for 0.0313 and 0.0319, since other BLEU-4 values are around 0.03. Please correct them.
  2. [Table 1] Table 1 reports qwen_oneshot BLEU-3 as 0.1780, which is likely a typo (should be 0.0178). Also, the Dist-1/Dist-2 values for qwen_base (0.2322/0.1720) are implausible as Dist-1 is usually lower than or comparable to Dist-2; please verify these numbers.
  3. [§4.4] In the human evaluation description, 'Informativity (Inf.) meatures' should be 'measures'. Additionally, the paper should state whether the 100 samples are drawn from the test set and whether the three annotators were blinded to the model identities.
  4. [Abstract] The abstract contains a formatting error: 'EMPATHETICMPATHETIC DIALOGUE' should be 'EMPATHETIC DIALOGUE'.
  5. [§3.3] The paper reports a 26.46% tool-call insertion rate but does not specify the resulting dataset size or the filtering rate of the Reflector. This information is useful for assessing the dataset quality and the potential selection bias.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical evaluation is self-contained, though the reference-conditioned dataset construction is a validity concern rather than a circular step.

full rationale

The paper makes an empirical claim that fine-tuning on TOOL-ED improves empathetic response generation, and it supports that claim with automatic metrics, human A/B testing, and an LLM judge against external baselines. No step in the method derives a prediction from its own input by construction: the COMET and CICERO observations are computed from the dialogue context by fixed external knowledge models, the LoRA objective in Eq. 5 is standard supervised next-token prediction, and the test-time tool-call decisions are made by the fine-tuned model rather than by the Annotator. The only self-citation, Zhang et al. (2024) for the tool-learning prompt template in Appendix A.2, is not load-bearing. One genuine concern is that the Annotator and Reflector prompts in Figures 6 and 7 include the golden assistant response, so the training tool-call labels are reference-conditioned; this is a leakage or validity confound for the claimed benefit of selective tool calling, but it is not a circular derivation because the evaluated responses are generated at inference without access to the golden response. The Limitation section notes only the absence of multi-tool support and does not address this leakage, which is a correctness risk rather than a circularity risk.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The framework relies on standard tool-learning and knowledge-base assumptions. The most significant burden is the golden-response-filtered dataset, which is a design choice that can bias training and evaluation.

free parameters (3)
  • LoRA configuration = not reported
    The paper states LoRA is used (Section 3.4) but gives no rank, alpha, or dropout values; these affect training quality and reproducibility.
  • Tool-call insertion rate = 26.46% of training examples
    Emerges from LLaMA-3 annotation and Reflector filtering (Section 3.3); not tuned, but it defines the training distribution the model learns.
  • Dataset split ratio = 80/10/10
    The authors re-split ED with ratio 8:1:1 (Section 4.3) instead of using the official ED split, which affects comparability with baselines.
assumptions (4)
  • domain assumption COMET's five commonsense relations (xIntent, xNeed, xWant, xEffect, xReact) provide useful information for empathetic response generation.
    Section 3.2 designates COMET as the empathetic tool and feeds its outputs as observations; the entire framework relies on this knowledge being beneficial.
  • domain assumption The assistant responses in the ED dataset are correct targets for training and evaluation.
    The paper uses these as golden responses to construct TOOL-ED and to compute all automatic metrics (Sections 3.3 and 4.4).
  • domain assumption LLaMA3-70B annotations of tool-use timing and relevance are reliable.
    Annotator and Reflector are implemented with LLaMA3-70B (Section 3.3), and no validation of annotation quality or agreement is reported.
  • ad hoc to paper Filtering tool outputs by relevance to the golden response does not introduce a train/test distribution shift.
    Reflector keeps only tool calls whose COMET output is judged relevant to the gold response (Figure 7), while at inference the tool output is unfiltered; the paper does not analyze this mismatch.
invented entities (1)
  • EmotionKnowledgeBase tool independent evidence
    purpose: A callable API wrapping COMET (or CICERO) that returns commonsense relations about the user's intent, needs, wants, effects, and reactions.
    The wrapper is a new software abstraction, but the underlying knowledge base is the public COMET/CICERO model, which provides external grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM." pith.science (2026). https://pith.science/paper/PFK3VZD3

@misc{pith2026241203096,
  author       = {Pith},
  title        = {Pith review of: TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFK3VZD3}},
  note         = {Machine review of arXiv:2412.03096}
}
read the original abstract

Empathetic conversation is a crucial characteristic in daily conversations between individuals. Nowadays, Large Language models (LLMs) have shown outstanding performance in generating empathetic responses. Knowledge bases like COMET can assist LLMs in mitigating illusions and enhancing the understanding of users' intentions and emotions. However, models remain heavily reliant on fixed knowledge bases and unrestricted incorporation of external knowledge can introduce noise. Tool learning is a flexible end-to-end approach that assists LLMs in handling complex problems. In this paper, we propose Emotional Knowledge Tool Calling (EKTC) framework, which encapsulates the commonsense knowledge bases as empathetic tools, enabling LLMs to integrate external knowledge flexibly through tool calling. In order to adapt the models to the new task, we construct a novel dataset TOOL-ED based on the EMPATHETICMPATHETIC DIALOGUE (ED) dataset. We validate EKTC on the ED dataset, and the experimental results demonstrate that our framework can enhance the ability of LLMs to generate empathetic responses effectively.

Figures

Figures reproduced from arXiv: 2412.03096 by the authors.

Figure 1
Figure 1. Architecture for the application of a specified [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of the EKTC framework consists of two stages: Dataset Reconstruction & Training [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Inference details of empathetic response gen [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Tool-calling ratio of the fine-tuned models. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Examples of conversations by the user inter [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Prompt for Anotator There are two roles in the conversation,including user and assistant. Assuming you are the role of assistant and you have an emotion Knowledge Base as a tool, which can provide the following additional knowledge to help you provide a better reply. A…
Figure 7
Figure 7. Figure 7: Prompt for Reflector 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Discription of the empathetic tool This is an empathetic dialogue task: The first worker (Speaker) is given an emotion label and writes his own description of a situation when he has felt that way. Then, Speaker tells his story in a conversation with a second worker (L…
Figure 9
Figure 9. Figure 9: Prompt for Tool learning You are an expert in empathy assessment. Here are two statements, please evaluate their level of empathy. Dialogue history context: {history} Statement 1: {statement1} Statement 2: {statement2} Please evaluate the level of empathy between two s…
Figure 10
Figure 10. Figure 10: Prompt Template for Empathy Scorer 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Prompt Template for Consistency Scorer You are an expert in language fluency assessment. Here are two statements, please evaluate their fluency. Statement 1: {statement1} Statement 2: {statement2} Please evaluate the fluency of two statements based on the following cr…
Figure 12
Figure 12. Figure 12: Prompt Template for Fluency Scorer user: "Hi! How are you doing today?" assistant: "I’m fine. How about you?" user: "Job interviews always make me sweat bullets, which makes me uncomfortable in general to be looked at under a microscope like that." function_call: "{\"…
Figure 13
Figure 13. Figure 13: Example from the TOOL-ED dataset 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

  1. [1]

    Understand the depth of the other person's emotions

  2. [2]

    name\": \

    Based on what you say in the conversation and emotional intensity of the user, please check if you have untilized this tool to answer the user's conversation, and tell me the reason. Your output should adhere to the format: ##Result one of [Yes, No] Annotator prompt Figure 6: Prompt for Anotator There are two roles in the conversation,including user and a...

  3. [3]

    Hi! How are you doing today?

    Whether the expression is natural or not If the fluency of statement 1 is higher than that of statement 2, please output 1; If the fluency of statement 1 is lower than that of statement 2, output 0. You don't need to output the analysis process, just output 1 or 0. ##Result one of [1, 0] LLM-based evaluation prompt (Fluency) Figure 12: Prompt Template for...

  4. [5]

    arXiv preprint arXiv:2309.17452

    Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452. 9 Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366. Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Al...

  5. [7]

    Multiview Contextual Commonsense Inference: A New Dataset and Task

    Multiview contextual commonsense infer- ence: A new dataset and task. arXiv preprint arXiv:2210.02890. Zhengliang Shi, Shen Gao, Xiuyi Chen, Lingyong Yan, Haibo Shi, Dawei Yin, Zhumin Chen, Pengjie Ren, Suzan Verberne, and Zhaochun Ren. 2024. Learning to use tools via cooperative and interactive agents. arXiv preprint arXiv:2403.03031. Zhenqiao Song, Xiao...

  6. [8]

    I’m fine. How about you?

    Don’t lose yourself! empathetic response generation via explicit self-other awareness. In Find- ings of the Association for Computational Linguistics: ACL 2023, pages 13331–13344, Toronto, Canada. As- sociation for Computational Linguistics. Yuyue Zhao, Jiancan Wu, Xiang Wang, Wei Tang, Dingxian Wang, and Maarten de Rijke. 2024. Let me do it for you: Towa...

  7. [11]

    Whether the response to the other person's emotions is sincere and warm If the empathy level of statement 1 is higher than that of statement 2, output 1; If the empathy level of statement1 is weaker than statement2, output 0. You don't need to output the analysis process, just output 1 or 0 ##Result one of [1, 0] LLM-based evaluation prompt (Empathy) Figu...

  8. [12]

    The relevance of sentence content to the theme of dialogue history

Show all 15 references
  1. [13]

    The correlation between sentences, including key points of the dialogue, and accurate expression If the relevance of statement 1 is higher than that of statement 2, output 1; If the correlation of statement 1 is weaker than that of statement 2, output 0. You don't need to outp...

  2. [14]

    Coherence of language

  3. [2019]

    In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Flo- rence, Italy

    COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Flo- rence, Italy. Association for Computational Linguis- tics. Tom B. Brown, Benjamin Man...

  4. [2020]

    In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA

    Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY , USA. Curran Associates Inc. Hua Cai, Xuli Shen, Qing Xu, Weilin Shen, Xiaomei Wang, Weifeng Ge, Xiaoqing Zheng, and Xia...

  5. [2022]

    In Proceedings of the 29th International Conference on Computational Linguistics, pages 669– 683

    Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and au- tomatic evaluation metric for empathetic dialogue generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 669– 683. Jiwei Li, Michel Galley...

  6. [2023]

    See https://vicuna

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6. Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the l...

  7. [2024]

    In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10631–10635

    Pecer: Empathetic response generation via dynamic personality extraction and contextual emo- tional reasoning. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10631–10635. IEEE. Hyungjoo Chae, Yongho Song, Kai Tzu ...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.