Pith. sign in

REVIEW 4 major objections 5 minor 72 references

Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-stage LLM reasoner that merges structured attributes with user reviews and filters knowledge by utility produces the strongest textual responses reported on the MMConv multimodal dialogue benchmark.

desk verdict A solid, incremental systems paper for multimodal task-oriented dialog: the dual-knowledge composition is new, the LLM self-judging filter is the one genuinely weak link, and the empirical reporting needs more rigor, but it deserves a normal peer-review slot. read the letter →

arxiv 2509.07817 v1 pith:HEZZDXHC submitted 2025-09-09 cs.CL cs.MM

Xiaolin Chen , Xuemeng Song , Haokun Wen , Weili Guan , Xiangyu Zhao , Liqiang Nie This is my paper · ORCID
classification cs.CLcs.MM
keywords multimodaltask-orienteddialoguetextualresponsegenerationunstructuredreviewknowledgetypefilteringLLMreasoningintention-responsedecouplingMMConvbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that textual responses in multimodal task-oriented dialogue improve when the system uses both structured attribute knowledge (addresses, scores, attributes) and unstructured review knowledge (user experiences), dynamically filters which knowledge type actually helps each query, and decouples intention understanding from response generation. On the MMConv benchmark, the proposed DK2R pipeline beats all baselines—including larger multimodal LLMs—on every automated metric, and human evaluators preferred its replies roughly three-quarters of the time over the strongest baseline. The contribution is treating knowledge selection as a separate, train-free step: an LLM first drafts provisional responses with each knowledge type, then judges which type contributed. A second innovation is splitting the task into intention-oriented key-clue reasoning followed by knowledge-aware response generation. The paper therefore argues that the gains come specifically from adding review knowledge, filtering knowledge types dynamically, and keeping intent reasoning clean of external knowledge noise.

What carries the argument

The central mechanism is probe-driven knowledge type filtering: for each candidate knowledge type, the LLM first generates a provisional response conditioned on that type, then acts as a judge and answers Yes/No whether the type supplied information that contributed to that response. This training-free filter determines which knowledge (structured, unstructured, or both) is passed to the final generator. The second key machinery is two-stage reasoning-enhanced response generation: an MLLM first summarizes 'user needs' and 'keywords' from the multimodal context, and those key clues are fed to a LoRA-finetuned text-only LLM alongside the filtered knowledge for the final response. Context image

What would settle it

Sample a random set of MMConv test dialogs, run only the probe-generation and knowledge-utility-assessment steps, and have three human annotators judge for each knowledge type whether it actually supplied information used in the ground-truth response. If the LLM's Yes/No agrees with the annotators at chance level, the filtering step is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The paper claims that textual responses in multimodal task-oriented dialogue improve substantially when the system (1) retrieves both structured attribute knowledge and unstructured review knowledge from an external knowledge base; (2) uses an LLM to generate provisional probe responses with each knowledge type and then have the same LLM judge whether each type contributed, keeping only useful types; and (3) separates intention reasoning—summarizing user needs and keywords from the multimodal context via an MLLM—from knowledge-aware response generation via a LoRA-finetuned text-only LLM. On MMConv, the full pipeline ('DK2R') surpasses every baseline, including Llama-3-8B, GPT-4 Turbo, and 7B

Load-bearing premise

The knowledge-type filter assumes that Llama-3-8B can reliably judge whether a knowledge type helped generate a good response, with no ground-truth labels of knowledge utility; if that self-assessment is unreliable, the filter may keep the wrong knowledge or discard the useful type.

Editorial extensions

If this is right

  • Adding unstructured review knowledge lets the system answer suggestion-oriented questions (e.g., 'any tips?') with specific, review-grounded details instead of generic descriptions.
  • Structured attribute knowledge is the larger contributor on this dataset; removing it hurts more than removing reviews, so domain-factual knowledge should remain a first-class input.
  • Separating intention reasoning from generation improves scores, with user-need clues contributing more than keyword clues.
  • The two-stage template generalizes: wrapping MLLM backbones with DK2R improves each over its unfiltered, single-stage counterpart.
  • Text-only LLMs can outperform same-scale MLLMs at textual response generation when images are captioned, supporting the paper's design choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The filter's Yes/No judgments could be made auditable: a human-annotated subset of knowledge-utility decisions would isolate judge accuracy from generation quality and reveal whether the filter is truly selecting the right knowledge type.
  • The probe-then-filter pattern should transfer to other retrieval-augmented generation tasks where knowledge is heterogeneous and only some types are useful per query.
  • Captioning images before reasoning likely loses fine-grained visual details (e.g., product appearance, spatial layout), so stronger vision-language fusion could yield further gains once reasoning quality is held constant.
  • Because structured knowledge proved more valuable than reviews in this dataset, an adaptive fusion weighting knowledge types by query type would probably improve results further.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DK2R, a pipeline for textual response generation in multimodal task-oriented dialog systems. DK2R first extracts both structured attribute knowledge and unstructured review knowledge from an external KB, using text matching and CLIP-based visual retrieval with a threshold. A probe-driven filter uses Llama-3-8B both to generate provisional responses conditioned on each knowledge type and then to judge whether that knowledge type contributed, yielding a selected knowledge set. A two-stage generation process first summarizes intention-oriented key clues with xGen-MM, then generates the response with LoRA-fine-tuned Llama-3-8B conditioned on the context, visual captions, key clues, and filtered knowledge. Experiments on MMConv report BLEU-1~4 and NIST scores, ablations over knowledge components and key clues, a 50-sample human evaluation, and qualitative examples. The authors release code and parameters.

Significance. If the reported gains are statistically robust, the paper makes a useful empirical contribution. It is, to my knowledge, the first in this line of MMConv-based work to integrate unstructured review knowledge into LLM-based response generation, and the two-stage decoupling of intention reasoning from knowledge-grounded generation is a plausible design that is supported by the ablations. The release of code and parameters is a strength, and the ablations consistently show each component helping in the right direction. The significance is currently limited by the lack of statistical evidence and by the absence of any direct validation of the learned knowledge-type filter, which is the component most central to the paper's claimed novelty. The human evaluation, while favorable, is small and lacks reliability metrics. With additional validation, the approach could be a practically useful recipe for LLM-based multimodal TOD response generation.

major comments (4)
  1. [§4.3, Table 2; §4.4, Table 3] All automatic results are point estimates from a single training run, with no confidence intervals, significance tests, or multiple random seeds. The headline margin over Llama-3-8B is modest (BLEU-1 49.63 vs. 47.48; NIST 5.6856 vs. 5.3308), and the key filtering ablation w/o-KTypeFilter vs. DK2R is only 49.63 vs. 48.61 BLEU-1. These differences could be within run-to-run variation. The claim in §4.3 that DK2R 'demonstrates superiority over all the baselines' therefore needs support from paired significance tests (e.g., bootstrap or paired resampling) or multiple seeds with variance reporting. Without this, the central comparison is not yet established.
  2. [§3.3.2, Eq. (3); §4.4, Fig. 6] The probe-driven knowledge type filter is never validated against ground truth. Llama-3-8B both generates the provisional response and then judges Yes/No whether the knowledge type contributed; the paper even notes that no annotations of knowledge-type utility exist. Yet the only evidence for this component is the end-to-end ablation in Table 3 and three illustrative examples in Fig. 6. This leaves the possibility that the filter’s BLEU gain comes not from correct selection but from a side effect, such as reduced prompt length or randomness. I ask the authors to directly measure judge accuracy: e.g., construct a labeled or human-annotated sample of (context, knowledge type, gold response) where the useful type is inferred from the gold response, and report agreement, kappa, and per-type confusion. Reporting the distribution of Yes/No decisions across the test set would also help.
  3. [§4.3, Fig. 4] The human evaluation is based on only 50 randomly selected test samples, with three annotators but no inter-annotator agreement reported and no error bars on the win rates. Given that the automatic metrics are also not statistically tested, the qualitative comparison is a useful complement but is not by itself sufficient to support the strength of the claimed superiority. Please report per-annotator agreement (e.g., Fleiss' kappa), exact counts rather than only percentages, and, if possible, a slightly larger sample or bootstrap confidence intervals.
  4. [§4.4, Table 3; §4.5, Table 4] The ablations compare variants that differ by a single component, but no significance testing is applied to these within-model comparisons either. This is particularly important for the filtering component because the delta is small. Also, the conclusion in §4.4 that structured knowledge contributes more than unstructured knowledge (w/o-StrucK worse than w/o-UnstrucK) may be confounded by the larger number of attributes vs. reviews per entity (13.7 vs. 24.2, Table 1) and by retrieval completeness. A length- or coverage-controlled analysis would make this interpretation more convincing.
minor comments (5)
  1. [§4.4, first variant description] The description of w/o-AllK says it removes K_F from Eqn. (4), but Eqn. (4) is the key-clue generation step; the removal should be from Eqn. (5), the response generation equation.
  2. [§4.3, last sentence before Fig. 4] 'further validating its effectiveness over DK2R' should presumably read 'over Llama-3-8B'.
  3. [§4.2 and §4.3] The experimental setting reports 5 epochs of LoRA fine-tuning but does not state the learning rate, LoRA rank/alpha, batch size, or whether all baselines were trained with the same number of steps / comparable compute. Please add these details, especially since the Llama-3-8B baseline is a critical comparison.
  4. [Algorithm 1] The pseudocode says 'repeat: draw C_i from D ... until objective converges', which is not a standard training loop; it does not specify the loss, optimizer, or minibatching. Because Eqn. (5) is trained with cross-entropy, it would be clearer to state the actual gradient-update step and stopping criterion.
  5. [§4.5 / Table 4] The names 'w/o-Need' and 'w/o-Keywords' are used; the text in §4.5 also writes 'w/o-Need' but the component is 'user needs'. Please unify naming to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DK2R is an empirical pipeline; the only self-referential element is an LLM-judged knowledge filter whose utility is tested by ablation rather than assumed.

full rationale

The paper's central claim is an empirical performance comparison (Section 4.3, Table 2) against external baselines on MMConv, and the proposed components are validated by ablations (Tables 3 and 4). No equation defines its output in terms of its input: Eq. (2) generates separate probe responses from each knowledge type, Eq. (3) obtains Yes/No utility judgments, and Eq. (5) generates the final response from the filtered knowledge and key clues; the filtering decision is not the quantity being predicted. The probe-driven filter does let the same Llama-3-8B model both create and judge provisional responses, but this is an operational definition of knowledge utility in the absence of labels, and its contribution is tested by the w/o-KTypeFilter ablation (BLEU-1 48.61 vs 49.63) rather than assumed. Self-citations to the authors' prior DKMD/MDS-S2 work are used for design continuity and baselines, but no load-bearing premise is justified solely by an author-overlapping citation and no uniqueness theorem is imported. The acknowledged limitation that only type-level, not item-level, filtering is performed is a scope limitation, not circularity. Thus the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the dataset's review annotations being meaningful, on the LLM judge being reliable, on image captions preserving visual information, and on BLEU/Nist reflecting response quality. These are domain assumptions about the task and evaluation, not standard mathematical axioms or new physical entities.

free parameters (2)
  • CLIP visual matching threshold theta = 0.1
    Threshold for retaining image-matched knowledge entities in Section 3.2, set as 0.1 in Section 4.2 with no sensitivity analysis reported; it controls how much visual knowledge enters the response pipeline.
  • LoRA fine-tuning configuration = not reported
    Rank, alpha, learning rate, batch size, and the 5-epoch schedule are chosen for the final model but not specified; the quantitative results depend on this tuning.
assumptions (5)
  • domain assumption MMConv ground-truth responses are the correct references, and BLEU/Nist n-gram overlap is a suitable quality proxy.
    Used in Section 4.2 and all quantitative comparisons; no semantic or human-based evaluation is performed on the full test set.
  • domain assumption LLM self-assessment can identify useful knowledge types without ground-truth utility labels.
    Section 3.3.2, Eq. 3 uses Llama-3-8B to judge whether its own knowledge-conditioned probe response used the knowledge; no direct validation of judge accuracy is provided.
  • domain assumption Visual context is adequately represented by BLIP captions and CLIP image matching.
    Section 3.2 and Section 3.3.1 convert images to text captions and use CLIP cosine similarity for retrieval; a lossy caption could miss details needed for grounded responses.
  • domain assumption A text-only LLM is preferable to an MLLM of comparable scale for this task.
    Sections 1 and 3.3.1 cite LogicVista [62] for this claim, but that benchmark is visual logical reasoning, not task-oriented dialog; Table 2 compares different model families, sizes, and training protocols.
  • domain assumption xGen-MM key-clue summarization provides reliable user-intention proxies.
    Section 3.4, Eq. 4 uses xGen-MM to extract user needs and keywords; clue quality is only validated indirectly through end-to-end ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems." pith.science (2026). https://pith.science/paper/HEZZDXHC

@misc{pith2026250907817,
  author       = {Pith},
  title        = {Pith review of: Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HEZZDXHC}},
  note         = {Machine review of arXiv:2509.07817}
}
read the original abstract

Textual response generation is pivotal for multimodal \mbox{task-oriented} dialog systems, which aims to generate proper textual responses based on the multimodal context. While existing efforts have demonstrated remarkable progress, there still exist the following limitations: 1) \textit{neglect of unstructured review knowledge} and 2) \textit{underutilization of large language models (LLMs)}. Inspired by this, we aim to fully utilize dual knowledge (\textit{i.e., } structured attribute and unstructured review knowledge) with LLMs to promote textual response generation in multimodal task-oriented dialog systems. However, this task is non-trivial due to two key challenges: 1) \textit{dynamic knowledge type selection} and 2) \textit{intention-response decoupling}. To address these challenges, we propose a novel dual knowledge-enhanced two-stage reasoner by adapting LLMs for multimodal dialog systems (named DK2R). To be specific, DK2R first extracts both structured attribute and unstructured review knowledge from external knowledge base given the dialog context. Thereafter, DK2R uses an LLM to evaluate each knowledge type's utility by analyzing LLM-generated provisional probe responses. Moreover, DK2R separately summarizes the intention-oriented key clues via dedicated reasoning, which are further used as auxiliary signals to enhance LLM-based textual response generation. Extensive experiments conducted on a public dataset verify the superiority of DK2R. We have released the codes and parameters.

Figures

Figures reproduced from arXiv: 2509.07817 by the authors.

Figure 1
Figure 1. Illustration of a multimodal dialog system. “ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed model for multimodal task-oriented dialog systems, which consists of three key components: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Workflow of the proposed DK2R. You are a helpful assistant. Please think and generate the response based on the given context, the pre-extracted context key clues, and related knowledge. Please prioritize using related knowledge to generate responses. If unable to answer, maintain critical thinking and use your own knowledge to generate responses. Furthermore, please do not rely solely on the pre-extracted context k… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Human evaluation on responses generated by DK2R and Llama- [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Comparison between our DK2R and Llama-3-8B on three testing dialog pairs. The smiling emoji indicates the preferred response, while the sad one marks the less desirable one. preferred response, while the sad one marks the less desirable one. As we can see, for Case 1, …
Figure 6
Figure 6. Figure 6: Illustration of the knowledge type filtering. The green tick denotes the knowledge type is judged as useful, while the red cross [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Word clouds of learned key clues derived from testing samples. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 65 canonical work pages

  1. [2]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023)

    arXiv 2023

  2. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, et al. 2025. Qwen2.5-VL Technical Report

    work page 2025

  3. [4]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, et al. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. CoRR abs/2401.02954 (2024)

    arXiv 2024

  4. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  5. [6]

    Hardik Chauhan, Mauajama Firdaus, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Ordinal and Attribute Aware Response Generation in a Multimodal Dialogue System. In Proceedings of the Conference of the Association for Computational Linguistics . Association for Computational Linguistics, 5437–5447

    work page 2019

  6. [7]

    Jiankang Chen, Ling Deng, Zhiyong Gan, Wei-Shi Zheng, and Ruixuan Wang. 2024. FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector. In Proceedings of the ACM International Conference on Multimedia . Association for Computing Machinery, 1981–1990

    work page 2024

  7. [8]

    Xiaolin Chen, Xuemeng Song, Liqiang Jing, Shuo Li, Linmei Hu, and Liqiang Nie. 2023. Multimodal Dialog Systems with Dual Knowledge-Enhanced Generative Pretrained Language Model. ACM Transactions on Information Systems 42, 2 (2023)

    work page 2023

  8. [9]

    Xiaolin Chen, Xuemeng Song, Yinwei Wei, Liqiang Nie, and Tat-Seng Chua. 2023. Dual Semantic Knowledge Composed Multimodal Dialog Systems. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM, 1518–1527

    work page 2023

Show all 72 references
  1. [10]

    Xiaolin Chen, Xuemeng Song, Jianhui Zuo, Yinwei Wei, Liqiang Nie, and Tat-Seng Chua. 2025. Domain-aware Multimodal Dialog Systems with Distribution-based User Characteristic Modeling. ACM Transactions on Multimedia Computing, Communications, and Applications 21, 2 (2025), 66:1–66:22

    2025

  2. [11]

    Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. 2023. DISCO: Distilling Counterfactuals with Large Language Models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistic...

    2023

  3. [12]

    Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR (2014)

    2014

  4. [13]

    John Joon Young Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics . Associ...

    2023

  5. [14]

    Chen Cui, Wenjie Wang, Xuemeng Song, Minlie Huang, Xin-Shun Xu, and Liqiang Nie. 2019. User Attention-guided Multimodal Dialog Systems. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM, 445–454

    2019

  6. [15]

    Yang Deng, Yaliang Li, Wenxuan Zhang, Bolin Ding, and Wai Lam. 2022. Toward Personalized Answer Generation in E-Commerce via Multi- perspective Preference Modeling. ACM Transactions on Information Systems 40, 4 (2022), 87:1–87:28

    2022

  7. [16]

    George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-Gram Co-Occurrence Statistics. In Proceedings of the International Conference on Human Language Technology Research . Morgan Kaufmann Publishers Inc., 138–145

    2002

  8. [17]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  9. [18]

    Jeffrey L. Elman. 1990. Finding Structure in Time. Cognitive Science 14, 2 (1990), 179–211

    1990

  10. [19]

    Alexey Dosovitskiy et al. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In Proceedings of the International Conference on Learning Representations . OpenReview.net

    2021

  11. [20]

    Weidong He, Zhi Li, Dongcai Lu, Enhong Chen, Tong Xu, Baoxing Huai, and Jing Yuan. 2020. Multimodal Dialogue Systems via Capturing Context-aware Dependencies of Semantic Elements. In Proceedings of the ACM International Conference on Multimedia . Association for Computing Mach...

    2020

  12. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of the International Conference on Learning Representations . OpenReview.net. Manuscript subm...

    2022

  13. [22]

    Linmei Hu, Xinyu Zhang, Dandan Song, Changzhi Zhou, Hongyu He, and Liqiang Nie. 2025. Efficient and Effective Role Player: A Compact Knowledge-grounded Persona-based Dialogue Model Enhanced by LLM Distillation. ACM Transactions on Information Systems 43, 3 (2025), 1-29 pages

    2025

  14. [23]

    Yuxuan Hu, Minghuan Tan, Chenwei Zhang, Zixuan Li, Xiaodan Liang, Min Yang, Chengming Li, and Xiping Hu. 2024. APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation. InProceedings of ACM International Conference on Informatio...

    2024

  15. [24]

    Yuxuan Hu, Minghuan Tan, Chenwei Zhang, Zixuan Li, Xiaodan Liang, Min Yang, Chengming Li, and Xiping Hu. 2024. APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation. In Proceedings of the ACM International Conference on Infor...

    2024

  16. [25]

    Yiyang Jiang, Wengyu Zhang, Xulu Zhang, Xiaoyong Wei, Chang Wen Chen, and Qing Li. 2024. Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval. In Proceedings of the ACM International Conference on Multimedia, MM 2024 . Association...

    2024

  17. [26]

    SeongKu Kang, Bowen Jin, Wonbin Kweon, Yu Zhang, Dongha Lee, Jiawei Han, and Hwanjo Yu. 2025. Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generation. In Proceedings of the ACM International Conference on Web Search and Data Mining . ACM, 895–904

    2025

  18. [27]

    Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures. In Proceedings of the Annual Meeting of the Association for Computational Linguistics . A...

    2018

  19. [28]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the Ann...

    2020

  20. [29]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Annual Confe...

    2020

  21. [30]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In Proceedings of the International Conference on Machine Learning . PMLR, 12888–12900

    2022

  22. [31]

    Jiahao Li, Weijian Ma, Xueyang Li, Yunzhong Lou, Guichun Zhou, and Xiangdong Zhou. 2025. CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashvill...

    2025

  23. [32]

    Shimin Li, Hang Yan, and Xipeng Qiu. 2022. Contrast and Generation Make BART a Good Dialogue Emotion Recognizer. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 11002–11010

    2022

  24. [33]

    Yanran Li, Wenjie Li, and Zhitao Wang. 2021. Graph-Structured Context Understanding for Knowledge-grounded Response Generation. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM, 1930–1934

    2021

  25. [34]

    Yanran Li, Ruixiang Zhang, Wenjie Li, and Ziqiang Cao. 2022. Hierarchical Prediction and Adversarial Learning For Conditional Response Generation. IEEE Transactions on Knowledge and Data Engineering 34, 1 (2022), 314–327

    2022

  26. [35]

    Lizi Liao, Le Hong Long, Zheng Zhang, Minlie Huang, and Tat-Seng Chua. 2021. MMConv: An Environment for Multimodal Conversational Search across Multiple Domains. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval . Ass...

    2021

  27. [36]

    Lizi Liao, Yunshan Ma, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2018. Knowledge-aware Multimodal Dialogue Systems. In Proceedings of the ACM International Conference on Multimedia . ACM, 801–809

    2018

  28. [37]

    Lizi Liao, Ryuichi Takanobu, Yunshan Ma, Xun Yang, Minlie Huang, and Tat-Seng Chua. 2022. Topic-Guided Conversational Recommender in Multiple Domains. IEEE Transactions on Knowledge and Data Engineering 34, 5 (2022), 2485–2496

    2022

  29. [38]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In Proceedings of the Advances in Neural Information Processing Systems. 34892–34916

    2023

  30. [39]

    Zhiyuan Ma, Jianjun Li, Guohui Li, and Yongjing Cheng. 2022. UniTranSeR: A Unified Transformer Semantic Representation Framework for Multimodal Task-Oriented Dialog System. In Proceedings of the Annual Meeting of the Association for Computational Linguistics . ACL, 103–114

    2022

  31. [40]

    Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems. In Proceedings of Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistic...

    2018

  32. [41]

    Liqiang Nie, Fangkai Jiao, Wenjie Wang, Yinglong Wang, and Qi Tian. 2021. Conversational Image Search. IEEE Transactions on Image Processing 30 (2021), 7732–7743

    2021

  33. [42]

    Liqiang Nie, Wenjie Wang, Richang Hong, Meng Wang, and Qi Tian. 2019. Multimodal Dialog System: Generating Responses via Adaptive Decoders. In Proceedings of the ACM International Conference on Multimedia . ACM, 1098–1106

    2019

  34. [43]

    OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023)

    2023 arXiv

  35. [44]

    Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. 2023. LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation. In Proceedings of the ACM International Conference on Multimedia . ACM, 643–654. Manuscript submitted to ACM Dual Knowledge...

    2023

  36. [45]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.The Journal of Machine Learning Research 21 (2020), 140:1–140:67

    2020

  37. [46]

    Khapra, and Karthik Sankaranarayanan

    Amrita Saha, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018. Towards Building Large Scale Multimodal Domain-Aware Conversation Systems. In Proceedings of the AAAI Conference on Artificial Intelligence . AAAI Press, 696–704

    2018

  38. [47]

    Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting Large Language Models with Answer Heuristics for Knowledge-Based Visual Question Answering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR . IEEE, 14974–14983

    2023

  39. [48]

    Noam Shazeer. 2020. GLU Variants Improve Transformer. CoRR (2020)

    2020

  40. [49]

    Lei Shen, Haolan Zhan, Xin Shen, Yonghao Song, and Xiaofang Zhao. 2021. Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation. In Proceedings of the ACM International Conference on Multimedia . Association for Computing Machinery, 4287–4296

    2021

  41. [50]

    Weiwei Sun, Shuyu Guo, Shuo Zhang, Pengjie Ren, Zhumin Chen, Maarten de Rijke, and Zhaochun Ren. 2024. Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems. ACM Transactions on Information Systems 42, 1 (2024), 17:1–17:29

    2024

  42. [51]

    Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Proceedings of the International Conference on Neural Information Processing Systems . MIT Press, 3104–3112

    2014

  43. [52]

    Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment Knowledge Enhanced Pre- training for Sentiment Analysis. In Proceedings of the Annual Meeting of the Association for Computational Linguistics . Association for Computat...

    2020

  44. [53]

    Bo Tong, Bokai Lai, Yiyi Zhou, Gen Luo, Yunhang Shen, Ke Li, Xiaoshuai Sun, and Rongrong Ji. 2025. FlashSloth : Lightning Multimodal Large Language Models via Embedded Visual Compression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR . Computer Vision...

    2025

  45. [54]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation La...

    2023 arXiv

  46. [55]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems . 5998–6008

    2017

  47. [56]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. CoRR (2017)

    2017

  48. [57]

    Jian Wang, Dongding Lin, and Wenjie Li. 2024. Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialogue. ACM Transactions on Information Systems 42, 5 (2024), 124:1–124:27

    2024

  49. [58]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned Language Models are Zero-Shot Learners. In Proceedings of the International Conference on Learning Representations . OpenReview.net, 1–46

    2022

  50. [59]

    Haokun Wen, Xuemeng Song, Xiaolin Chen, Yinwei Wei, Liqiang Nie, and Tat-Seng Chua. 2024. Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Ret...

    2024

  51. [60]

    David Wilmot and Frank Keller. 2021. Memory and Knowledge Augmented Language Models for Inferring Salience in Long-Form Stories. In Proceedings of the Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 851–865

    2021

  52. [61]

    Yuxiang Wu, Yu Zhao, Baotian Hu, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2022. An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing . Association fo...

    2022

  53. [62]

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. 2024. LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts. CoRR abs/2407.04973 (2024)

    2024 arXiv

  54. [63]

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S. Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin ...

    2024

  55. [64]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, et al. 2025. Qwen2.5 Technical Report

    2025

  56. [65]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yu...

    2024

  57. [66]

    Wei Yuan, Chaoqun Yang, Guanhua Ye, Tong Chen, Nguyen Quoc Viet Hung, and Hongzhi Yin. 2024. FELLAS: Enhancing Federated Sequential Recommendation with LLM as External Services. ACM Transactions on Information Systems (2024)

    2024

  58. [67]

    Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. In Annual Conference on Neural Information Processing Systems . 12360–12371

    2019

  59. [68]

    Haoyu Zhang, Meng Liu, Zan Gao, Xiaoqiang Lei, Yinglong Wang, and Liqiang Nie. 2021. Multimodal Dialog System: Relational Graph-based Context-aware Question Understanding. In Proceedings of the ACM Multimedia Conference . ACM, 695–703. Manuscript submitted to ACM 24 Xiaolin Ch...

    2021

  60. [69]

    Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Task-Oriented Dialog Systems That Consider Multiple Appropriate Responses under the Same Context. In Proceedings of the AAAI Conference on Artificial Intelligence . AAAI Press, 9604–9611

    2020

  61. [70]

    Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large- Scale Generative Pre-training for Conversational Response Generation. In Proceedings of the Annual Meeting of the Association for...

    2020

  62. [71]

    Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced Language Representation with Informative Entities. In Proceedings of the Conference of the Association for Computational Linguistics . Association for Computational Linguistics, 1441–1451

    2019

  63. [72]

    Zheng Zhang, Minlie Huang, Zhongzhou Zhao, Feng Ji, Haiqing Chen, and Xiaoyan Zhu. 2019. Memory-Augmented Dialogue Management for Task-Oriented Dialogue Systems. ACM Transactions on Information Systems 37, 3 (2019), 34:1–34:30

    2019

  64. [73]

    Xiuwen Zheng, Arun Kumar, and Amarnath Gupta. 2024. Generating Cross-model Analytics Workloads Using LLMs. In Proceedings of the ACM International Conference on Information and Knowledge Management , Edoardo Serra and Francesca Spezzano (Eds.). Association for Computing Machin...

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.