Pith. sign in

REVIEW 4 major objections 5 minor 84 references

Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new synthetic dataset and model let a chatbot see and hear a shared scene, talk with multiple speakers across sessions, and retrieve relevant multimodal memories.

desk verdict M3C is a genuinely useful new multimodal conversation dataset, but the paper overclaims what its evaluation shows and the retrieval benchmark needs human-verified labels. read the letter →

arxiv 2506.00421 v1 pith:MLFJ4WZJ submitted 2025-05-31 cs.CL cs.AIcs.CV

Jihyoung Jang , Minwook Bae , Minji Kim , Dilek Hakkani-Tur , Hyounghun Kim This is my paper
classification cs.CLcs.AIcs.CV
keywords multimodalconversationmulti-sessiondialoguemulti-partyaudio-visualperceptionmemoryretrievalsyntheticdatasetlong-termconversationalAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to move chatbots beyond image-centric, static dialogues into conversations where a system simultaneously sees and hears a shared scene, talks with several people at once, and carries memories across sessions. To that end, it introduces M3C, a machine-generated dataset of multi-party, multi-session dialogues in which every participant experiences the same images and sounds in real time. It also proposes a two-module model—a dialogue generator and a multimodal memory retriever—built on a vision-language backbone augmented with an audio encoder. The paper reports that human evaluators rate the dataset and the model highly on naturalness, immersion, and memorability, and that the retriever outperforms existing vision- and audio-only models at recalling relevant past moments.

What carries the argument

The load-bearing mechanism is the multimodal memory retrieval loop: at storage time, the model links each new memory to semantically related prior memories and modalities; at retrieval time, a contrastively trained encoder maps the ongoing session (text, images, and audio embeddings) and each stored memory into a common embedding space, scoring by cosine similarity and returning the top-1 memory. The audio path is added through a frozen contrastive language-audio encoder adapted by a linear layer, and the whole system is fine-tuned with LoRA on a vision-language backbone. The dataset itself is also machinery: image and audio caption corpora are grouped by LLM-assigned locations, scenarios are generated, and a filtering step checks that settings are spatially and temporally consistent and that speakers engage in real time.

What would settle it

Have three independent human annotators judge whether each retrieved memory in a random sample of 200 turns is genuinely relevant to that turn's context, and measure inter-annotator agreement. If the humans largely disagree with the model's retrieved memories while agreeing with each other, the memory-linking signal that trains the retriever is not a valid measure of conversational relevance.

Watch

Extended reading notes

Core claim

The central claim is that a conversational agent can be endowed with 'eyes and ears'—simultaneous processing of visual and auditory inputs—and still keep a coherent, long-term conversation when speakers change across sessions. The authors argue that existing multimodal dialogue work is largely image-centric, treats modalities as items to be described rather than experienced, and either ignores audio or handles memory as text only. Against that, they construct M3C, where a main speaker talks with two partners per session over three sessions, all participants share the same two real-time modality events per session, and session memories are summarized and linked across modalities. The proposed model couples a dialogue module that generates utterances, session summaries, and memory links with a retriever module that embeds the current context and past multimodal memories in a shared space and fetches the top-1 relevant memory. On human and machine evaluations, the model is reported to respond to images and sounds as shared experiences, recall events from prior sessions at natural moments, and take appropriate turns in a multi-party setting; the retriever achieves recall@1 above 92% for image and audio memory, against roughly two-thirds to three-quarters for baselines.

Load-bearing premise

The load-bearing premise is that the automatically generated memory links—created by an LLM rather than verified by humans—are reliable enough to train a retriever and a dialogue model to recall genuinely relevant past events; if those links are arbitrary or stylized, the model learns to imitate an artificial annotation function rather than to remember real conversations.

Editorial extensions

If this is right

  • A chatbot built this way can join open-domain group conversations in a shared space, reacting to what all participants see and hear rather than answering questions about a static image.
  • Because memory is stored separately and retrieved by relevance, long-term coherence no longer depends on fitting every previous turn into the model's context window.
  • The dataset construction pipeline—LLM-generated scenarios plus consistency filtering—provides a recipe for creating large-scale synthetic immersive dialogue datasets without human annotations.
  • The reported retrieval and next-speaker results suggest the approach could extend beyond three sessions, which the paper demonstrates with a fourth-session example.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the memory-link labels used for training and evaluation are generated by the same LLM family that writes the dialogues, the retriever may be learning a synthetic notion of relevance; a human-annotated relevance subset would test whether that signal transfers to real conversations.
  • The ablation compares the model with and without retrieval, but not against retrieval of unlinked memories, so the value of specifically linking memories at storage time remains untested; one could measure how much of the coherence gain comes from link structure versus simple similarity search.
  • A live test with a human in the loop—where a person shares a room with the model, both seeing a scene and hearing a sound—would be the sharpest test of whether the 'eyes and ears' experience is genuinely immersive, since all current evaluations are either fully synthetic or on pre-recorded transcripts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces M3C, a large-scale machine-generated dataset of multi-party, multi-session conversations in which all speakers share the same visual and audio stimuli, and proposes a multimodal conversation model built on Qwen2-VL-2B-Instruct with a CLAP-based audio adapter. The model consists of a dialogue module that generates responses and session memories and a retrieval module that retrieves relevant multimodal memories from previous sessions. The paper claims that human evaluations show the model engages coherently and immersively in long-term, multimodal multi-party conversations, and that the dataset and model constitute a step toward chatbots with both 'eyes and ears'.

Significance. If the central claims were fully supported, this would be a meaningful advance in multimodal conversational AI: it combines image and audio understanding with multi-party, multi-session interactions, and it releases a public dataset, code, and model. The retrieval-based memory design and the model-to-model evaluation setup are interesting, and the integration of raw audio via a CLAP adapter is a useful technical contribution. However, the current evaluation is not yet convincing. The retrieval ground truth is generated by the same LLM that wrote the dialogues, so the quantitative retrieval scores measure fit to the generator's linking style rather than human-relevant memory recall. The human evaluation lacks a baseline and has low inter-annotator agreement, and the machine evaluation substitutes audio with text captions. With substantially stronger evaluation, the work could be a valuable contribution to the field.

major comments (4)
  1. [§3.3, §5.4, Table 4] The retriever's training and evaluation labels are produced by GPT-4o-mini during dataset construction (memory linking in §3.3; prompts in Tables 18–19). The retrieval scores in Table 4 therefore measure how well the model reproduces the generator's linking decisions, not whether the retrieved memories are relevant to a human. This is a circular evaluation: the same model that wrote the dialogues also defines the ground truth for memory relevance. To support the long-term memory claim in the abstract, the paper should validate a random sample of memory links with human annotators, report human agreement with the LLM-produced links, and re-evaluate the retriever on human-verified ground truth.
  2. [§5.2, Table 2, Table 27] The human evaluation of model performance is conducted on episodes generated by the model itself, with no comparison against a baseline such as the untuned Qwen2-VL-2B-Instruct or an ablated version without the retriever. The high absolute scores in Table 2 are therefore difficult to interpret: they may reflect the model's ability to generate fluent, self-consistent synthetic dialogue rather than superior conversational competence. Moreover, the inter-annotator agreement rates in Table 27 are low (0.20–0.54, with Naturalness at 0.20), and the paper does not report a standard agreement measure such as Krippendorff's alpha. A proper baseline comparison and a more rigorous agreement analysis are needed before the human-evaluation results can support the main claims.
  3. [Table 4] All retrieval baselines in Table 4 are evaluated zero-shot, without any fine-tuning on M3C. The large gap between the proposed model (R@1 ≈ 93) and the baselines (63–72) is unsurprising, because the baselines have never seen the M3C distribution or the memory-link training signal. A fair comparison would fine-tune each baseline with the same LoRA configuration on the M3C training split, or at least clearly report zero-shot versus fine-tuned performance. In addition, no variance or significance information is provided for any retrieval number, so the reader cannot judge whether the differences are stable across random seeds or test subsets.
  4. [§5.3, Table 2] The machine evaluation substitutes audio with text captions because o3-mini cannot accept audio input. As a result, the machine evaluation does not test the model's auditory processing at all, yet its scores are used to corroborate the model's 'ears' capability. The machine scores are also uniformly higher than the human scores (Table 2), suggesting an optimistic bias. The paper should report machine–human agreement, explicitly mark the machine evaluation as not probing audio understanding, and discuss the implications of the caption substitution in the main text and the limitations section.
minor comments (5)
  1. [§5.4] There is a typo in the section heading: 'performace' should be 'performance'.
  2. [§3.2] The K-means cluster count K=30 is selected without any sensitivity analysis; the paper should report how the dataset characteristics change with K.
  3. [§5.2] The rule that a modality is inserted at a random subsequent turn if none appears by the fifth turn is an arbitrary simulation choice; its effect on the immersion and naturalness ratings should be discussed or varied in an ablation.
  4. [Appendix B, Table 13] In the scenario-generation prompt, the fields for the two partners in each session both say 'insert partner name 1', which is likely a typo for 'partner name 1' and 'partner name 2'.
  5. [§6.4, Table 7] The claim that the model 'can effectively be utilized in longer sessions' is supported only by a single anecdotal example in Table 7; a quantitative evaluation over more than three sessions would be needed to support this claim.

Circularity Check

1 steps flagged · score 6.0 of 10

Retriever gold labels are GPT-4o-mini's own memory-link/tag outputs, so Table 4 largely measures imitation of the generator's annotation policy; the long-term-memory claim is only partially independently validated.

  1. self definitional [Sections 3.3 and 5.4 (Table 4); Appendix B, Tables 18-19]
    "We employ GPT-4o mini to generate the dataset. ... We also employ memory linking to connect related elements, ensuring that linked memories are retrieved together for an enriched conversational context. After generating the conversation and memory, we perform tagging to indicate where modality elements begin in the conversation and which utterances are associated with specific memory elements. ..."

    The memory-link and utterance-memory tagging labels that define which memory is relevant at each turn are generated by GPT-4o-mini (Appendix B, Tables 18 and 19), the same model that authors the dialogues and memory summaries. These labels serve as both the contrastive-training targets for the retriever and the test-set gold labels for R@1/MRR in Table 4. Retrieval accuracy therefore reduces to agreement with GPT-4o-mini's linking/tagging choices; it does not measure relevance against independent human judgment, and the paper reports no human verification of these link labels. The central claim of seamless long-term multimodal memory retrieval is thus supported by a metric whose ground truth is generated by the paper's own data-construction pipeline.

full rationale

The only substantive circularity is in the retrieval evaluation. GPT-4o-mini creates the memory links and utterance-memory tags during dataset construction, and those same machine-generated annotations are used as the targets for training and as the gold labels for the retrieval numbers. Consequently, the high retrieval scores in Table 4 can be read as measuring how well the retriever reproduces the generator's annotation style rather than how well it finds memories a human would deem relevant. The paper's other evidence—human ratings of dataset quality and of model self-chat episodes (Table 2), and comparative machine evaluations (Table 3)—is procedurally independent, although the model-performance human scores assess overall naturalness, immersion, and memorability rather than the correctness of specific retrieved memories, and inter-annotator agreement is low (0.20-0.54 in Table 27). Self-citations to Jang et al. (2024) for dataset design and memory-linking are present but not load-bearing; they do not forbid alternatives or supply the retrieval ground truth. Overall, one central quantitative claim is partially circular, while the dataset contribution and human-validated conversational fluency retain independent content, so a score of 6 is appropriate.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the assumption that machine-generated dialogues and memory links are valid proxies for real human multimodal conversation, and that the evaluation instruments (human raters and LLM judges) faithfully measure quality. The dataset construction and retrieval labels are produced by GPT-4o-mini, which is also the source of the training signal for the model, creating a circular dependence.

free parameters (2)
  • K-means cluster count K = 30
    Section 3.2: K=30 is chosen for grouping images and audio by location tags; no tuning procedure or sensitivity analysis is given, and it affects which modalities co-occur in a session.
  • Modality insertion threshold = 5 turns
    Section 5.2: in model-only evaluation, if the main speaker has not introduced a modality by the fifth turn, a modality is inserted at a random later turn; this hand-chosen rule directly affects the measured 'immersion' scores.
assumptions (5)
  • domain assumption GPT-4o-mini generated conversations and memory summaries are faithful to natural human dialogue.
    The dataset is entirely LLM-generated (Section 3.3); no comparison to human-human conversation is provided, and human ratings are only on 300 episodes.
  • domain assumption GPT-4o-mini memory link labels are correct relevance judgments.
    The retriever is trained and evaluated against these labels (Section 4.2, Table 4); there is no human verification of memory relevance.
  • domain assumption CLAP with a linear adapter provides sufficient audio understanding.
    The audio adapter is a frozen CLAP plus linear layer (Appendix D.1); the paper does not evaluate which acoustic details are lost.
  • domain assumption Human evaluator agreement is sufficient for validation.
    Table 27 reports agreement rates 0.20-0.54, which are commonly considered low; the paper nonetheless uses the mean scores as primary evidence.
  • domain assumption COCO, AudioCaps, and Clotho cover realistic shared experiences.
    These source datasets are not collected for multi-party dialogue; the paper does not justify their representativeness for the target setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions." pith.science (2026). https://pith.science/paper/MLFJ4WZJ

@misc{pith2026250600421,
  author       = {Pith},
  title        = {Pith review of: Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MLFJ4WZJ}},
  note         = {Machine review of arXiv:2506.00421}
}
abstract

As chatbots continue to evolve toward human-like, real-world, interactions, multimodality remains an active area of research and exploration. So far, efforts to integrate multimodality into chatbots have primarily focused on image-centric tasks, such as visual dialogue and image-based instructions, placing emphasis on the "eyes" of human perception while neglecting the "ears", namely auditory aspects. Moreover, these studies often center around static interactions that focus on discussing the modality rather than naturally incorporating it into the conversation, which limits the richness of simultaneous, dynamic engagement. Furthermore, while multimodality has been explored in multi-party and multi-session conversations, task-specific constraints have hindered its seamless integration into dynamic, natural conversations. To address these challenges, this study aims to equip chatbots with "eyes and ears" capable of more immersive interactions with humans. As part of this effort, we introduce a new multimodal conversation dataset, Multimodal Multi-Session Multi-Party Conversation ($M^3C$), and propose a novel multimodal conversation model featuring multimodal memory retrieval. Our model, trained on the $M^3C$, demonstrates the ability to seamlessly engage in long-term conversations with multiple speakers in complex, real-world-like settings, effectively processing visual and auditory inputs to understand and respond appropriately. Human evaluations highlight the model's strong performance in maintaining coherent and dynamic interactions, demonstrating its potential for advanced multimodal conversational agents.

Figures

Figures reproduced from arXiv: 2506.00421 by the authors.

Figure 1
Figure 1. The main speaker (Alex) engages in conversation with two different partners per session, where all [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of our model. In Session N- [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the retrieval module architecture, [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 76 canonical work pages

  1. [1]

    Images are directly provided as input without modification

    Session Utterance The model is trained to process session utter- ances. Images are directly provided as input without modification. However, audio is con- verted into captions and represented in the for- mat <start_audio> caption <end_audio>. During this stage, the model learns using only the text-based representation of audio

    work page

  2. [2]

    In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13584–13606, Singapore

    Conversation chronicles: Towards diverse tem- poral and relational dynamics in multi-session con- versations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13584–13606, Singapore. Association for Computational Linguistics. Jihyoung Jang, Taeyoung Kim, and Hyounghun Kim

    2023

  3. [3]

    Also, it learns to establish memory links between ses- sion memory and observed images/audio, as well as between session memory and chat his- tory

    Summarization & Memory Linking The model is trained to generate session sum- maries to create structured memory. Also, it learns to establish memory links between ses- sion memory and observed images/audio, as well as between session memory and chat his- tory

    work page

  4. [4]

    In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , pages 796– 806

    Doing personal laps: Llm-augmented dialogue construction for personalized multi-session conver- sational search. In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval , pages 796– 806. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. AudioCaps: Generating cap- tions for a...

    2019

  5. [5]

    Advances in Neural Information Processing Systems, 36

    Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36. Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. Audio flamingo: A novel audio language model with few- shot learning and dialogue abilities. In Forty-first International Conference on Machine Learning. Satw...

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2409.12524

    Should rag chatbots forget unimportant conversations? exploring importance and forget- ting with psychological insights. arXiv preprint arXiv:2409.12524. Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yam- ing Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng, and Daxin Jiang. 2022. Multimodal dialogue response generation. In Proceedings of the 60th An- nu...

    arXiv 2022

  7. [7]

    In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing , pages 6268– 6278, Singapore

    Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing , pages 6268– 6278, Singapore. Association for Computational Lin- guistics. Seiichi Yamamoto, Keiko Taguchi, Koki Ijuin, Ichiro Umata, and Masafumi Nishida. 2015. Multimodal corpu...

    2023

  8. [8]

    for our next scene

    Mind the gap between conversations for im- proved long-term dialogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10735–10762, Singapore. Asso- ciation for Computational Linguistics. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: ...

    2024

Show all 84 references
  1. [9]

    When summarizing {MAIN SPEAKER NAME}’s memory about themselves, start the memory entry with ”I ”

  2. [10]

    It decides when to take the role of the main speaker by pro- ducing a [YES] or [NO] token

    Main Speaker & Memory Retrieval The model is trained when to output turn to- kens and retrieval tokens. It decides when to take the role of the main speaker by pro- ducing a [YES] or [NO] token. Similarly, it learns the timing for memory retrieval us- ing tokens like [RET_IMG]...

  3. [11]

    (about all)

    Do not use group references like “(about all)” or combine multiple people into one memory entry

  4. [12]

    Audio Understanding via CLAP In the final stage, we enhance the model with a frozen CLAP, using a linear layer adapter to enable audio comprehension. Instead of using only captions, the model now receives audio embeddings between <start_audio> and <end_audio>, allowing for dir...

  5. [13]

    ###Response: Table 17: Prompt for memory generation

    Avoid adding extra explanations, introductions, or conclusions. ###Response: Table 17: Prompt for memory generation. ###First session setting: 1 - {CAPTION} 2 - {CAPTION} ###Second session setting: 3 - {CAPTION} 4 - {CAPTION} ###{MAIN SPEAKER NAME}’s memory from first session:...

  6. [14]

    Instead, audio is replaced by its corresponding captions, and only the session’s chat history and images are included for tuning

    Text & Image-Based Retrieval In the first stage, the model is trained without direct audio input. Instead, audio is replaced by its corresponding captions, and only the session’s chat history and images are included for tuning. This allows the model to focus on retrieving memo...

  7. [15]

    In this image

    Audio-Enhanced Retrieval In the second stage, similar to the Dialogue Module, we integrate the CLAP with a linear adapter to enable direct audio comprehension. By incorporating audio embeddings, the re- AUDIO Memory Retrieval Lora Memory Retrieval Image Encoder Image Audio Enc...

  8. [16]

    Given an image caption, identify the location using a single, general word

  9. [18]

    If the specific location is unclear, make an edu- cated guess based on where the described elements are typically found

  10. [19]

    kitchen”. Similarly, a desk, computer, and books might suggest “office

    For example, if objects like a refrigerator, stove, and cooking utensils are visible, the location would be “kitchen”. Similarly, a desk, computer, and books might suggest “office”

  11. [20]

    ###Caption: {CAPTION} ###Answer: For Audio ###Instruction:

    Avoid adding any additional explanations, intro- ductions, or conclusions. ###Caption: {CAPTION} ###Answer: For Audio ###Instruction:

  12. [21]

    Given an audio caption, identify the location using a single, general word

  13. [22]

    Ensure the location is broadly applicable and captures the essence of the description

  14. [23]

    If the specific location is unclear, output “none”

  15. [24]

    If the sounds are fictional, unrealistic, or not commonly associated with real-world locations, output “none”

  16. [25]

    For example, sounds like sizzling, a refrigerator hum, and chopping might suggest “kitchen”

  17. [26]

    ###Caption: {CAPTION} ###Answer: Table 12: Prompt for specifying location based on cap- tion

    Avoid adding any additional explanations, intro- ductions, or conclusions. ###Caption: {CAPTION} ###Answer: Table 12: Prompt for specifying location based on cap- tion. ###Instruction:

  18. [27]

    Each session will feature the main speaker alongside two different partners

    Four speakers are involved in a conversation episode, consisting of one main speaker and three partners. Each session will feature the main speaker alongside two different partners

  19. [28]

    In each session, the main speaker and two different partners will witness two settings unfolding in real-time right in front of them

    There will be three continuous sessions. In each session, the main speaker and two different partners will witness two settings unfolding in real-time right in front of them. These settings are live observations, experienced not only visually but also audibly, and are not base...

  20. [29]

    Ensure that no setting is repeated across the sessions

    From a provided list of settings, select two unique settings for each session. Ensure that no setting is repeated across the sessions

  21. [30]

    a few hours later,

    There will be a time gap between each session. Choose suitable time intervals from the following options: “a few hours later,” “a few days later”, “a few weeks later”, “a few months later”, and “a couple of years later”

  22. [31]

    Clearly define the names and relationships of all speakers to provide context for their interactions and to enhance the flow of conversation

  23. [32]

    For instance, if the setting involves cooking, one might inquire about the dish being prepared

    During each session, the speakers should engage in discussions that logically connect to the context of the settings they observe, without directly referencing specific details of those settings. For instance, if the setting involves cooking, one might inquire about the dish b...

  24. [33]

    Ensure that the combination of settings in each session does not include contradictory elements, such as differing weather conditions or inconsistent times of day that would not logically coexist

  25. [34]

    Please do not generate any other opening, closing, and explanations. ###Setting list: - {MODALITY LIST} ###Response: - Main speaker name: {insert name} - Main speaker relationship: {insert relationship} - Partner 1 name: {insert name} - Partner 1 relationship: {insert relation...

  26. [35]

    Two captions are provided, each describing either an image or an audio clip

  27. [36]

    Determine whether the two captions are aligned and compatible

  28. [37]

    Assume that the speakers are observing or lis- tening to the described image or audio in real-time within the same context

  29. [38]

    no” Otherwise, output “yes

    If the two captions cannot coexist within the same context, output “no” Otherwise, output “yes”

  30. [39]

    For example, if one caption describes a ski resort and the other describes a meadow, the two captions cannot coexist, and the output should be “no”

  31. [40]

    [{PARTNER 1 NAME}]

    Please do not generate any other opening, clos- ing, and explanations. ###Caption: - {CAPTION A} - {CAPTION B} ###Answer: Table 14: Prompt for checking modality alignment within a session. ###First session conversation: {FIRST SESSION} ###Second session conversation: {SECOND S...

  32. [41]

    Two settings and a list of dialogue utterances are provided

  33. [42]

    Based on the dialogue, identify the utterance number where each of the two settings is first men- tioned, seen, or heard by the speakers in real-time, including both visual and auditory elements

  34. [43]

    Select only one utterance number for each set- ting

  35. [44]

    ###Settings: - {CAPTION A} - {CAPTION B} ###Utterances: {UTTERANCE LIST} ###Response: Table 16: Prompt for modality tagging

    Please do not generate any other opening, clos- ing, and explanations. ###Settings: - {CAPTION A} - {CAPTION B} ###Utterances: {UTTERANCE LIST} ###Response: Table 16: Prompt for modality tagging. ###Second session conversation: - First Setting: {FIRST SETTING} - Second Setting...

  36. [45]

    The summary should focus on what {MAIN SPEAKER NAME} needs to remember for the next conversation

    Please summarize the conversation from {MAIN SPEAKER NAME}’s perspective. The summary should focus on what {MAIN SPEAKER NAME} needs to remember for the next conversation

  37. [46]

    Exclude general descriptions or background information about the setting

    The summary should include emotions, thoughts, facts, and commitments expressed during the conversation, but only those explicitly revealed during the discussion. Exclude general descriptions or background information about the setting

  38. [47]

    Only include information shared during the conversation

    Do not include content from the setting descriptions in the memory. Only include information shared during the conversation

  39. [48]

    Each memory entry must focus on one person only, without combining multiple speakers into one sentence or perspective

    Ensure the memory entries are concise and focus only on unique, relevant information necessary for the next session. Each memory entry must focus on one person only, without combining multiple speakers into one sentence or perspective

  40. [49]

    /”. For example, “{{sentence 1}} / {{sentence 2}}

    Each memory entry should be a separate sentence or key idea, but avoid summarizing every single statement unless it is crucial for the next session. All memory entries must be separated by a “/”. For example, “{{sentence 1}} / {{sentence 2}} ...”

  41. [50]

    The memory must be presented from {MAIN SPEAKER NAME}’s perspective, focusing on summa- rizing the overall key points or themes rather than individual conversational details

  42. [51]

    Consolidate similar points into a single entry if possible

    Avoid duplicating information already included in the summary. Consolidate similar points into a single entry if possible

  43. [52]

    If a memory element updates or replaces existing information, ensure the updated version reflects the latest understanding and avoid redundancy

  44. [54]

    (about {{name}}) . For example, “{{Memory Sentence}} (about {{name}})

    At the end of each memory sentence, specify who the memory is about from {MAIN SPEAKER NAME}’s perspective using the format “(about {{name}}) . For example, “{{Memory Sentence}} (about {{name}})”. Ensure that the parentheses end with a period to complete the sentence

  45. [56]

    no memory

    If there are no memories to summarize, output “no memory”

  46. [58]

    Summarized conversation memories from the perspective of {MAIN SPEAKER NAME} and the settings used in the dialogue are provided

  47. [59]

    {MAIN SPEAKER NAME} references the set- tings and memories from previous sessions during subsequent conversations to ensure seamless con- tinuity in the dialogue

  48. [60]

    When referring to memories, the goal is to pro- vide rich context by connecting related elements

  49. [61]

    If the provided settings and memories pertain to the same context or reflect subsequent updates, they should be connected

  50. [62]

    {{NUMBER}}-{{NUMBER}}

    The format for expressing connections is “{{NUMBER}}-{{NUMBER}}”and should be output on separate lines

  51. [63]

    When multiple memories are connected, write each connection on a new line

    Each {{NUMBER}} should contain only a sin- gle digit. When multiple memories are connected, write each connection on a new line

  52. [64]

    ###Response: Table 18: Prompt for memory linking

    Avoid adding extra explanations, introductions, or conclusions. ###Response: Table 18: Prompt for memory linking. ###Third session conversation: {SESSION CONVERSATION} ###{MAIN SPEAKER NAME}’s memory: {MEMORY LIST} ###Instruction:

  53. [65]

    {MAIN SPEAKER NAME} engages in a third session conversation based on their memory

  54. [66]

    The utterances by {MAIN SPEAKER NAME} are informed by their referenced memory

  55. [67]

    During the conversation, if any part of {MAIN SPEAKER NAME}’s utterances relies on memory, that part must be specifically identified

  56. [68]

    The memory consists of two components: (1) Summarized information from previous session conversations, and (2)Real-time observations by {MAIN SPEAKER NAME} during prior sessions

  57. [69]

    Utterance Letter-Memory Number

    The output format should follow the structure: “Utterance Letter-Memory Number” (e.g., “A-3”)

  58. [70]

    Utterance Letter-Memory Number

    Each line should contain a single “Utterance Letter-Memory Number” entry, separated by a newline

  59. [71]

    If an utterance references multiple memory entries, each reference must be listed on a new line rather than combining memory numbers on the same line

  60. [72]

    If the same information needs to be referenced, prioritize using the memory from the most recent session

  61. [73]

    If no memory is referenced and there is nothing to output, return “none”

  62. [74]

    ###Response: Table 19: Prompt for memory tagging

    Avoid adding extra explanations, introductions, or conclusions. ###Response: Table 19: Prompt for memory tagging. ###Conversation episode: * First session: {FIRST SESSION} * Second session: {SECOND SESSION} * Third session: {THIRD SESSION} ###Instruction: - Each conversation e...

  63. [75]

    (Yes or No)

    Is there complete consistency between the environmental, spatial, and temporal features of the settings within the session? For example, it would be contradictory if one setting depicts daytime while the other depicts nighttime, or if spatial features (e.g., location or layout...

  64. [76]

    (Yes or No)

    Do all sessions within the episode maintain a plausible continuity in time, space, and context? For example, any stated time intervals or implied transitions between settings should be logical and coherent. (Yes or No)

  65. [77]

    for our next scene

    Are all participants depicted as fully engaging with the setting in real time? References to past or future events should not imply detachment from the present interaction (e.g., avoiding phrases like “for our next scene” or references to reviewing recorded footage). (Yes or No)

  66. [78]

    (Yes or No)

    Are all settings within the session entirely realistic? Any elements that seem exaggerated, cartoonish, or overly stylized for natural conversation or interaction should be avoided. (Yes or No)

  67. [79]

    (Yes or No)

    Is each setting fully utilized and referenced in the conversation? All settings presented within the session must have a clear role in the dialogue or interaction, without any being neglected. (Yes or No)

  68. [80]

    (Yes or No) ###Response: 1

    Do all spoken lines reflect the tone and context of natural, real-time interaction? For instance, lines should avoid referring to the setting or events in a way that suggests they are pre-recorded, staged, or viewed from an external perspective. (Yes or No) ###Response: 1. 2. ...

  69. [81]

    (from first session, about me)

    I enjoy watching snowboarders and skiers perform tricks in the snow. (from first session, about me)

  70. [82]

    (from first session, about Jamie)

    Jamie suggested recording some stunts, and I think it would be fun to have footage to remember this day. (from first session, about Jamie)

  71. [83]

    (from first session, about Sam)

    Sam is interested in capturing wipeouts, which adds an entertaining element to our video. (from first session, about Sam)

  72. [84]

    (from first session, about me)

    I believe that each sport has its own unique style and that blending both could create something interesting. (from first session, about me)

  73. [85]

    (from first session, about me) Table 24: An example of a memory summarized from the perspective of the main speaker (Alex) in a conversation

    It’s important to cheer on the performers while filming, as it creates a more exciting atmosphere. (from first session, about me) Table 24: An example of a memory summarized from the perspective of the main speaker (Alex) in a conversation. The summarized memory is combined wi...

    2025

  74. [2022]

    We use early stopping with a maximum of 3 epochs

    to fine-tune Qwen2-VL-2B-Instruct, train- ing with a cross-entropy loss, a maximum input length of 1024, and a learning rate of1 × 10−3. We use early stopping with a maximum of 3 epochs. For LoRA configuration, we set r = 8, α = 16, dropout=0.05, and apply it to the q_proj and...

    2024

  75. [2023]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 18392–18402

    Instructpix2pix: Learning to follow image edit- ing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 18392–18402. Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec...

    2006 arXiv

  76. [2024]

    In Findings of the Association for Com- putational Linguistics: EMNLP 2024, pages 11786– 11815, Miami, Florida, USA

    Mixed-session conversation with egocentric memory. In Findings of the Association for Com- putational Linguistics: EMNLP 2024, pages 11786– 11815, Miami, Florida, USA. Association for Com- putational Linguistics. Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.