Pith. sign in

REVIEW 4 major objections 5 minor 27 references

From What to Respond to When to Respond: Timely Response Generation for Open-domain Dialogue Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fine-tuned 3B dialogue model can predict response delays and generate event-timed responses, beating prompted GPT-4 on the TimelyChat benchmark.

desk verdict A genuinely new dialogue-timing task and a cleanly trained 3B model, but the ground-truth delay is just the event duration—an unvalidated proxy for human response timing that the human evaluation does not actually validate. read the letter →

arxiv 2506.14285 v1 pith:VGRJCCQX submitted 2025-06-17 cs.CL

Seongbo Jang , Minjin Jeon , Jaehoon Lee , Seonghyeon Lee , Dongha Lee , Hwanjo Yu This is my paper · ORCID
classification cs.CL
keywords timelydialogueresponsegenerationtimingpredictiontime-conditionedtemporalcommonsenseevent-drivendialoguesChatbenchmarkmulti-tasklearningagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces timely dialogue response generation, a task in which an agent decides not only what to say but when to say it, choosing a delay from the temporal context of an ongoing event. The authors build the TimelyChat benchmark, where the correct delay before the next message is the typical duration of the event being discussed, and they synthesize 55K such event-driven dialogues for training. They train TIMER, a 3B-parameter model with a multi-task objective that first predicts the time interval and then generates a response conditioned on that interval. On the benchmark, TIMER outperforms prompting-based LLMs and fine-tuned baselines on both subtasks, reaching F1 0.79 for delay prediction versus the best prompted GPT-4 score of 0.33. If the event-duration-to-delay mapping is accepted, the result shows that response timing can be learned by a small fine-tuned model.

What carries the argument

The mechanism is event-duration knowledge injected into dialogue plus a two-part training objective. The authors draw human-annotated event-duration pairs from MC-TACO for the benchmark and $ATOMIC20^{20}$ event triplets with GPT-3.5 pseudo-labeled durations for training, then instruct an LLM to write conversations in which one speaker is in the middle of an event while the other is apart. Each training turn is formatted as speaker, time interval, and utterance, with zero delay for ordinary instant turns, and the model is trained with a response-generation loss plus a time-prediction loss, $\mathcal{L} = \mathcal{L}_{\text{response}} + \lambda \mathcal{L}_{\text{time}}$, so that it learns to predict the interval before generating the reply. The benchmark's mutual-exclusivity instruction, which requires delayed and instant responses to be non-interchangeable, is what makes time-specificity a measurable property.

What would settle it

Collect timestamped real-world messaging logs in which users record what event they were doing between messages, then compare actual delays with the typical durations of those events; if the correlation is weak, or delays cluster around social rhythms such as work breaks rather than event durations, the training signal and evaluation labels lose their foundation.

Watch

Extended reading notes

Core claim

The central claim is that response timing is a learnable skill separable from response wording: a 3B model fine-tuned on event-duration knowledge can outperform much larger prompting-based LLMs at predicting whether to delay a reply and by how much, and at producing replies that only make sense after that delay. Concretely, TIMER 3B reaches F1 0.79 and false-positive rate 0.04 on delay classification, while the best prompted GPT-4 reaches F1 0.33, and TIMER also leads on time-specificity while matching the naturalness of much larger models. The authors present this as evidence that treating when to respond as a first-class task, alongside what to respond, yields more human-like open-domain dialogue agents.

Load-bearing premise

The benchmark treats the typical duration of an ongoing event as the correct delay before the next message, a mapping that is assumed in the data construction and never checked against real messaging behavior.

Editorial extensions

If this is right

  • A dialogue agent could use predicted delays to avoid repetitive instant replies and to hold a conversation over an event's natural timescale, such as waiting through a commute before reporting back.
  • Fine-tuning on event-duration knowledge transfers timing ability better than in-context prompting, suggesting that timing should be trained explicitly rather than expected from instruction-following alone.
  • The 55K synthetic event-driven dialogue set provides a template for scaling other under-annotated dialogue skills from unlabeled knowledge graphs.
  • Evaluating timing and wording separately, at turn level and dialogue level, makes it possible to tell whether a failure comes from a wrong delay or from a wrong response.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The same architecture could be extended to personalized timing by conditioning on a user's observed response rhythm, which the paper does not address.
  • Editorial inference: The mutual-exclusivity property suggests a cheap automatic check: pair each response with the wrong time interval and measure how much coherence drops, which could serve as a training signal without human labels.
  • Editorial inference: The strongest external validation would be a real messaging study comparing the model's delays with actual human response times for matched events, since the benchmark is synthetic by design.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a new task, timely dialogue response generation, in which an open-domain dialogue agent must predict the time interval before its next response and then generate a response conditioned on that interval. The authors construct the TimelyChat benchmark from MC-TACO event-duration pairs via GPT-4, and a 55K-dialogue training set from ATOMIC2020 event knowledge with GPT-3.5 pseudo-labeled durations. They train TIMER3B with a multi-task loss that jointly predicts the time interval and generates the time-conditioned response (Eq. 5). On turn-level and dialogue-level evaluations, TIMER3B outperforms prompting-based LLMs (including GPT-4) and fine-tuned baselines, and a small human evaluation is reported. The central claim is that TIMER3B generates both appropriate time intervals and responses while maintaining natural conversational flow.

Significance. If the construct validity concerns can be addressed, this is a genuinely novel task formulation that moves beyond 'what to respond' to 'when to respond,' and the released data, model, and code would be useful resources for the dialogue community. The multi-task learning setup is clean, the task decomposition in Eqs. (2)-(3) is clear, and the authors attempt both automatic and human evaluation. However, the paper's central empirical claim rests on an unvalidated equivalence between the typical duration of an ongoing event and the appropriate delay before the next message. The evidence supports only the narrower claim that TIMER3B learns the event-duration-based rule used to generate both its training data and the benchmark; it does not yet establish human-aligned timing behavior. The gap between the stated conclusion and the evidence is the main barrier to acceptance.

major comments (4)
  1. [§4.1, Appendix B.3] The ground-truth delay is equated with the typical duration of the ongoing event. In Section 4.1, each dialogue is constructed from an MC-TACO event-duration pair and the delayed response is inserted after '[duration later]'; the data-generation instructions explicitly ask the speaker to respond after the event completes. No evidence is provided that human messaging behavior follows this mapping: there are no timestamped chat logs, no human response-time measurements, and no analysis of alternative factors such as social norms, availability, or conversational closure. The evaluation metric 'delay-appropriateness' in Appendix B.3 instructs the judge to check whether the delay 'reflect[s] the typical duration of events implied in the corresponding message,' so the metric internalizes the very assumption the paper needs to validate. Consequently, the high F1 and delay-appropriateness scores in Tables 2 and Figure 3 demonstrate competence at the construction rule, not at human-aligned timing. The authors should either provide external validation (e.g., human annotations of appropriate delays on these dialogues, or comparison with real messaging logs) or explicitly scope the conclusions to the synthetic event-duration definition.
  2. [§5.1, §5.2, Eq. (4)] The training signal for the timing subtask is entirely pseudo-labeled: GPT-3.5 estimates event durations in ATOMIC2020 with no human agreement check, calibration, or quality filter beyond a 1-minute-to-24-hour range. The same duration-to-dialogue construction rule then generates the 55K training dialogues and the TimelyChat evaluation set. This means the training and evaluation distributions share the same generative pipeline, so TIMER3B's large margin over GPT-4 (F1 0.7908 vs. 0.3262 in Table 2) may reflect learning the construction artifacts rather than learning a generalizable notion of response timing. I ask the authors to (a) report human agreement on a sample of the GPT-3.5 duration pseudo-labels, (b) measure train/test event leakage between the 55K set and the 324-dialogue benchmark, and (c) test on a held-out set built with a different generation rule or from real timestamped messages to show that the result is not an artifact of the shared pipeline.
  3. [§6.3, Appendix B.3] There is a circularity risk in the automatic evaluation. The G-Eval time-specificity criteria in Appendix B.3 operationalize exactly the Mutual Exclusivity condition used to generate the dialogues in Section 4.1, and GPT-4 serves as both the generator of the dialogues and the judge of the responses. The human evaluation in Section 6.4 uses the same criteria, is conducted by only three graduate students on 90 examples, and reports no inter-annotator agreement, so it does not break the circularity. To support the claim that TIMER3B is better at 'time-specific' responses, the authors should provide an independent evaluation protocol (e.g., human judges who are not given the construction rule, or a held-out corpus of natural delayed/instant message pairs), and should report per-item human-judge agreement with the G-Eval scores.
  4. [§6.4, Tables 4 and 5] The text states that TIMER3B 'significantly outperforms' GPT-4 in delay-appropriateness and time-specificity in the dialogue-level human evaluation, but no significance test, confidence interval, or effect-size statistic is reported for the win/tie/loss rates in Table 5. With only 90 pairwise comparisons and win rates of 46% and 40%, it is not clear whether the differences exceed chance or annotation noise. The claim of 'significant outperformance' should be backed by an appropriate statistical test, and the lack of inter-annotator agreement is a further concern given that the criteria are subjective.
minor comments (5)
  1. [Eq. (1)] The notation is inconsistent: the left-hand side uses r_t but the conditional distribution is over u_t; either r_t should be defined as the response token sequence or the equation should use u_t consistently.
  2. [Throughout] The model name is written as 'TIMER3B' in most places but as 'TimeR 3B' in Figure 3; please unify.
  3. [§5.1 and Table 1] The ATOMIC2020 dataset name appears as 'ATOMIC20 20' due to a line-break artifact; it should be typeset as ATOMIC2020 throughout.
  4. [Figure 1] The caption contains a typo: 'Agent responds to use r' should be 'Agent responds to user'.
  5. [§6.3] The authors say they 'randomly sample 100 dialogues' from TimelyChat, but the benchmark has 324 dialogues; please clarify the sampling procedure and report how many dialogues were excluded because they had no delayed response.

Circularity Check

1 steps flagged · score 4.0 of 10

The 'appropriate time interval' construct is defined by the same event-duration rule used to generate the benchmark and training labels, so the timing claim is partially self-definitional.

  1. self definitional [Section 4.1 / Appendix A.1 (data construction) and Appendix B.3 (G-Eval delay-appropriateness criterion)]
    "After {B,A}'s last turn, add '[{duration} later]', where duration is the amount of time passed in real world. ... Evaluate whether the amounts of delays (if not 0 minutes) reflect the typical duration of events implied in the corresponding message."

    The benchmark's ground-truth delay is set by construction to the event's typical duration: GPT-4 is instructed to insert the delayed response after '[duration later]', where duration comes from the MC-TACO event-duration pair. The 55K training dialogues are generated with the same rule, using GPT-3.5 pseudo-labeled durations. The dialogue-level delay-appropriateness metric then scores models by exactly that rule: whether delays reflect the typical duration of events implied in the message. TIMER's high delay-appropriateness and timing F1 therefore measure how faithfully the model reproduces the generative label equation delay = event duration.

full rationale

The paper's core supervised comparisons are not statistically circular: TIMER is trained on GPT-3.5 pseudo-labeled dialogues and evaluated on held-out MC-TACO-derived dialogues with human-annotated durations, so the turn-level F1 and RMSLE numbers are genuine held-out predictions, and the time-specificity results are partially grounded by human pairwise evaluation. I found no fitted-input-called-prediction loop and no load-bearing self-citation chain. However, the benchmark's target construct, 'appropriate' response delay, is defined by the data-generation instruction that inserts the delayed turn after '[duration later]', where duration is the event's typical duration. The dialogue-level delay-appropriateness metric asks a judge to score exactly whether delays 'reflect the typical duration of events implied in the corresponding message.' Thus the central claim that TIMER produces 'appropriate time intervals' is true by construction relative to the benchmark's own label rule, and the metric cannot validate the assumed equivalence between event duration and human response delay. The G-Eval time-specificity criterion also mirrors the Mutual Exclusivity constraint used to generate the gold responses, though this is a consistent task definition rather than a derivation. Overall, the circularity is partial and confined to the definition of the timing construct; the response-generation results have independent content.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central results rest on three unproven domain assumptions: (1) event duration equals appropriate response delay, (2) LLM-generated dialogues represent real human timely conversations, and (3) GPT-4-based G-Eval is a valid judge of time-specificity. The only fitted numeric quantity is the loss weight lambda=1.0; more consequential are the 55K GPT-3.5 pseudo-labels used as training targets, which are effectively fitted values without human validation. No new physical or theoretical entities are introduced; the TimelyChat benchmark and TIMER model are artifacts of the method rather than postulated entities.

free parameters (3)
  • lambda (multi-task loss weight) = 1.0
    Equation 5 combines the response loss and time-prediction loss via L = L_response + lambda * L_time. The value 1.0 is chosen without ablation (Appendix B.1); the central results could shift if lambda is tuned.
  • GPT-3.5 event-duration pseudo-labels = 55K discrete durations (e.g., '30 minutes')
    Section 5.1: durations for ATOMIC20_20 event descriptions are estimated by GPT-3.5 and used as ground truth for training the time-interval predictor. These labels are never validated against human annotations, so the model fits its weights to whatever durations GPT-3.5 produces.
  • Duration range filter = 1 minute to 24 hours
    Both training and evaluation data exclude durations shorter than 1 minute or longer than 24 hours (Sections 4.1 and 5.1). This range choice defines the task and removes extreme labels without reported sensitivity analysis.
assumptions (3)
  • domain assumption Appropriate response delay equals the typical duration of the ongoing event
    The benchmark and training data use event-duration pairs from MC-TACO and ATOMIC20_20 as ground-truth delays (Sections 4.1 and 5.1). No evidence from real messaging data supports this mapping.
  • domain assumption LLM-generated dialogues are a valid proxy for real human timely conversations
    Both the training set and TimelyChat are synthesized by GPT-3.5/GPT-4 with author-written instructions and six seed examples (Sections 4.1, 5.1); the paper does not validate against human conversation logs.
  • domain assumption GPT-4 based G-Eval scores (naturalness, time-specificity, delay-appropriateness) are reliable
    Used for turn-level and dialogue-level metrics (Section 4.2, Appendix B.3). The human study partially validates the direction, but agreement statistics are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From What to Respond to When to Respond: Timely Response Generation for Open-domain Dialogue Agents." pith.science (2026). https://pith.science/paper/VGRJCCQX

@misc{pith2026250614285,
  author       = {Pith},
  title        = {Pith review of: From What to Respond to When to Respond: Timely Response Generation for Open-domain Dialogue Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VGRJCCQX}},
  note         = {Machine review of arXiv:2506.14285}
}
read the original abstract

While research on dialogue response generation has primarily focused on generating coherent responses conditioning on textual context, the critical question of when to respond grounded on the temporal context remains underexplored. To bridge this gap, we propose a novel task called timely dialogue response generation and introduce the TimelyChat benchmark, which evaluates the capabilities of language models to predict appropriate time intervals and generate time-conditioned responses. Additionally, we construct a large-scale training dataset by leveraging unlabeled event knowledge from a temporal commonsense knowledge graph and employing a large language model (LLM) to synthesize 55K event-driven dialogues. We then train Timer, a dialogue agent designed to proactively predict time intervals and generate timely responses that align with those intervals. Experimental results show that Timer outperforms prompting-based LLMs and other fine-tuned baselines in both turn-level and dialogue-level evaluations. We publicly release our data, model, and code.

Figures

Figures reproduced from arXiv: 2506.14285 by the authors.

Figure 1
Figure 1. An illustrative example of a timely dialogue [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall process of data construction method. Two different knowledge sources represent the same example [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Results of simulated interactions for timely [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of timely dialogue simulations with GPT-4 as a user simulator. All examples begin with the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 25 canonical work pages

  1. [1]

    Speaker {A,B} is in the middle of the event now, while speaker {B,A} is physically apart from

    work page

  2. [2]

    Do not directly mention the duration in the dialogue

    work page

  3. [3]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13153–13187, Singapore

    Character-LLM: A trainable agent for role- playing. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13153–13187, Singapore. Association for Computational Linguistics. Yuiko Tsunomori, Masakazu Ishihata, and Hiroaki Sugiyama. 2023. Time-considerable dialogue mod- els via reranking by time dependency. InFind...

    work page 2023

  4. [4]

    InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10735–10762, Singapore

    Mind the gap between conversations for im- proved long-term dialogue generation. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10735–10762, Singapore. Asso- ciation for Computational Linguistics. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I...

    work page 2023

  5. [8]

    [{duration} later]

    After {B,A}’s last turn, add "[{duration} later]", where duration is the amount of time passed in real world. 4-1. Generate {A,B}’s last message which is timely as if {A,B} spent time to finish the event. 4-2. In contrast, generate {A,B}’s last message as if {A,B} is responding instanta- neously right before the event to happen. Make sure that the timely ...

    work page 2018

  6. [9]

    0 minutes

    agent responds after the elapsed time specified in the parentheses from the previous user utterance. If the delay is "0 minutes", agent is assumed to respond immediately

    work page

  7. [10]

    user is assumed to respond to agent without any delay. Conversation: {context} 3https://huggingface.co 4https://huggingface.co/jihyoung/ rebot-generation 5https://parl.ai 6https://www.deepspeed.ai 16 B.3 G-Eval Details We elucidate the G-Eval prompts used in turn-level and dialogue-level evaluations, along with the eval- uation criteria and steps for each...

    work page

  8. [11]

    Assess the flow and coherence of the response in the conversation: Consider how seamlessly the response connects with the previous message

    work page

Show all 27 references
  1. [12]

    Evaluate the tone and style compatibility: De- termine if the response’s tone and style match those of the previous messages

  2. [13]

    Rate on a scale from 1 to 5, where 1 indicates the response is unnatural or inappropriate, and 5 indicates a perfectly natural continuation of the conversation. • (Turn-level)Time-specificity(1-5): the extent to which the response ONLY makes sense when the specified time has p...

  3. [14]

    Read the provided conversation and take note of the elapsed time since the previous message

  4. [15]

    Consider the context of the conversation, fo- cusing on how the passage of time might affect the relevance or appropriateness of the response

  5. [16]

    That is, determine if the re- sponse directly relates to or is clearly influenced by the elapsed time between the last utterance and the response

    Evaluate whether the potential response pro- vided is time-specific. That is, determine if the re- sponse directly relates to or is clearly influenced by the elapsed time between the last utterance and the response

  6. [17]

    • Coherence(1-5): the extent to which the agent maintains a good conversation flow

    Rate on a scale from 1 to 5, where 1 indi- cates the response is completely time-agnostic and unaffected by the passage of time, and 5 in- dicates the response is entirely time-specific; it only makes sense because of the amount of time that has passed since the previous messa...

  7. [18]

    Assess the flow and coherence of the agent’s responses in the conversation

  8. [19]

    Evaluate the tone and style compatibility throughout the conversation

  9. [20]

    • Delay-appropriateness(1-5): the extent to which the agent poses delays with appropriate frequency and amount

    Rate on a scale from 1 to 5, where 1 indicates the agent’s responses are incoherent or inappro- priate, and 5 indicates the agent’s responses are perfectly coherent and appropriate. • Delay-appropriateness(1-5): the extent to which the agent poses delays with appropriate frequ...

  10. [21]

    Assess whether the agent poses unnecessary or excessively frequent delays that could harm the conversation flow

  11. [22]

    Evaluate whether the amounts of delays (if not 0 minutes) reflect the typical duration of events implied in the corresponding message

  12. [23]

    Rate on a scale from 1 to 5, where 1 indicates the agent overuses and misuses delays, and 5 indicates the agent uses delays appropriately in terms of frequency and amount. • (Dialogue-level)Time-specificity(1-5): the ex- 17 tent to which the agent’s responses ONLY make sense w...

  13. [24]

    Read the provided conversation and take note of the elapsed times since the previous messages

  14. [25]

    Consider the context of the conversation, fo- cusing on how the passage of time might affect the relevance or appropriateness of the agent’s responses

  15. [26]

    That is, determine if the responses directly relate to or are clearly influenced by the elapsed times

    Evaluate whether the agent’s responses are time-specific. That is, determine if the responses directly relate to or are clearly influenced by the elapsed times

  16. [27]

    Rate on a scale from 1 to 5, where 1 indi- cates the agent’s responses are completely time- agnostic and unaffected by the passage of time, and 5 indicates the agent’s responses are entirely time-specific; they only make sense because of the amount of time that has passed sinc...

  17. [2019]

    going on a vacation

    “going on a vacation” takes longer than “go- ing for a walk”: A study of temporal commonsense understanding. InProceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (...

    2019

  18. [2023]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13584–13606, Singapore

    Conversation chronicles: Towards diverse tem- poral and relational dynamics in multi-session con- versations. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13584–13606, Singapore. Association for Computational Linguistics. Ta...

    2023 arXiv

  19. [2024]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851– 13870, Bangkok, Thailand

    Evaluating very long-term conversational memory of LLM agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851– 13870, Bangkok, Thailand. Association for Compu- tational Linguistics. Shikib Mehri, J...

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.