{"id":"a0ddfb84-b879-4115-8104-4fa0c55e28a3","arxiv_id":"2506.14285","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A dialogue agent trained on 55K synthetic dialogues predicts when a response should be delayed and generates replies that only make sense after that delay.","lead":"This paper proposes that chatbots should decide when to respond, not just what to say. The authors release a benchmark, a large synthetic dialogue dataset, and a model that predicts response delays and generates time-conditioned replies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's ground-truth delay is defined as the ongoing event's duration, but this mapping is never validated against human response-timing behavior, so the central superiority claim rests on an unvalidated construct.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the benchmark's definition of an appropriate delay as the typical duration of an ongoing event is never validated against human response-timing behavior. This concern is load-bearing because every headline result—TIMER3B's high F1, low RMSLE, high delay-appropriateness, and high time-specificity—is measured against labels generated from this equation. If the equation is false, the model is being rewarded for reproducing the authors' synthetic data-generation rule, not for human-like timing. The paper does have independent support: it releases data and code, includes a human evaluation, and its internal consistency is good. However, the human evaluation only validates relative model ordering under the same criterion; it cannot validate the criterion itself. The reader's conditional verdict is appropriate: the contribution is real as a synthetic benchmark and training recipe, but the central claim about 'appropriate' timing should be accepted only after the construct is validated. Therefore my stress-test does not move the verdict; it reinforces the same condition. I would add one clarifier: the abstract's claim of overall superiority is too strong even under the benchmark, since the human evaluation shows TIMER3B is less natural and less coherent than GPT-4, but that is a presentation issue rather than a separate load-bearing flaw.","tokens_in":17666,"tokens_out":2900,"duration_ms":35103,"concrete_test":"Take 100 randomly sampled turns from TimelyChat. Strip away the ground-truth interval and the delayed reference response; present independent human raters with only the dialogue history and a short description of the messaging scenario. Ask each rater: 'After the last message, how long should this speaker wait before replying?' with options including instant, 10 minutes, 30 minutes, 1 hour, 2 hours, 6 hours, 12 hours, and 1 day. Compute agreement between the raters' modal delay and the benchmark's event-duration label (exact match and within a factor of 2). As a complementary check, locate or collect a timestamped messaging corpus (e.g., SMS or WhatsApp logs) where an ongoing event is mentioned, and test whether the duration of the mentioned event correlates with the actual next-message delay.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that TIMER3B generates appropriate time intervals and responses—depends on TimelyChat's ground-truth delays being genuinely appropriate. In Section 4.1, each dialogue is constructed from an MC-TACO event-duration pair, and the delayed response is explicitly inserted after '[duration later]'. Thus the benchmark equates the typical duration of the ongoing event with the correct delay before the next message. Section 5.1 extends this same assumption to the 55K-dialogue training set, where GPT-3.5 pseudo-labels event durations and the same construction rule generates training targets. The paper offers no evidence that this equivalence holds for human messaging: no timestamped chat logs, no human response-time data, and no validation that social norms, availability, or conversational closure are secondary to event duration. The data-generation constraints (Spatial Separation, Mutual Exclusivity, Temporal Implicitness) ensure that the delayed and instant responses differ, but they do not establish that the chosen duration is the natural delay. Consequently, the evaluation metrics—delay-appropriateness, time-specificity, and even the F1 timing prediction—are all defined relative to this unvalidated mapping. TIMER3B may simply be learning the generative rule used to create both its training and evaluation data, rather than a human-aligned sense of when to respond. The human evaluation in Section 6.4 compares TIMER3B against GPT-4 using the same event-duration criterion, so it cannot validate the mapping itself. If the mapping is wrong, the headline result reduces to 'TIMER3B best reproduces the authors' synthetic construction rule'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, timely dialogue response generation, in which an open-domain dialogue agent must predict the time interval before its next response and then generate a response conditioned on that interval. The authors construct the TimelyChat benchmark from MC-TACO event-duration pairs via GPT-4, and a 55K-dialogue training set from ATOMIC2020 event knowledge with GPT-3.5 pseudo-labeled durations. They train TIMER3B with a multi-task loss that jointly predicts the time interval and generates the time-conditioned response (Eq. 5). On turn-level and dialogue-level evaluations, TIMER3B outperforms prompting-based LLMs (including GPT-4) and fine-tuned baselines, and a small human evaluation is reported. The central claim is that TIMER3B generates both appropriate time intervals and responses while maintaining natural conversational flow.","tokens_in":17992,"tokens_out":4663,"duration_ms":52234,"significance":"If the construct validity concerns can be addressed, this is a genuinely novel task formulation that moves beyond 'what to respond' to 'when to respond,' and the released data, model, and code would be useful resources for the dialogue community. The multi-task learning setup is clean, the task decomposition in Eqs. (2)-(3) is clear, and the authors attempt both automatic and human evaluation. However, the paper's central empirical claim rests on an unvalidated equivalence between the typical duration of an ongoing event and the appropriate delay before the next message. The evidence supports only the narrower claim that TIMER3B learns the event-duration-based rule used to generate both its training data and the benchmark; it does not yet establish human-aligned timing behavior. The gap between the stated conclusion and the evidence is the main barrier to acceptance.","major_comments":[{"comment":"The ground-truth delay is equated with the typical duration of the ongoing event. In Section 4.1, each dialogue is constructed from an MC-TACO event-duration pair and the delayed response is inserted after '[duration later]'; the data-generation instructions explicitly ask the speaker to respond after the event completes. No evidence is provided that human messaging behavior follows this mapping: there are no timestamped chat logs, no human response-time measurements, and no analysis of alternative factors such as social norms, availability, or conversational closure. The evaluation metric 'delay-appropriateness' in Appendix B.3 instructs the judge to check whether the delay 'reflect[s] the typical duration of events implied in the corresponding message,' so the metric internalizes the very assumption the paper needs to validate. Consequently, the high F1 and delay-appropriateness scores in Tables 2 and Figure 3 demonstrate competence at the construction rule, not at human-aligned timing. The authors should either provide external validation (e.g., human annotations of appropriate delays on these dialogues, or comparison with real messaging logs) or explicitly scope the conclusions to the synthetic event-duration definition.","section":"§4.1, Appendix B.3"},{"comment":"The training signal for the timing subtask is entirely pseudo-labeled: GPT-3.5 estimates event durations in ATOMIC2020 with no human agreement check, calibration, or quality filter beyond a 1-minute-to-24-hour range. The same duration-to-dialogue construction rule then generates the 55K training dialogues and the TimelyChat evaluation set. This means the training and evaluation distributions share the same generative pipeline, so TIMER3B's large margin over GPT-4 (F1 0.7908 vs. 0.3262 in Table 2) may reflect learning the construction artifacts rather than learning a generalizable notion of response timing. I ask the authors to (a) report human agreement on a sample of the GPT-3.5 duration pseudo-labels, (b) measure train/test event leakage between the 55K set and the 324-dialogue benchmark, and (c) test on a held-out set built with a different generation rule or from real timestamped messages to show that the result is not an artifact of the shared pipeline.","section":"§5.1, §5.2, Eq. (4)"},{"comment":"There is a circularity risk in the automatic evaluation. The G-Eval time-specificity criteria in Appendix B.3 operationalize exactly the Mutual Exclusivity condition used to generate the dialogues in Section 4.1, and GPT-4 serves as both the generator of the dialogues and the judge of the responses. The human evaluation in Section 6.4 uses the same criteria, is conducted by only three graduate students on 90 examples, and reports no inter-annotator agreement, so it does not break the circularity. To support the claim that TIMER3B is better at 'time-specific' responses, the authors should provide an independent evaluation protocol (e.g., human judges who are not given the construction rule, or a held-out corpus of natural delayed/instant message pairs), and should report per-item human-judge agreement with the G-Eval scores.","section":"§6.3, Appendix B.3"},{"comment":"The text states that TIMER3B 'significantly outperforms' GPT-4 in delay-appropriateness and time-specificity in the dialogue-level human evaluation, but no significance test, confidence interval, or effect-size statistic is reported for the win/tie/loss rates in Table 5. With only 90 pairwise comparisons and win rates of 46% and 40%, it is not clear whether the differences exceed chance or annotation noise. The claim of 'significant outperformance' should be backed by an appropriate statistical test, and the lack of inter-annotator agreement is a further concern given that the criteria are subjective.","section":"§6.4, Tables 4 and 5"}],"minor_comments":[{"comment":"The notation is inconsistent: the left-hand side uses r_t but the conditional distribution is over u_t; either r_t should be defined as the response token sequence or the equation should use u_t consistently.","section":"Eq. (1)"},{"comment":"The model name is written as 'TIMER3B' in most places but as 'TimeR 3B' in Figure 3; please unify.","section":"Throughout"},{"comment":"The ATOMIC2020 dataset name appears as 'ATOMIC20\n20' due to a line-break artifact; it should be typeset as ATOMIC2020 throughout.","section":"§5.1 and Table 1"},{"comment":"The caption contains a typo: 'Agent responds to use r' should be 'Agent responds to user'.","section":"Figure 1"},{"comment":"The authors say they 'randomly sample 100 dialogues' from TimelyChat, but the benchmark has 324 dialogues; please clarify the sampling procedure and report how many dialogues were excluded because they had no delayed response.","section":"§6.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution—a new task and benchmark for response timing—is interesting and likely of value to the dialogue community, but the empirical claims currently overreach the evidence. The main issue is not experimental sloppiness but construct validity: the benchmark equates event duration with response delay, and the training and evaluation data share the same generative pipeline. I would not reject the paper on novelty grounds, but the authors should be asked to either validate the event-duration-to-delay assumption with human data or substantially soften the claims to the synthetic setting. The small human evaluation also needs proper statistical reporting. The paper fits the journal's scope well if these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo quick things to know about this one.\n\nThe task is genuinely new. No one in the long-term dialogue line (MSC, CC, LoCoMo, GapChat) asks the model to predict the delay before the next message. GapChat uses predetermined intervals; here the agent has to decide when and then generate a response conditioned on that time. The TimelyChat evaluation set and the 55K synthetic training set are real resources, shipped with code and model. The internal experiments look clean: TIMER3B reaches F1 0.79 on delay classification versus 0.26 for GPT-4, and the dialogue-level human evaluation, small as it is (three annotators), tracks the automatic metrics on the timing axes. That is honest progress.\n\nThe soft spot is the central construct. The benchmark defines the correct delay as the typical duration of the ongoing event: a six-hour date means a six-hour silence. That assumption is baked into Section 4.1 (the delayed response is inserted after '[duration later]') and Section 5.1 (GPT-3.5 estimates event durations for training labels). It is never validated against timestamped messaging logs or any human response-timing data. The human evaluation in Section 6.4 uses the same event-duration criterion, so it cannot validate the mapping either. If reply timing is governed by social norms, availability, or conversational closure rather than event duration, the ground-truth labels are off, and the headline result reduces to 'TIMER reproduces the synthetic construction rule.' That is the load-bearing risk.\n\nTwo secondary issues. Training duration labels come from GPT-3.5 without a human agreement check, so label noise is unknown. And the G-Eval time-specificity metric operationalizes the same mutual exclusivity condition used to generate the dialogues, with GPT-4 both generating and judging—a mild circularity, not disqualifying. The abstract also overstates: in the human evaluation TIMER is less natural and less coherent than GPT-4; the wins are specifically in time-specificity and delay appropriateness.\n\nWho should read this: anyone working on dialogue timing or temporal reasoning in chat systems. It deserves a serious referee. The fixes are manageable: validate the delay-duration mapping with real chat data or a rating study, add human agreement checks, soften the abstract. I'd send it to review with those requests, not desk-reject it.","headline":"A genuinely new dialogue-timing task and a cleanly trained 3B model, but the ground-truth delay is just the event duration—an unvalidated proxy for human response timing that the human evaluation does not actually validate.","tokens_in":18528,"tokens_out":3146,"would_cite":true,"duration_ms":31491,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned 3B dialogue model can predict response delays and generate event-timed responses, beating prompted GPT-4 on the TimelyChat benchmark.","keywords":["timely dialogue response generation","response timing prediction","time-conditioned response generation","temporal commonsense","event-driven dialogues","TimelyChat benchmark","multi-task learning","dialogue agents"],"falsifier":"Collect timestamped real-world messaging logs in which users record what event they were doing between messages, then compare actual delays with the typical durations of those events; if the correlation is weak, or delays cluster around social rhythms such as work breaks rather than event durations, the training signal and evaluation labels lose their foundation.","tokens_in":17397,"feed_emoji":"⏰","tokens_out":8021,"duration_ms":68312,"temperature":0.7,"pith_summary":"This paper introduces timely dialogue response generation, a task in which an agent decides not only what to say but when to say it, choosing a delay from the temporal context of an ongoing event. The authors build the TimelyChat benchmark, where the correct delay before the next message is the typical duration of the event being discussed, and they synthesize 55K such event-driven dialogues for training. They train TIMER, a 3B-parameter model with a multi-task objective that first predicts the time interval and then generates a response conditioned on that interval. On the benchmark, TIMER outperforms prompting-based LLMs and fine-tuned baselines on both subtasks, reaching F1 0.79 for delay prediction versus the best prompted GPT-4 score of 0.33. If the event-duration-to-delay mapping is accepted, the result shows that response timing can be learned by a small fine-tuned model.","feed_headline":"3B model beats GPT-4 at choosing reply timing","feed_subtitle":"Fine-tuned on event-duration knowledge, TIMER scores F1 0.79 versus 0.33 for the best prompted GPT-4 baseline.","key_machinery":"The mechanism is event-duration knowledge injected into dialogue plus a two-part training objective. The authors draw human-annotated event-duration pairs from MC-TACO for the benchmark and $ATOMIC20^{20}$ event triplets with GPT-3.5 pseudo-labeled durations for training, then instruct an LLM to write conversations in which one speaker is in the middle of an event while the other is apart. Each training turn is formatted as speaker, time interval, and utterance, with zero delay for ordinary instant turns, and the model is trained with a response-generation loss plus a time-prediction loss, $\\mathcal{L} = \\mathcal{L}_{\\text{response}} + \\lambda \\mathcal{L}_{\\text{time}}$, so that it learns to predict the interval before generating the reply. The benchmark's mutual-exclusivity instruction, which requires delayed and instant responses to be non-interchangeable, is what makes time-specificity a measurable property.","core_discovery":"The central claim is that response timing is a learnable skill separable from response wording: a 3B model fine-tuned on event-duration knowledge can outperform much larger prompting-based LLMs at predicting whether to delay a reply and by how much, and at producing replies that only make sense after that delay. Concretely, TIMER 3B reaches F1 0.79 and false-positive rate 0.04 on delay classification, while the best prompted GPT-4 reaches F1 0.33, and TIMER also leads on time-specificity while matching the naturalness of much larger models. The authors present this as evidence that treating when to respond as a first-class task, alongside what to respond, yields more human-like open-domain dialogue agents.","pith_inferences":["Editorial inference: The same architecture could be extended to personalized timing by conditioning on a user's observed response rhythm, which the paper does not address.","Editorial inference: The mutual-exclusivity property suggests a cheap automatic check: pair each response with the wrong time interval and measure how much coherence drops, which could serve as a training signal without human labels.","Editorial inference: The strongest external validation would be a real messaging study comparing the model's delays with actual human response times for matched events, since the benchmark is synthetic by design."],"forward_implications":["A dialogue agent could use predicted delays to avoid repetitive instant replies and to hold a conversation over an event's natural timescale, such as waiting through a commute before reporting back.","Fine-tuning on event-duration knowledge transfers timing ability better than in-context prompting, suggesting that timing should be trained explicitly rather than expected from instruction-following alone.","The 55K synthetic event-driven dialogue set provides a template for scaling other under-annotated dialogue skills from unlabeled knowledge graphs.","Evaluating timing and wording separately, at turn level and dialogue level, makes it possible to tell whether a failure comes from a wrong delay or from a wrong response."],"supporting_citations":[{"why":"Supplies the human-annotated event-duration pairs from MC-TACO that define the ground-truth delays in the TimelyChat benchmark.","marker":"(Zhou et al., 2019)"},{"why":"Supplies the ATOMIC20^20 event triplets that are expanded and pseudo-labeled to build the 55K training dialogues.","marker":"(Hwang et al., 2021)"},{"why":"GPT-4 writes the event-driven dialogues, acts as the user simulator in dialogue-level evaluation, and powers the G-Eval metrics.","marker":"(Achiam et al., 2023)"},{"why":"Defines Multi-Session Chat, the long-term dialogue baseline with inter-session intervals that TIMER is compared against.","marker":"(Xu et al., 2022a)"},{"why":"Introduces GapChat, the event-progress dialogue baseline whose preset intervals the paper moves beyond.","marker":"(Zhang et al., 2023)"},{"why":"Provides G-Eval, the LLM-based evaluation used for naturalness, time-specificity, coherence, and delay-appropriateness.","marker":"(Liu et al., 2023)"}],"fun_headline_variants":["TIMER 3B outscores GPT-4 on when to reply","Reply timing is learnable: TIMER 3B beats GPT-4","Dialogue agents learn to pause: TIMER tops GPT-4","TimelyChat: TIMER 3B excels at knowing when to speak"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark treats the typical duration of an ongoing event as the correct delay before the next message, a mapping that is assumed in the data construction and never checked against real messaging behavior.","fun_headline_variants_meta":{"raw":{"variants":["TIMER 3B outscores GPT-4 on when to reply","Reply timing is learnable: TIMER 3B beats GPT-4","Dialogue agents learn to pause: TIMER tops GPT-4","TimelyChat: TIMER 3B excels at knowing when to speak"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1740,"prompt_tokens":862,"completion_tokens":878,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":798}},"tokens_in":478,"tokens_out":878,"duration_ms":8380,"temperature":1.0,"reasoning_tokens":798,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:53:35.415167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect timestamped real-world messaging logs in which users record what event they were doing between messages, then compare actual delays with the typical durations of those events; if the correlation is weak, or delays cluster around social rhythms such as work breaks rather than event durations, the training signal and evaluation labels lose their foundation.","supporting_citations":[],"review_version":1}