{"id":"ab6e0394-44ad-4d87-b360-63657633384f","arxiv_id":"2501.00152","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A knowledge-distillation framework that injects temporal reasoning into a small language model improves its timeline summaries on out-of-domain mental-health social media data.","lead":"Researchers built a dataset of temporal relations between events in stories and used it to teach a large language model to reason about time. They then distilled this skill into a smaller model while training it to summarize timelines, and showed the distilled model produced better mental-health summaries of social media posts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Student is never evaluated on temporal reasoning, so the paper's causal claim that distilled temporal reasoning drives the timeline-summary gains is not yet supported.","rationale":"I read the paper's central claim as causal: teaching temporal reasoning to a teacher and distilling it into a student is what makes the timeline summaries more factual and less hallucinated. The strongest support is the contrast among P-Phi(NST&PRT), KDtimeline, and KDorigin in Table 2; that contrast does isolate the teacher's NarrativeReason SFT as the source of the gain. However, it does not establish that the transferred knowledge is temporal reasoning. The student is a different model; acquisition of a skill by the teacher does not imply transfer under KD. The representation analyses in §5.1 (UMAP/CKA) are suggestive but are correlation-only and do not test temporal competence. I therefore agree with the reader's weakest_assumption. I considered making the 30-sample/no-significance issue the primary concern; it is real and should be fixed, but it attacks the reliability of the measurement rather than the identity of the mechanism, and the reader already flags it. The missing student-side temporal evaluation is the more load-bearing concern because it targets the paper's unique causal claim. The proposed check would settle it: if the distilled student shows no temporal-reasoning gain over controls, the title-level claim should be weakened, even though the KD procedure might still be a practical recipe.","tokens_in":16724,"tokens_out":6773,"duration_ms":71524,"concrete_test":"Evaluate the distilled student models P-Phi(NST&PRT) and L-Phi(NST&PRT) on the TEMPREASON L3 event-event temporal reasoning benchmark (or a held-out split of NarrativeReason) using the same QA format as Appendix A.2, and compare their accuracy with Phitemp, Phitl, PhiICL, and the student trained with KDtimeline. If the KD students do not significantly outperform the non-temporal controls on this temporal reasoning task, the timeline-summary gains reported in Table 2 cannot be attributed to successful transfer of temporal reasoning, and the paper's central causal claim would need to be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 shows that P-Phi(NST&PRT) (FC .438, EA .973) beats the KDtimeline (.330/.965) and KDorigin (.332/.967) controls, which is evidence that the teacher's NarrativeReason fine-tuning matters. But the paper's stated mechanism is that temporal reasoning is distilled into the student (§3.3). The only direct evidence of temporal reasoning acquisition is for the teacher, not the student: Appendix A.2 reports a +0.07 F1 improvement on TEMPREASON L3 after teacher fine-tuning. No equivalent evaluation is run on P-Phi or L-Phi. The student could in principle be benefiting from a different teacher property, e.g., better instruction following, more robust long-context representations, or better alignment of the summary prompt, that is correlated with NarrativeReason SFT but is not temporal reasoning per se. Because the paper's contribution is explicitly 'leveraging temporal reasoning to improve timeline summarisation', the missing student-side temporal evaluation is load-bearing for the central causal claim. This is compounded by the fact that all conclusions rest on 30 TalkLife timelines with no significance testing, but the decisive missing piece is the student-side reasoning check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to improve timeline summarisation by first fine-tuning a large teacher LLM (LLaMA-3-8B) on a new temporal-reasoning dataset, NarrativeReason, built from NarrativeTime, and then distilling the teacher's knowledge into a smaller Phi-3 student while simultaneously fine-tuning the student on news timeline summarisation. The resulting student is applied out-of-domain to mental-health timeline summarisation on 30 TalkLife timelines, where it is compared with several baselines and ablations using Factual Consistency (FC), Evidence Appropriateness (EA), and a small human evaluation by two clinical psychology students. The central claim is that the KD approach 'produces more accurate summaries while significantly reducing hallucinations'.","tokens_in":16904,"tokens_out":3867,"duration_ms":40631,"significance":"If the central claim were fully supported, the paper would make a useful contribution: NarrativeReason is a new dataset with dense event-event temporal relations in narrative form, and the KD setup with timeline-summarisation-specific controls (KDtimeline, KDorigin) is a reasonable way to isolate the effect of temporal-reasoning fine-tuning. The out-of-domain evaluation on TalkLife is also a strength, as is the attempt to explain the gains through representation analyses (UMAP, CKA). However, the current evidence is not yet sufficient to establish the causal mechanism: the student is never evaluated on a temporal-reasoning task, the automatic results lack significance testing, and the evaluation protocol gives systems differential access to gold evidence. These issues affect the paper's main conclusion, so the manuscript needs major revision.","major_comments":[{"comment":"The evaluation protocol is confounded by differential access to gold evidence. The paper states that 'we use the annotated timeline evidence from Song et al. (2024) to directly generate high-level summaries,' whereas the TH-VAE baseline must first extract evidence from timelines and then translate it to high-level summaries. The reported gains over TH-VAE (FC .378 to .438; EA .970 to .973) are therefore uninterpretable as evidence for temporal reasoning: the comparison mixes the effect of the proposed method with the effect of bypassing evidence extraction. A fair comparison would either provide the same gold evidence to all systems or evaluate end-to-end evidence extraction and summarisation for every system.","section":"§4.3, Table 2"},{"comment":"All automatic conclusions rest on single-point FC and EA values computed over only 30 TalkLife timelines, with no confidence intervals, bootstrap estimates, or paired significance tests. Differences such as FC .424 vs .438 and EA .971 vs .973 are plausibly within noise. The abstract and conclusion claim that the KD approach 'significantly reduces hallucinations,' but 'significantly' is not supported by any statistical test. Per-timeline paired tests or bootstrap confidence intervals should be reported before making the headline claim.","section":"§5, Table 2"},{"comment":"The paper's causal mechanism is that temporal-reasoning knowledge is distilled into the student, yet the student is never evaluated on a temporal-reasoning benchmark. Appendix A.2 reports that fine-tuning improves the teacher on TEMPREASON L3 by +0.07 F1, but no equivalent evaluation is given for P-Phi or L-Phi. The observed gains on TalkLife could therefore arise from other teacher properties that are correlated with NarrativeReason SFT, such as better instruction following or different output style, rather than from temporal reasoning per se. A direct student-side temporal-reasoning evaluation is load-bearing for the central claim and should be added.","section":"§3.3, Appendix A.2"},{"comment":"The best KD configuration is selected using the same 30 TalkLife timelines that are later used for the final evaluation. Specifically, Table 1 compares several KD strategies on FC and EA, and NST&PRT is chosen as the best; the same 30 timelines then appear in Table 2 as the headline result. This is test-set selection and can inflate the reported numbers. A held-out validation split, nested evaluation, or at least an explicit acknowledgment of this selection effect is needed.","section":"§5, Table 1"},{"comment":"The human evaluation uses only two annotators, each rating 30 summaries on 5-point Likert scales, with no inter-annotator reliability measure and no significance testing. The paper states that clinical psychologists have 'a clear preference' for L-Phi, but the Likert differences are small (e.g., MOC 3.47 vs 3.42 for LLaMA) and the claim is not statistically supported. Reporting Cohen's kappa or similar, and at least paired tests on per-item ratings, would be necessary to support the preference claim.","section":"§5, Table 3"}],"minor_comments":[{"comment":"The acronym is used inconsistently: the methodology defines PRT, but the results section says 'PKT performed the best.' Please unify the notation throughout.","section":"§5, Table 1"},{"comment":"The row label 'KDtimline' is a typo; it should read 'KDtimeline.' Also, the column header 'Data for Fine-Tuning Teacher Model' is redundant and could be simplified.","section":"Table 2"},{"comment":"There is a typo in 'negtive' in the CRD loss equation, and the text says h should be normalized by L-2 norm but the displayed formula does not show the normalization explicitly.","section":"Eq. (3)"},{"comment":"The example outputs contain apparent generation artifacts such as 'mental wellbes' and 'life-thyesing thoughts.' If these are verbatim model outputs, they should be clearly marked as such; otherwise the appendix should be proofread.","section":"Appendix A.4"},{"comment":"The paper does not report how the 5,000 Wikipedia timelines are split into training and validation sets, nor how teacher and student training epochs and checkpoints were selected. Please add these details.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper does not need to be rejected on novelty or scope grounds; the dataset and the KD-with-controls setup are reasonable. However, the three load-bearing issues in the automatic evaluation, student-side temporal-reasoning validation, and statistical rigor must be addressed before the central claim can be accepted. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper has a genuinely new dataset and one very good control, but the central causal claim—that distilled temporal reasoning is what improves the summaries—is not yet proven. The student is only evaluated on the timeline summarisation task, never on a temporal reasoning task, so the paper can't rule out that the teacher's NarrativeReason fine-tuning helps for other reasons, such as better instruction following or more robust representations.\n\nWhat's actually new: NarrativeReason, built from NarrativeTime, targets temporal relations among sequences of events rather than isolated pairs. That fills a real gap. The KD setup is a sensible combination, and the control is the strongest part: KDtimeline (teacher fine-tuned on the same timeline data) and KDorigin (untuned teacher) both score clearly below P-Phi (FC .330/.332 vs .438). That's decent evidence that the extra teacher fine-tuning on NarrativeReason matters. The UMAP/CKA analysis is a good-faith attempt to explain why KD works.\n\nThe soft spots are real. Most importantly, the missing student-side temporal evaluation. If the claim is that temporal reasoning is transferred, evaluate the student on a temporal reasoning task. Without that, the mechanism is unconfirmed. Second, the evaluation is small: 30 TalkLife timelines, two annotators, no inter-annotator reliability, no significance testing. The best KD strategy (NST&PRT) is selected by comparing results on those same 30 timelines, so the reported numbers are optimistic. Third, the TH-VAE comparison isn't apples-to-apples: the authors feed gold evidence into their generator while TH-VAE has to extract evidence from scratch. They justify this in Section 4.3, which is fair, but the TH-VAE numbers shouldn't be read as a direct win. Minor: Table 2 has a typo (\"KDtimline\").\n\nThe paper is honest about its limitations, including overconfident DSM diagnoses and generic summaries, which counts in its favor.\n\nWho should read it: people working on temporal reasoning for downstream generation, and anyone in mental-health NLP. It deserves a rigorous referee; the dataset and setup merit attention, but the evaluation needs tightening.\n\nMy recommendation: send to peer review. Ask for a student-side temporal reasoning evaluation, significance testing or more timelines, and a fairer TH-VAE comparison. With those changes, it would be a solid contribution.","headline":"A useful new dataset and a clean control, but the temporal-reasoning mechanism is unproven and the evaluation is too small to carry the claims.","tokens_in":17427,"tokens_out":4146,"would_cite":true,"duration_ms":37398,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that temporal reasoning over event sequences, transferred from a large teacher to a small student by knowledge distillation, improves out-of-domain timeline summarisation and reduces hallucinations.","keywords":["temporal reasoning","timeline summarisation","knowledge distillation","large language models","mental health","social media","NarrativeReason","hallucination reduction"],"falsifier":"Run the distilled student on an event-ordering benchmark, for example the event-event portion of TEMPREASON, and compare it with a student distilled from a teacher fine-tuned on a non-temporal task of the same format; equal performance would show the gains are not caused by temporal reasoning transfer.","tokens_in":16490,"feed_emoji":"🧠","tokens_out":7464,"duration_ms":68028,"temperature":0.7,"pith_summary":"The paper claims that temporal reasoning over full event sequences, not just event pairs, is a transferable skill that improves timeline summarisation in language models. It introduces NarrativeReason, a dataset of 19,614 temporal-relation question/answer pairs built from densely annotated narratives, fine-tunes a large teacher model on it, and distills the teacher into a smaller student while the student is simultaneously trained on news timeline summarisation. Applied to an out-of-domain mental-health timeline task, the distilled student produces summaries that score higher on factual consistency and evidence appropriateness than baselines including joint training, in-context learning, and a prior hierarchical VAE, and clinical raters prefer them. The paper also uses representation analyses to argue that distillation, unlike joint training, creates task-aligned internal representations that explain the improvement.","feed_headline":"Event-order training cuts hallucinations in AI timeline summaries","feed_subtitle":"Distilled event-order knowledge yields more faithful, less hallucinated mental-health timeline summaries.","key_machinery":"The load-bearing mechanism is the two-stage distillation pipeline. NarrativeReason converts dense NarrativeTime annotations into event triples and full-sequence temporal-relation question/answer pairs, so the teacher learns relations among a series of events rather than isolated pairs. The teacher is fine-tuned with LoRA, then frozen; the student is trained on news timeline summarisation while three representation-matching losses, neuron selectivity transfer (NST), probabilistic knowledge transfer (PRT), and contrastive representation distillation (CRD), pull its hidden representations toward the teacher's. NST plus PRT combined gives the best results, and the authors argue these structure-matching losses are better suited to temporal coherence than CRD's instance discrimination.","core_discovery":"The central claim is that knowledge distillation is the right vehicle for combining temporal reasoning with timeline summarisation: fine-tuning on temporal reasoning alone, timeline summarisation alone, or both jointly all underperform, while a student that learns timeline summarisation and simultaneously matches a temporal-reasoning teacher's representations transfers the temporal skill without task interference. The empirical anchor is the out-of-domain TalkLife setting, where the best KD model with NST and PRT reaches a factual consistency of 0.438 and an evidence appropriateness of 0.973, and human raters give it the highest scores on factual consistency and general usefulness. The authors attribute the gain to the student learning more polysemantic, task-aligned representations, shown through UMAP and CKA analyses comparing the distilled student with the jointly trained model.","pith_inferences":["The causal story would be strengthened by a direct student-side test; the paper only verifies the teacher on TEMPREASON, so student-side temporal reasoning gains remain an inference.","The same recipe could transfer to other longitudinal summarisation settings with scarce in-domain labels, such as clinical notes or support conversations, where event order is clinically meaningful.","Because joint training fails while distillation succeeds, the teacher may be acting as a regulariser; a matched non-temporal reasoning teacher would reveal whether temporal content or general representation-matching drives the gain."],"forward_implications":["A smaller model can acquire temporal reasoning from a larger teacher and apply it in a domain the teacher never saw, outperforming models fine-tuned on either task alone.","Joint training on temporal reasoning and timeline summarisation actively hurts performance, so distillation's separation of the two learning signals is necessary, not optional.","Matching structural activation and probability distributions (NST and PRT) beats contrastive representation matching (CRD) for temporally coherent summarisation.","The best distilled model's factual-consistency score of 0.438 substantially exceeds timeline-only distillation (0.330) and joint training (0.238), supporting the paper's claim that temporal reasoning specifically reduces hallucinations."],"supporting_citations":[{"why":"Supplies the NarrativeTime dense timeline annotations from which NarrativeReason's event triples and temporal relations are built.","marker":"Rogers et al. (2024)"},{"why":"Defines the event-relation categories and question-answer format used in NarrativeReason, and provides the TEMPREASON benchmark used to validate the teacher.","marker":"Tan et al. (2023)"},{"why":"Defines the mental-health timeline summarisation task, the TalkLife summaries and evidence, the FC/EA evaluation metrics, and the TH-VAE baseline.","marker":"Song et al. (2024)"},{"why":"Supplies the Wikipedia timeline summarisation corpus on which the student model is fine-tuned.","marker":"Chen et al. (2023)"},{"why":"Provides the TalkLife longitudinal user timelines that the out-of-domain evaluation uses.","marker":"Tsakalidis et al. (2022)"},{"why":"Defines the NST distillation objective that matches teacher and student neuron selectivity patterns.","marker":"Huang and Wang (2017)"},{"why":"Defines the PRT distillation objective that matches conditional probability distributions in feature space.","marker":"Passalis and Tefas (2018)"},{"why":"Defines the CRD contrastive representation distillation objective used as one of the transfer strategies.","marker":"Tian et al. (2019)"}],"fun_headline_variants":["Distilled event-order knowledge improves timeline summaries","Temporal reasoning distillation boosts timeline summarization","Event-order knowledge transfer improves mental-health timelines","Distillation gives LLMs better event-order sense for summarization","Event-order distillation reduces hallucinations in AI summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the student's better summaries come from the event-ordering knowledge it absorbed from the teacher, but the student is never tested on event ordering directly, so other differences between teacher and student could explain the gains.","fun_headline_variants_meta":{"raw":{"variants":["Distilled event-order knowledge improves timeline summaries","Temporal reasoning distillation boosts timeline summarization","Event-order knowledge transfer improves mental-health timelines","Distillation gives LLMs better event-order sense for summarization","Event-order distillation reduces hallucinations in AI summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00082,"raw_usage":{"total_tokens":3538,"prompt_tokens":842,"completion_tokens":2696,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2626}},"tokens_in":458,"tokens_out":2696,"duration_ms":18374,"temperature":1.0,"reasoning_tokens":2626,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:57:27.440678+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the distilled student on an event-ordering benchmark, for example the event-event portion of TEMPREASON, and compare it with a student distilled from a teacher fine-tuned on a non-temporal task of the same format; equal performance would show the gains are not caused by temporal reasoning transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NarrativeTime dense timeline annotations from which NarrativeReason's event triples and temporal relations are built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Wikipedia timeline summarisation corpus on which the student model is fine-tuned."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PRT distillation objective that matches conditional probability distributions in feature space."}],"review_version":1}