{"id":"2b975b69-1944-4d3b-94ba-fcbad979f905","arxiv_id":"2506.21616","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A timeline summarization model trained on a new 1,189-topic dataset outperforms much larger general LLMs on the authors' open-domain test set.","lead":"This paper builds TLS-I, a large news timeline dataset with 1,189 topics and over 3,000 annotated timelines, and trains TIM, a model specialized in open-domain timeline summarization. TIM outperforms general large language models on the authors' 132-query test set, but the dataset and code are not yet released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o-built gold standard creates a closed loop: TIM is trained and evaluated against the same Alignment F1 metric on the same style of reference, so the 14% gain over DeepSeek may reflect style matching rather than intrinsic timeline quality.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the proxy gold standard is generated by the same GPT-4o pipeline that produces training data and reward pairs. My reading sharpens this into a concrete closed loop: the reward function used in Section 5.2 is Alignment F1 against GPT-4o-derived references, and the evaluation metric in Section 6.1 is the same Alignment F1 against references built by the same protocol. This makes the comparison with zero-shot general LLMs structurally favorable to TIM, independent of whether the gold is 'correct' in an absolute sense. The paper's own evidence does not break the loop: the 132-query test set is new only at the query level, not at the gold-construction level, and no external human-only benchmark is reported. Additional inconsistencies, such as the abstract and Section 4.2 stating about 100,000 source articles while Table 1 implies 1,252,017, and the absence of error bars or significance tests, further support a conditional verdict, but they are secondary to the evaluation-independence problem. Because the reader already assigned CONDITIONAL, my recommendation is not to change the verdict; the concern confirms that conditional status rather than overturning it. The proposed re-annotation test directly targets the weakest point and would settle whether the reported superiority is intrinsic or stylistic.","tokens_in":14138,"tokens_out":4764,"duration_ms":49458,"concrete_test":"Re-annotate the 132 test queries with a fresh team of experts who construct gold timelines directly from the retrieved article sets, with no GPT-4o draft and no exposure to TIM or DeepSeek outputs, then recompute Table 2 Alignment F1 for TIM-14B (Pro) versus DeepSeek-V3-0326. If the average margin falls materially or becomes non-significant, the SOTA claim is largely an artifact of matching GPT-4o-style gold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim rests on the 132-query test set (Section 6.1). Its gold timelines are produced by the same GPT-4o pipeline used to build the training and preference data (Section 4.1, Figure 3): GPT-4o generates a basic timeline, then no fewer than five human experts review and vote on it. There is no evidence that the test gold was independently written from source articles, and no inter-annotator agreement is reported for the test set. In parallel, the dual-alignment reward (Section 5.2, Eq. 3) and the headline metric Alignment F1 (Section 6.1) are the same measure against the same reference distribution. Consequently, TIM-14B (Pro) is optimized to reproduce GPT-4o-style selection and phrasing, while DeepSeek-V3-0326 and other general LLMs receive only zero-shot prompts. Table 2's over-14% margin is therefore consistent with a closed-loop effect: the model learns the exact annotation style and metric used to grade it, so the evaluation cannot distinguish better timelines from better GPT-4o mimicry. This is not a claim of query-level leakage; the 132 queries are absent from training. It is a claim that the evaluation gold is not independent of the training target. Human refinement mitigates but does not remove the concern: expert voting on an LLM draft anchors the gold to the draft's choices, and the same voting protocol is used at test time.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TLS-I, a large open-domain timeline summarization dataset in Chinese (1,189 topics, 3,567 timelines, over 1.2 million source articles, 12 domains), and proposes TIM, a timeline intelligence model trained with a progressive strategy. The strategy first applies instruction tuning with topic-aware sampling that mixes high- and low-relevance documents (Eq. 2), then applies a dual-alignment reward learning stage based on a DPO-style objective (Eq. 3) to improve semantic and temporal alignment. The authors evaluate TIM-7B and TIM-14B against general LLMs on a held-out set of 132 queries, reporting that TIM-14B (Pro) outperforms DeepSeek-V3-0326 by over 14% on alignment-related metrics (Section 6.2, Table 2). The paper also includes ablations on sampling strategy and an analysis of generation versus merging model sizes.","tokens_in":14463,"tokens_out":4933,"duration_ms":51309,"significance":"If the reported results hold, this is a useful contribution to timeline summarization: TLS-I is substantially larger than prior datasets (Table 1), the progressive training recipe is clearly motivated, and demonstrating that a 14B model can outperform a 685B general LLM on a structured, retrieval-augmented task would be practically significant. The authors are explicit about releasing dataset, code, and models. However, the central SOTA claim is currently supported only by a single test set whose gold data are produced by the same GPT-4o-plus-human pipeline used to construct training data and preference rewards, and by results without error bars or significance testing. The closed-loop concern raised in the stress-test note is real and needs to be addressed with an independent evaluation or at least additional evidence.","major_comments":[{"comment":"The stated dual-alignment objective is not DPO as written. Standard DPO requires the difference of implicit reward log-probabilities relative to a frozen reference policy, i.e., log(pi_theta(y_w|x)/pi_ref(y_w|x)) - log(pi_theta(y_l|x)/pi_ref(y_l|x)), with a KL-divergence term. Equation (3) instead contains only log pi_theta(s+_i|a_i) - log p_theta(s-_i|a_i), omitting pi_ref and using inconsistent notation (p_theta vs pi_theta, s vs y). This is a load-bearing part of the proposed method because the paper attributes the TIM (Pro) gain to this objective. Please provide the correct loss, state whether a reference model is used, and specify the construction of positive and 'partially aligned negative' samples, including how Alignment F1 is used to rank them.","section":"Section 5.2, Eq. (3)"},{"comment":"The evaluation design creates a closed loop that undermines the SOTA claim. Training data, preference pairs, and the 132-query test gold are all produced by the same GPT-4o generation plus human-expert voting pipeline (Section 4.1, Figure 3), and the dual-alignment reward uses Alignment F1, the same metric used for evaluation. Even though the 132 test queries are disjoint from training queries, the gold reference style and the metric are identical in training and evaluation, so the reported margin over zero-shot general LLMs may reflect learned mimicry of GPT-4o-style outputs rather than intrinsically better timelines. Human review mitigates but does not eliminate this concern. Please report inter-annotator agreement on the test set, provide evidence that test gold was independently constructed from source articles, or evaluate on an external benchmark such as Crisis or T17 with their established gold timelines.","section":"Sections 4.1, 5.2, 6.1"},{"comment":"No variance or significance information is reported. Table 2 gives single values for each model, and several key differences are small (e.g., TIM-7B Pro Alignment F1 0.368 vs TIM-7B Standard 0.338; TIM-7B Standard vs DeepSeek-V3-0326 on full Alignment F1, 0.338 vs 0.336). Without confidence intervals, bootstrap estimates, or significance tests on the 132-query set, the claim that TIM (Pro) consistently outperforms all baselines is not yet established. Please add such statistics and a per-domain breakdown, since open-domain generalization is a central claim.","section":"Section 6.2, Table 2"},{"comment":"The reported dataset scale is internally inconsistent. Section 4.2 states the dataset contains 'more than 100,000 collected source news,' while Table 1 reports 1,252,017 articles for TLS-I. The abstract also emphasizes scale. Please correct the numbers and ensure the article count is consistent throughout, as this is a primary claimed contribution.","section":"Section 4.2 and Table 1"}],"minor_comments":[{"comment":"The evaluation does not state the language of the 132 test queries or the prompt language used for the baselines. Since TLS-I is listed as Chinese in Table 1, please clarify whether all queries and prompts are in Chinese and whether the English-named baselines were run in Chinese.","section":"Section 6.1"},{"comment":"The text says 'their results are not reported in the Table 3,' but the overall results appear in Table 2. Please fix the reference.","section":"Appendix A.3"},{"comment":"The heading 'Training preferences for models with different parameter sizes' is unclear; the table appears to compare TIM (Pro) against the base instruct models on base and enhanced splits rather than displaying preferences. Please rename the heading and explain what 'preferences' means here.","section":"Section 6.3.2, Table 3"},{"comment":"There are several typos and unclear formulations, e.g., 'the validated of the model's generalization capabilities' in Section 2.2, the column header 'the volume of refusal' in Table 2, and inconsistent use of y and s in Section 5.2. A careful proofread is needed.","section":"Throughout"},{"comment":"The reference list contains entries that appear to be template artifacts (e.g., Aho and Ullman 1972, APA 1983, Gusfield 1997) and many entries ending in 'and 1 others.' Please clean the bibliography so that only cited works appear with complete author lists, or use a consistent abbreviation style.","section":"References"},{"comment":"Fleiss' Kappa is mentioned as the inter-annotator agreement measure, but no actual Kappa value is reported for the training split or the test set. Please report the observed agreement values.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the closed-loop evaluation: training data, preference reward, and test gold all come from the same GPT-4o-plus-human pipeline, and the reward metric is the evaluation metric. This is not necessarily indicative of bad faith, but it is a correctness risk that must be addressed before the SOTA claim can be accepted. The dataset itself, once corrected for the article-count inconsistency, could be a valuable community resource. Please also ensure the final version fixes the DPO equation, which is currently incomplete as a specification of the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — worth a read if you work on summarization, but the headline number should be treated with caution. The genuinely new thing is TLS-I: 1,189 topics, 3,567 timelines, over a million articles, across 12 domains. That is real scale for TLS, and it makes training a dedicated model feasible. The progressive optimization (topic-aware instruction tuning then DPO-like reward learning) is not methodologically exotic — it is standard SFT plus preference optimization — but the paper shows consistent gains over strong baselines, and the ablations suggest the topic-aware sampling helps.\n\nThe main problem is the closed loop. Gold timelines are built by GPT-4o then refined by five human experts. The test gold is “consistent with the data collection process,” so it is hard to rule out that TIM learns to imitate GPT-4o’s selection and phrasing. The reward signal and the headline metric are the same Alignment F1 against the same reference style. Human refinement mitigates but does not break the loop, because the experts vote on an LLM draft. No inter-annotator agreement is reported for the 132-query test set, and no significance tests or error bars appear. That makes the 14% margin over DeepSeek plausible but not established.\n\nAlso, Eq. 3 is not the standard DPO objective: it omits the reference-model term. Maybe that’s intentional, but it needs explanation. And the paper says 100,000 articles in the abstract and intro while Table 1 lists 1,252,017. That is a serious counting discrepancy that should have been caught.\n\nBottom line: the dataset contribution is strong and deserves referee time. The training pipeline is a reasonable engineering contribution. But the evaluation needs an independent reference — human-written from source articles, or at least a blind human preference study — before the SOTA claim can be trusted. I’d send this to peer review, with the expectation that data/code release and evaluation fix-ups are required.","headline":"A big new TLS dataset and a plausible but not yet proven SOTA claim, because the evaluation gold and the training reward come from the same GPT-4o pipeline.","tokens_in":14938,"tokens_out":2255,"would_cite":false,"duration_ms":21514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 14-billion-parameter model, trained with topic-aware instruction tuning and dual-alignment reward learning on a new 1,189-topic dataset, outperforms much larger general-purpose LLMs on open-domain timeline summarization.","keywords":["open-domain timeline summarization","topic-aware sampling","dual-alignment reward learning","instruction tuning","direct preference optimization","temporal grounding","news timeline dataset","retrieval-augmented generation"],"falsifier":"Take the same 132 test queries and have annotators independently build gold timelines directly from primary news sources, then score TIM-14B (Pro) and a top general LLM on Date F1 and Alignment F1 against that independent standard; if TIM's margin disappears or reverses, the reported superiority is an artifact of sharing the same generation pipeline.","tokens_in":13941,"feed_emoji":"📅","tokens_out":6655,"duration_ms":61769,"temperature":0.7,"pith_summary":"The paper argues that general-purpose LLMs, despite strong zero-shot summarization, fail at open-domain timeline summarization because they cannot reliably decide which news is on-topic and cannot track how a topic evolves over time. To fix this, it introduces TLS-I, a dataset of more than 1,000 news topics and 3,000+ annotated timelines, and TIM, a 7B/14B model trained in two stages: instruction tuning with a topic-aware mix of relevant and irrelevant news, then dual-alignment reward learning that scores both semantic match and timestamp match. The paper reports that TIM-14B (Pro) surpasses a 685-billion-parameter general LLM by more than 14% on average across the evaluation metrics, and that performance scales with model size. If this holds, a comparatively small, task-specialized model can replace much larger general models for structured, date-aligned news summaries.","feed_headline":"14B model beats 685B LLM on open-domain timelines","feed_subtitle":"A new 1,189-topic dataset and two-stage tuning give a small model better date-aligned timeline summaries.","key_machinery":"The machinery is a progressive optimization pipeline with three load-bearing pieces. First, instruction tuning uses a topic-aware sampling objective in which high-relevance and low-relevance news documents are weighted by a learnable sigmoid parameter, so the model learns both to summarize salient content and to discard off-topic noise. Second, dual-alignment reward learning applies a preference-optimization loss, inspired by direct preference optimization, in which positive and negative timeline candidates are constructed using the Alignment F1 metric, rewarding the model for matching both the reference wording and the reference dates. Third, the pipeline rests on TLS-I, a dataset of 1,189 topics, 3,567 annotated timelines, and over 100,000 source articles across 12 domains, with each topic carrying separate basic-retrieval, enhanced-retrieval, and merged timelines.","core_discovery":"The central claim is that task-specific training, not raw scale, is what open-domain timeline summarization requires. General LLMs produce redundant content and inaccurate timestamps because they lack task-specific knowledge of topic relevance and topic evolution; TIM, trained on a large purpose-built dataset, learns to filter off-topic news, select salient events, and align them to correct dates. The paper demonstrates this with TIM-7B and TIM-14B, showing state-of-the-art results on Alignment F1, Date F1, and ROUGE metrics over all compared general LLMs, including a 685B parameter model, with TIM-14B (Pro) achieving an average improvement of more than 14% over that model.","pith_inferences":["Because both the training data and the 132-query test set share the same automated LLM-plus-human generation pipeline, an independent gold standard built directly from primary news sources would reveal how much of TIM's reported advantage is genuine temporal grounding rather than stylistic alignment with the reference generator.","The dual-alignment reward recipe could transfer to other date-sensitive tasks such as event tracking, temporal question answering, or historical fact-checking, where matching content and timestamps simultaneously is the core difficulty.","The reported margin over a 685B parameter general LLM may be understated on sensitive queries, because several API models refused to answer politically sensitive topics while TIM produced outputs for all queries.","Applying the same progressive optimization to a 32B or 72B base model is a direct, untested extension; the authors cite resource limits rather than any evidence that the gains would plateau."],"forward_implications":["A compact 14B parameter model can set the state of the art in open-domain timeline summarization, suggesting that current frontier LLMs are not necessary for this structured summarization task.","Performance improves consistently from 7B to 14B parameters, indicating that the same recipe should scale to larger model sizes and yield further gains.","The dataset's split into basic and enhanced timelines supports separate study of retrieval quality, event extraction, and redundancy filtering across twelve news domains.","General LLMs tend to lose timestamps and inject irrelevant time nodes; TIM reduces both failure modes, making timelines more reliable for monitoring evolving news topics.","The paper's finding that an untrained model suffices for merging two generated timelines means the trainable effort can be concentrated on the generation step alone."],"supporting_citations":[{"why":"Defines open-domain timeline summarization and supplies the retrieval-plus-LLM framework that TIM extends and improves upon.","marker":"(Wu et al., 2025)"},{"why":"Defines Alignment F1, the metric used both as the reward signal in dual-alignment learning and as a headline evaluation measure.","marker":"(Martschat and Markert, 2018)"},{"why":"Introduces the ROUGE and DATE evaluation measures used to score generated timelines against the ground truth.","marker":"(Martschat and Markert, 2017)"},{"why":"Provides the direct preference optimization principle on which the dual-alignment reward learning is based.","marker":"(Rafailov et al., 2023)"},{"why":"One of the small prior timeline summarization datasets used to motivate the need for a large-scale corpus.","marker":"(Binh Tran et al., 2013)"},{"why":"Defines the timeline summarization task and another small dataset against which TLS-I is compared in scale and coverage.","marker":"(Tran et al., 2015)"},{"why":"Supplies the inter-annotator agreement measure used to validate the quality of the human-refined dataset.","marker":"(Fleiss, 1971)"},{"why":"Provides the reranker used to select high-relevance and low-relevance documents for topic-aware sampling.","marker":"(Chen et al., 2024)"}],"fun_headline_variants":["TIM-14B outranks 685B LLM on timeline summaries","Small TIM model beats 685B giant on news timelines","New dataset and two-stage tuning boost timeline summarization","TIM: task-specific training beats raw scale for timelines","Open-domain timeline summaries improved by TIM model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the reference timelines produced by an LLM and then reviewed by human experts are the correct gold standard; if that gold standard is biased, both the training rewards and the evaluation scores are biased in the same direction.","fun_headline_variants_meta":{"raw":{"variants":["TIM-14B outranks 685B LLM on timeline summaries","Small TIM model beats 685B giant on news timelines","New dataset and two-stage tuning boost timeline summarization","TIM: task-specific training beats raw scale for timelines","Open-domain timeline summaries improved by TIM model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000926,"raw_usage":{"total_tokens":3949,"prompt_tokens":906,"completion_tokens":3043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":2963}},"tokens_in":522,"tokens_out":3043,"duration_ms":20166,"temperature":1.0,"reasoning_tokens":2963,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:55:00.378161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 132 test queries and have annotators independently build gold timelines directly from primary news sources, then score TIM-14B (Pro) and a top general LLM on Date F1 and Alignment F1 against that independent standard; if TIM's margin disappears or reverses, the reported superiority is an artifact of sharing the same generation pipeline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Alignment F1, the metric used both as the reward signal in dual-alignment learning and as a headline evaluation measure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the ROUGE and DATE evaluation measures used to score generated timelines against the ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the small prior timeline summarization datasets used to motivate the need for a large-scale corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the timeline summarization task and another small dataset against which TLS-I is compared in scale and coverage."}],"review_version":2}