{"id":"cdbfc0ec-0a32-4ea9-a2d3-7053364fdbb2","arxiv_id":"2501.00888","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CHRONOS applies iterative self-questioning and retrieval to build news timelines, and the authors release Open-TLS, a 50-topic benchmark of journalist-written timelines.","lead":"This paper presents CHRONOS, a system that uses large language models to repeatedly ask questions about a news topic, search for answers online or in a fixed corpus, and assemble a chronological timeline of events. It also introduces Open-TLS, a new dataset of 50 journalist-written timelines for evaluating open-domain timeline summarization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline comparisons are selected on test labels (best model per dataset, optimal round per Table 2) and few-shot examples are curated with reference timelines from the same benchmarks, so the reported gains and SOTA parity may be selection artifacts.","rationale":"The central claim is that CHRONOS improves open-domain TLS and rivals closed-domain SOTA. For this to hold, the reported gains must reflect the method rather than the freedom to choose the best backbone, the best self-questioning round, and the best few-shot demonstrators after seeing the test labels. The paper explicitly states in Section 5.4.2 that it 'select[s] the best-performing model to report its performance' and Table 2 says outcomes are from 'the optimal self-questioning round.' Because the baselines (DIRECT, REWRITE, LLM-TLS) are not given the same selection freedom, the comparisons in Tables 2 and 3 are not between fixed systems. The few-shot pool further couples evaluation to the test benchmarks: Chrono-Informativeness is defined as Date F1 against reference timelines (Sec. 3.1), and the pool is built from Crisis, T17, and Open-TLS topics, so for each test topic the demonstrators are chosen with the help of reference timelines from the same datasets. The ablated variants (Random Exemplar, w/o Rewrite) are informative, but they are evaluated under the same test-set selection, so they do not resolve the bias. The right fix is a held-out validation protocol. This is not a dispute about the plausibility of the method; it is a dispute about whether the reported numbers measure what they claim. The concrete test above would settle it.","tokens_in":14758,"tokens_out":7671,"duration_ms":69526,"concrete_test":"Split Open-TLS into 40 development topics and 10 held-out test topics. Build the example pool and select the self-questioning round and backbone model using only the development topics (with their reference timelines for CI), then run CHRONOS, DIRECT, and REWRITE on the held-out topics with the selected configuration, and re-run the closed-domain comparison with a single backbone model (Qwen2.5-72B) and the round count fixed on the development split. If the open-domain gains and the closed-domain parity with LLM-TLS are not reproduced on the held-out topics, the reported results are explained by test-set selection rather than the method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is that the headline comparisons in Tables 2 and 3 are selected on test labels. Section 5.4.2 states 'We select the best-performing model to report its performance', and Table 2 reports 'outcomes from the optimal self-questioning round'. This gives CHRONOS a search over three backbone LLMs and over several rounds, all chosen after seeing the test results. The baselines DIRECT/REWRITE have no rounds, and LLM-TLS⋆ is fixed to Qwen2.5-72B, so CHRONOS is compared at its own best point. The reported closed-domain advantage on AR-2 (0.045 vs 0.036 on Crisis; 0.042 vs 0.040 on T17) is small enough that model selection could account for it. The example-pool construction adds a second path for test information: Sec. 3.1 defines Chrono-Informativeness using Date_F1 against a reference timeline, and Sec. 5.1 says the pool is built from Crisis, T17, and Open-TLS topics. Even after excluding the target topic's own pair, the few-shot demonstrators for a test topic are selected using reference timelines of the same benchmark, which can inflate especially Date F1. No held-out validation is provided to show the selected configuration transfers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CHRONOS, an LLM-driven iterative self-questioning and retrieval framework for timeline summarization (TLS). CHRONOS alternates between generating 5W1H questions about a target news topic, rewriting those questions into search queries, retrieving documents from the web or an offline corpus, and generating/merging chronologically ordered timelines. The authors also introduce Open-TLS, a new dataset of 50 professionally authored timelines on recent news topics. Experiments compare CHRONOS against DIRECT and REWRITE baselines on Open-TLS and against prior TLS systems on the closed-domain Crisis and T17 benchmarks, reporting improvements on several metrics and claiming to rival state-of-the-art closed-domain systems.","tokens_in":15052,"tokens_out":2686,"duration_ms":25876,"significance":"If the central claims hold, the paper would make a useful practical contribution: a retrieval-based self-questioning loop that builds timelines without a pre-collected corpus, plus a new, larger, and more current open-domain TLS dataset. The authors release code and data, which aids reproducibility. The ablations on exemplar selection, question rewriting, and the number of retrieval rounds provide structured evidence about which components matter. However, the headline comparisons are undermined by test-label-based selection of models and rounds, and by few-shot example curation that uses reference timelines from the same benchmarks; these issues call into question the magnitude and even the direction of the claimed gains, particularly the small margins on closed-domain datasets.","major_comments":[{"comment":"The headline comparisons are selected on test labels. Section 5.4.2 states 'We select the best-performing model to report its performance,' and Table 2 reports 'outcomes from the optimal self-questioning round.' This is a search over three backbone LLMs and over several rounds, all chosen after seeing the test results, while the DIRECT/REWRITE baselines have no rounds and LLM-TLS⋆ is fixed to Qwen2.5-72B. The small closed-domain margins (e.g., AR-2 0.045 vs 0.036 on Crisis; 0.042 vs 0.040 on T17) could plausibly be artifacts of this selection. The paper should report results for all models and all rounds, or select configurations on a held-out validation split, before claiming consistent improvement or SOTA parity.","section":"§5.4.2, Table 2"},{"comment":"The few-shot example pool is curated using reference timelines from the same benchmarks used for evaluation. Chrono-Informativeness (Eq. in §3.1) is defined as Date_F1(T_Q,N, T_ref), and Section 5.1 states that the example pool is built from Crisis, T17, and Open-TLS topics. Even when the target topic's own pair is excluded, the few-shot demonstrations for a test topic are selected using reference timelines from the same benchmark, which injects test-set information into the prompt construction and can inflate Date F1 in particular. The authors should construct the example pool without using evaluation benchmarks' references, or demonstrate that the selection does not change when using a development split.","section":"§3.1, §5.1"},{"comment":"The paper reports averages of 3 runs but gives no standard deviations, confidence intervals, or significance tests. This is especially important because several headline comparisons are small (notably AR-2 on T17: 0.042 vs 0.040, and Date F1 on Crisis: 0.323 vs 0.329), and the selection of the best model and best round from the test set makes the results even more sensitive to variance. Without error bars or significance testing, the reader cannot tell whether the claimed advantages are reliable.","section":"§5.1, Tables 2 and 3"}],"minor_comments":[{"comment":"The text says the removal of the rewriter causes 'a slight improvement (+0.02)' on Crisis AR-2, but Table 4 shows 0.047 without the rewriter versus 0.045 with it, a difference of +0.002. The stated value appears to be off by an order of magnitude.","section":"§5.5.2, Table 4"},{"comment":"The column headers 'A vg.l' and 'A vg.k' are rendered with a spurious space. This formatting issue obscures the meaning of the statistics.","section":"Table 1"},{"comment":"The caption and text do not specify whether the scores in Figure 3 are averaged across all Open-TLS topics or shown per topic. Clarifying this would make the round-selection behavior easier to interpret.","section":"Figure 3"},{"comment":"The case study would benefit from a note on how the final merged timeline is produced from the two rounds; the table shows Round 1 and Round 2 generated timelines but not the merged output that would be evaluated.","section":"§5.7.2, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The central methodological contribution is plausible, and the Open-TLS dataset plus code release are valuable. The load-bearing problem is not that the method is unsound in principle, but that the reported numbers are selected on test labels and the example-pool construction uses test references. Both are fixable with a held-out validation split and by reporting all configurations with confidence intervals. If the authors can redo the evaluation without these selection artifacts, the paper could become publishable. I do not see an irreparable flaw in the framework itself, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper is that it ships a genuinely useful new resource — Open-TLS, 50 journalist-authored timelines on recent news topics — and a coherent RAG-style pipeline for timeline summarization that iteratively asks questions and retrieves news. The method is clearly described and the ablations show each component (few-shot example selection, question rewriting, multiple rounds) matters. The efficiency comparison against LLM-TLS is also a real point in its favor: the retrieval loop reads far fewer documents.\n\nBut the empirical headline — that CHRONOS rivals closed-domain state-of-the-art — is not supported as stated. The evaluation picks the best self-questioning round per model (Table 2) and the best model per benchmark (Section 5.4.2) after seeing test results. That gives CHRONOS a search over configurations that the baselines and prior systems don't get. On top of that, the few-shot question examples are chosen using Chrono-Informativeness, which is Date F1 against the reference timelines, and the example pool is built from the same benchmarks used for evaluation (Crisis, T17, and Open-TLS). So test information leaks into both the prompt construction and the final configuration selection. The closed-domain margins over the REWRITE baseline (e.g., 0.045 vs 0.036 on Crisis AR-2) are small enough that this selection could easily account for them. The paper also reports averages over 3 runs but gives no variance, so we can't tell whether any of these differences are stable.\n\nThe framework itself is sensible; the problem is the evaluation protocol. A clean fix would be to fix the number of rounds and model choice on a validation split, exclude Open-TLS topics from the example pool when evaluating on Open-TLS, and report all rounds rather than the best. If the authors did that, the 'rivals SOTA' claim might shrink, but the dataset and the basic retrieval-loop idea would still stand on their own.\n\nWho should read this: anyone working on timeline summarization, retrieval-augmented generation for news, or benchmark construction. The dataset alone is worth citing. The paper deserves a serious referee — this is a fixable methodology issue, not a fundamentally broken approach. I would send it to review, but the reviewers should insist on the corrected evaluation before acceptance.","headline":"Useful new benchmark and a coherent retrieval-based TLS framework, but the headline results are picked on test labels, so the empirical claims need a corrected evaluation before they can be trusted.","tokens_in":15562,"tokens_out":2797,"would_cite":true,"duration_ms":24401,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval loop in which an LLM repeatedly asks itself 5W1H questions about a news topic, rewrites them into search queries, and merges per-round timelines can match or beat dedicated timeline-summarization systems without a prebuilt…","keywords":["timeline summarization","retrieval-augmented generation","self-questioning","5W1H questions","question rewriting","open-domain news","few-shot prompting","Open-TLS dataset"],"falsifier":"Re-run the Open-TLS and closed-domain experiments with the self-questioning round and model chosen before seeing any test metric, say, a fixed round and a fixed model, or a validation split, and compare CHRONOS with DIRECT, REWRITE, and LLM-TLS; if the consistent improvements and the closed-domain parity disappear, the central claim is selection-dependent.","tokens_in":14572,"feed_emoji":"📰","tokens_out":9036,"duration_ms":78690,"temperature":0.7,"pith_summary":"This paper is trying to establish that timeline summarization for news does not need a pre-collected corpus: an LLM can build a chronology by repeatedly asking itself 5W1H questions about a target story, rewriting those questions into search-engine queries, retrieving fresh articles, and merging per-round timelines. The authors introduce Open-TLS, a dataset of 50 professionally authored timelines on recent events, to measure open-domain performance, and they compare the resulting system, CHRONOS, against two retrieval baselines and prior closed-domain methods. Across all models and metrics in the open-domain experiments, CHRONOS improves over direct search and over search with rewritten queries; on the closed-domain benchmarks Crisis and T17 it matches or exceeds the alignment-based ROUGE of existing state-of-the-art systems while using a small fraction of their inference time. If the claim holds, it means a general-purpose LLM equipped with a search API can do useful timeline summarization where no prebuilt news corpus exists.","feed_headline":"LLM self-questioning rivals curated timeline systems without a corpus","feed_subtitle":"Iterative 5W1H questions beat direct search and match specialized timeline systems using a fraction of the runtime.","key_machinery":"The load-bearing mechanism is the iterative self-questioning loop. Starting from a target headline, the LLM is prompted as an experienced journalist to ask at least five 5W1H questions (What, Who, Why, Where, When, How) that the current news database cannot answer; a rewriter decomposes each question into two or three search-ready queries; retrieved headlines and articles are read and stored; and a divide-and-conquer summarizer generates a timeline per round, then merges them by aligning dates and descriptions. Few-shot exemplars are selected by Chrono-Informativeness, a score that measures whether the documents a question retrieves produce dates matching a reference timeline (Date F1). The loop is what turns search into an event graph: each retrieved article is treated as evidence of a temporal or causal edge between the target event and another node.","core_discovery":"On the paper's own terms, the discovery is that retrieval quality, not the summarizer, is the bottleneck in timeline summarization, and that a self-questioning retrieval loop relieves that bottleneck. CHRONOS poses 5W1H questions about the target news and previously retrieved documents, rewrites each question into focused queries, searches the web or an offline corpus, and asks the LLM to generate and then merge timelines round by round. The paper reports consistent improvement across all metrics over DIRECT and REWRITE baselines on Open-TLS for GPT-3.5-Turbo, GPT-4o, and Qwen2.5-72B, and reports that on Crisis and T17 CHRONOS is comparable to LLM-TLS, the strongest published closed-domain system, while taking about 5.6 percent of its runtime on Crisis. It also claims that the ratio of correctly predicted dates (Date F1) is among the strongest, and that the Open-TLS dataset is larger and more topically diverse than existing public TLS benchmarks.","pith_inferences":["Editorial inference: if the reported gains survive a properly held-out selection of round count and model, the 5W1H self-questioning template is a reusable control strategy for temporal information-gathering tasks beyond news, such as biography generation or conflict tracking.","Editorial inference: the peak-then-decline curve across rounds suggests an adaptive stopping rule, stop when a new round retrieves no articles whose dates are absent from the current timeline, would be cheaper and more robust than a fixed round count.","Editorial inference: because Chrono-Informativeness is defined against reference timelines, the few-shot pool is tied to labeled data; replacing that score with a retrieval-diversity proxy would test whether the method's benefit comes from example selection or from the questioning loop itself.","Editorial inference: the efficiency comparison counts only inference time; end-to-end deployment would also pay search-API latency and cost, so the practical speed advantage may be smaller than the reported 5.6 percent figure."],"forward_implications":["Open-domain timeline summarization becomes practical: a search API plus an LLM can replace a curated corpus, so new topics can be summarized as they break.","The same retrieval loop transfers to closed-domain settings, where it matches prior event-graph and LLM-clustering systems at much lower inference cost.","Question rewriting is load-bearing: ablations show that removing the rewriter lowers Date F1 substantially on Open-TLS, Crisis, and T17.","Relevant few-shot exemplars matter: choosing examples by Chrono-Informativeness outperforms random exemplars and zero-shot prompting.","There is an optimal number of self-questioning rounds; performance rises then falls, so a system should stop before retrieval noise swamps the merge step."],"supporting_citations":[{"why":"Supplies the berrypicking-style model of iterative information search that motivates the self-questioning design.","marker":"Bates, 1989"},{"why":"Supplies the orienteering view of information seeking that justifies formulating questions and subquestions during retrieval.","marker":"O’Day and Jeffries, 1993"},{"why":"Provides the in-context learning basis for the few-shot prompts used in self-questioning.","marker":"Brown et al., 2020"},{"why":"Supplies the question-rewriting mechanism that decomposes complex queries before search.","marker":"Ma et al., 2023"},{"why":"Defines the Date F1 and alignment-based ROUGE metrics used in all reported comparisons.","marker":"Martschat and Markert, 2017, 2018"},{"why":"Provides the T17 benchmark used for closed-domain timeline summarization evaluation.","marker":"Binh Tran et al., 2013"},{"why":"Provides the Crisis benchmark used for closed-domain timeline summarization evaluation.","marker":"Tran et al., 2015b"},{"why":"Provides the CLUST baseline that CHRONOS is compared against on closed-domain benchmarks.","marker":"Gholipour Ghalandari and Ifrim, 2020"},{"why":"Provides the EGC event-graph baseline that CHRONOS is compared against on closed-domain benchmarks.","marker":"Li et al., 2021"},{"why":"Provides the LLM-TLS baseline, the strongest closed-domain system that CHRONOS claims to match at lower runtime.","marker":"Hu et al., 2024"}],"fun_headline_variants":["LLM self-questioning beats baselines for news timelines","CHRONOS: iterative self-questioning for timeline summaries","Self-questioning retrieval matches curated timeline systems","LLM asks own questions to rival specialized timeline tools","Without a corpus, self-questioning LLM builds strong timelines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validity of the headline comparison rests on the assumption that the optimal self-questioning round (Table 2) and the best-performing model (Section 5.4.2) were not selected using test-set labels; if they were, the claimed improvements are partly a consequence of that selection.","fun_headline_variants_meta":{"raw":{"variants":["LLM self-questioning beats baselines for news timelines","CHRONOS: iterative self-questioning for timeline summaries","Self-questioning retrieval matches curated timeline systems","LLM asks own questions to rival specialized timeline tools","Without a corpus, self-questioning LLM builds strong timelines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1282,"prompt_tokens":975,"completion_tokens":307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":225}},"tokens_in":591,"tokens_out":307,"duration_ms":3498,"temperature":1.0,"reasoning_tokens":225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:39:58.659893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Open-TLS and closed-domain experiments with the self-questioning round and model chosen before seeing any test metric, say, a fixed round and a fixed model, or a validation split, and compare CHRONOS with DIRECT, REWRITE, and LLM-TLS; if the consistent improvements and the closed-domain parity disappear, the central claim is selection-dependent.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Date F1 and alignment-based ROUGE metrics used in all reported comparisons."}],"review_version":1}