{"id":"513c2170-b4a3-4dd4-ae78-385d97548f8c","arxiv_id":"2502.01220","paper_version":6,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across 2,003 popular temporal facts, even the best of 18 language models perfectly distinguishes correct from incorrect years, months, or dates for only 11% of facts.","lead":"This paper builds a dataset of 2,000 popular dated facts and tests 18 language models on whether they prefer the right date over the wrong date for each fact. The best model is perfect on only 11% of facts, and all models sometimes favor dates far outside a fact's true validity period, suggesting language models store facts without a robust sense of when they were true.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 11% robustness headline depends on a single GPT-4o question per fact combined with an all-or-nothing per-fact metric; without a per-template sensitivity check the central number is not yet established as a stable property.","rationale":"The central claim is quantitative: the best LM is robust on only 11% of TimeStress facts. For that claim to hold as stated, the measurement must reflect the model's temporal knowledge rather than incidental properties of the single natural-language question attached to each fact. Given R's all-or-nothing definition, any per-fact noise source, such as awkward phrasing, an object that is a strong default answer, or a relation whose wording invites date-independent completion, can zero out a fact. The paper checks question quality on 50 samples, but it does not check variance across the four generated questions, even though all four exist in the dataset. This is the weakest link because it is upstream of every downstream analysis, including granularity transfer and rare-error anomalies, and because the released data make the check trivial. I do not see an internal inconsistency; the concern is that the headline number is not proven stable under verbalization. If the sensitivity check shows a small spread, the paper's claims stand. If it shows a large spread, the paper needs a multi-question consensus or per-template reporting. This does not change the reader's CONDITIONAL verdict.","tokens_in":24668,"tokens_out":6088,"duration_ms":645232,"concrete_test":"For gemma-2-27b-it and the other four most robust LMs, recompute per-fact and global robustness exactly as in Section 4.2, but repeat the computation four times, once for each of the four GPT-4o questions available per fact in TimeStress, holding all temporal contexts and comparison pairs fixed. Report min/max global R across the four runs, the fraction of facts robust under all four questions, and the fraction robust under at least one. If the spread across runs exceeds a few points, or the consensus-robust fraction differs materially from the reported 11%, the headline result is template-dependent and should be reported with a multi-question consensus metric rather than a single draw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.2 states that after generating four questions per fact, \"each fact is randomly assigned one of its four associated questions.\" Section 3.1 then defines R(M,f)=1 only when every correct-context statement beats every incorrect-context statement for that fact. With an average of 11 correct and 74 incorrect contexts (Section 3.2), a fact contributes to the 11% headline only if the model wins roughly 800 pairwise comparisons under a single fixed verbalization. This makes R an intersection over both temporal contexts and one arbitrary linguistic realization. The Appendix A.3 QA (1 error in 50 questions, Wilson CI [0.85, 0.99]) checks that questions are grammatical and factually answerable, not that the selected question is neutral across temporal contexts or that it does not inflate the probability of the object for dates outside the validity period. A single question with mild lexical ambiguity, an implicit default date, or a relation whose object is plausible in many years can zero out an entire fact even if the model knows the validity period perfectly. Consequently, the headline \"best LM achieves a perfect distinction for only 11% of facts\" is not yet distinguishable from \"best LM is perfect on 11% of facts under the one randomly chosen question\"; latent temporal robustness could be higher (or lower) once verbalization variance is removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TimeStress, a dataset of 2,003 past Wikidata facts with a total of 521,000 natural-language statements, and uses it to evaluate 18 language models on the task of preferring correct temporal contexts over incorrect ones. The evaluation defines a win rate W(M,f) and an all-or-nothing robustness metric R(M,f), and analyzes performance along two dimensions: the distance of an incorrect context from the fact's validity period, and the granularity of the context (year, year-month, year-month-day). The central empirical claim is that the best model, gemma-2-27b-it, achieves a global robustness of only about 11% of facts, meaning it prefers every correct context to every incorrect context for only a small fraction of the dataset. The paper further reports that LMs are vulnerable to 'easy' incorrect contexts far from the validity period even on facts with near-perfect win rates, and that knowledge does not fully transfer between granularities.","tokens_in":24908,"tokens_out":5453,"duration_ms":48377,"significance":"The paper makes a useful empirical contribution: it provides a publicly released, temporally annotated dataset with careful experimental detail, evaluates a diverse set of 18 open models, reports bootstrap confidence intervals, and gives a transparent account of the probability-based evaluation protocol in Appendix D. The headline result—that even the strongest model is robust on only about 11% of facts—is striking and, if stable, would be a meaningful data point for the community's understanding of temporal knowledge in LMs. The anomaly analysis (e.g., failures on distant incorrect contexts despite high win rates) is a falsifiable finding that can inform future benchmarks. However, the central number inherits a nontrivial sensitivity to the single verbalization used per fact, so the strength of the claim currently outruns the evidence.","major_comments":[{"comment":"The headline robustness figure is confounded with a single arbitrarily chosen question per fact. Section 3.2.2 states that 'each fact is randomly assigned one of its four associated questions,' while Section 3.1 defines R(M,f)=1 only when every correct-context statement beats every incorrect-context statement for that fact. With an average of 11 correct and 74 incorrect contexts, one fact contributes to the 11% only if the model wins roughly 800 pairwise comparisons under one fixed verbalization. The Appendix A.3 QA (1 error in 50 questions; Wilson CI [0.85,0.99]) verifies grammaticality and answerability, but not that the selected question is neutral across temporal contexts or that it does not inflate the probability of the object for dates outside the validity period. A single question with mild lexical ambiguity, an implicit default date, or a relation whose object is plausible in many years can zero out an entire fact even if the model knows the validity period perfectly. The authors should report robustness computed per question (e.g., the average of R over the four generated questions per fact, or the distribution across questions) to show that the 11% is not an artifact of this random selection.","section":"§3.2.2 and §3.1"},{"comment":"The granularity-transfer analysis inherits the same single-question sensitivity. The 'known' definition used in Figure 6 is R(M,f)=1 for a given granularity, and the claim that LMs fail to generalize across granularities in 28% of cases is based on this all-or-nothing condition. Since R=1 requires winning all comparisons under the one assigned question for that fact, a fact that is robust under the selected question need not be robust under another of the four generated questions. The observed transfer rates (e.g., 74±5% for gemma-2-27b-it from Y to YM) could therefore partly reflect question-level noise rather than a genuine inability to propagate temporal knowledge across granularities. The authors should recompute the transfer matrices using an aggregate over the four questions (majority voting or average R) to demonstrate that the reported gap is robust to verbalization choice.","section":"§4.2, Figure 6"}],"minor_comments":[{"comment":"The caption contains the garbled fragment 'Thetrophymeans'; the intended symbol is not rendered. Please clarify the caption and ensure the figure displays the checkmark/cross symbols correctly.","section":"Figure 1 caption"},{"comment":"The text refers to 'Table 4' for the vulnerability-to-easy-contexts analysis, but the displayed table is labeled 'Figure 4.' Update the cross-reference and the surrounding citations for consistency.","section":"Table 4 reference"},{"comment":"The labels in Figure 10 use the French word 'connu' (e.g., 'P(X connu | Y connu)') while the rest of the paper uses 'known.' Please localize these labels to English.","section":"Appendix C.3, Figure 10"},{"comment":"The Gemma Team reference lists 'et el' instead of 'et al.'; correct this typo.","section":"References"},{"comment":"The paper uses 'MLs' (e.g., 'the 5 most robust MLs') in several places; these should be 'LMs' to match the terminology used throughout the manuscript.","section":"Appendix C.1 and Figure 8"},{"comment":"The phrase 'errors, certainly rare, but critical that humans would not make' is grammatically ambiguous; consider rewriting, e.g., 'errors that are rare but critical, and that humans would not make.'","section":"Abstract and §4.2"}],"recommendation":"major_revision","confidential_remarks":"The central 11% claim is potentially important, but the single-question confound is easy to address because the authors already have four questions per fact. A revision that reports per-question robustness (or an average over the four questions) would substantially strengthen the paper. The self-citation to Ammar Khodja et al. (2025) for the Wikidata preprocessing and popularity measure is legitimate but should be carefully scoped: the popularity measure is from the same group, and the paper should not give the impression that its validity is independently established. Overall, the dataset and experimental detail are solid; the required changes are local to the analysis of the headline metric."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. TimeStress is a genuinely useful artifact: 2,003 Wikidata facts, 521k statements, 18 models, all three granularities, and the code and data are public under permissive licenses. The win-rate results and the distance/granularity analyses are competently done and survive scrutiny. The paper earns its place as a benchmark-plus-failure-mode contribution.\n\nThe soft spot is exactly where the stress-test note lands. The headline \"best LM achieves perfect distinction for only 11% of facts\" is the most fragile number in the paper, not the most important one. Robustness R requires every correct context to beat every incorrect context under a single randomly assigned GPT-4o question. With roughly 11 correct and 74 incorrect contexts per fact, that is hundreds of pairwise comparisons, all won with one fixed verbalization. The Appendix A.3 QA checks that questions are grammatical and factually answerable; it does not check whether the chosen question is neutral across temporal contexts. One subtly awkward question can zero out a fact even if the model knows the validity period cold. So the 11% is not yet cleanly separable from \"one randomly chosen verbalization removed the fact.\" A per-template sensitivity analysis, or averaging over all four generated questions, would settle this. That is the main thing I would ask for in review.\n\nTwo smaller reservations. First, the \"errors humans would not make\" claim has no human baseline, and the paper itself admits the transitional-context probability pattern is a hypothesis. The anomaly framing is a bit stronger than the evidence. Second, Wikidata validity periods are treated as ground truth, which is reasonable for a benchmark but worth stating as a limitation more prominently.\n\nI do not think these flaws sink the paper. The win-rate data, the far-distance errors, and the granularity transfer failure are all more robust than the all-or-nothing headline, and they align with a plausible conclusion: LMs have coarse temporal knowledge but not reliable temporal binding. The methodological point about all-or-nothing metrics over many comparisons is itself worth publishing.\n\nRecommendation: send it to serious peer review. Require the template-sensitivity analysis and a human or gold-standard comparison for the anomaly claim. The dataset alone justifies the referee time.","headline":"A solid, transparent empirical study with a reusable dataset; the 11% headline is real but hostage to one GPT-4o question per fact, so treat it as provisional.","tokens_in":25488,"tokens_out":1592,"would_cite":true,"duration_ms":17703,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even the most robust language model tested, gemma-2-27b-it, assigns the highest probability to the correct temporal context for only about 11 percent of 2,003 popular historical facts, and most models score below 3 percent.","keywords":["temporal robustness","factual knowledge","language models","TimeStress","temporal context","knowledge probing","robustness metric"],"falsifier":"Recompute the all-granularity robustness of gemma-2-27b-it on TimeStress using ten independently generated, human-verified questions per fact instead of one; if the aggregate robustness rises well above 11 percent, the low score is partly a verbalization artifact. A complementary check is to take 100 facts with manually verified validity intervals and see whether the same win-rate-by-distance curve and the persistent failures at relative distances of four or more still appear.","tokens_in":24447,"feed_emoji":"🕰️","tokens_out":9545,"duration_ms":78201,"temperature":0.7,"pith_summary":"This paper asks whether language models can tell when a fact is true. The authors build TimeStress, a dataset of 2,003 popular historical facts with natural-language statements paired against correct and incorrect temporal contexts at three granularities: year, month, and day. They test 18 models by asking, for each pair of statements, which one the model considers more probable. The central finding is that models have a coarse sense of time but no robust one: the best model, gemma-2-27b-it, prefers the correct context over every incorrect context for only about 11 percent of facts, and most models stay below 3 percent. If this holds, current language models cannot be trusted to answer when a fact held, even for facts they plainly know.","feed_headline":"Best language model keeps every date correct for 11% of facts","feed_subtitle":"A 521,000-question stress test finds even the top model rejects a wrong year, month, or day for most popular facts.","key_machinery":"The central object is the pairwise temporal-context comparison protocol instantiated by TimeStress. For each fact (s, r, o, [a, b]), the dataset constructs statements of the form 'In [tau], [question about s and r]? o' and samples many correct and incorrect contexts at year, year-month, and year-month-day granularity, with equal counts across granularities and a relative-distance measure alpha = (midpoint(tau) - midpoint([a, b])) / duration([a, b]). A model is said to distinguish a correct from an incorrect context when Pr_M(o | s, r, tau+) > Pr_M(o | s, r, tau-). These pairwise wins are aggregated into a per-fact win rate W(M, f), and robustness R(M, f) is the indicator that all comparisons are won; the dataset is large enough, with about 11 correct and 74 incorrect contexts per fact on average, that random guessing essentially never produces a robust fact. This protocol is what lets the paper separate coarse temporal sensitivity from reliable temporal binding.","core_discovery":"The paper's central claim is that the factual knowledge stored in language models is only weakly bound to its validity period. Models assign higher probability to an answer when the stated date lies inside a fact's validity interval, and that probability decays as the date moves away, evidence of a genuine but coarse temporal representation. Yet robustness, defined as winning every correct-versus-incorrect comparison for a fact, is almost never achieved: gemma-2-27b-it, the most robust of the 18 models, scores 11 percent when all granularities are combined and 17 percent for years alone, while nearly all other models stay under 3 percent. The failures are not confined to dates near the boundaries of the validity period; even facts with a 95 percent win rate are sometimes defeated by dates very far away, with errors persisting at relative distances of four validity periods or more. The paper also shows that knowing a fact at year granularity does not reliably transfer to month or day granularity, with average transfer rates around 72 percent and no model achieving perfect transfer.","pith_inferences":["Because each fact is assigned a single randomly chosen question, the 11 percent figure conflates temporal robustness with verbalization robustness; evaluating the same facts with several human-verified phrasings would separate the two and could move the number either way.","The pairwise-preference design measures what a model would choose under forced comparison, not what it would volunteer in open generation, so applications that let a model abstain or retrieve a timestamp may behave better than these numbers suggest.","A natural next probe is to shift the temporal phrase from the beginning of the statement to the end, or embed it in a relative clause, to test whether the observed failures are tied to the 'In [year],' surface pattern.","If temporal binding is this weak for popular facts, temporal knowledge editing should target validity intervals rather than only object replacement, and evaluation should include the granularity-transfer and far-date failure modes introduced here."],"forward_implications":["Average win rates of 78 to 87 percent on the top five models show that language models are not temporally blind; the failure is specifically a lack of consistency across all comparisons.","Because robustness requires every comparison to be won, an all-or-nothing score near 11 percent means even the best model has only a fragile hold on when facts hold.","Instruction-tuned models usually beat their base versions, but reformatting statements as chat instructions does not reliably improve robustness and can degrade it for some models.","Vulnerability to easy incorrect contexts persists at relative distances of four validity periods or more even when a fact's win rate is 99 percent, so the errors are not just boundary effects.","Knowledge at one granularity transfers to another in roughly 72 percent of cases on average, with coarse-to-fine transfers failing more often, and adding explanatory prompts about date hierarchies improves this only slightly."],"supporting_citations":[{"why":"It supplies the preprocessed knowledge-base snapshot and the entity popularity index that TimeStress builds on.","marker":"Ammar Khodja et al. (2025)"},{"why":"It provides the factual-knowledge probing paradigm that the paper's probability-based preference protocol extends to temporal contexts.","marker":"Petroni et al. (2019)"},{"why":"It contributes the prompt template adapted to generate the natural-language questions used in TimeStress.","marker":"Ammar Khodja et al. (2024)"},{"why":"It supports the assumption that probability-based factual evaluation tracks the model's generative knowledge, which the protocol relies on.","marker":"Dong et al. (2023)"},{"why":"It grounds the claim that popular facts are more likely to be learned, which backs the paper's upper-bound interpretation of its results.","marker":"Kandpal et al. (2023)"},{"why":"It also grounds the popularity-knowledge link used to argue that the studied facts are among those language models should know best.","marker":"Kang and Choi (2023)"}],"fun_headline_variants":["Best LM keeps dates correct for only 11% of facts","Even top language model fails to date most facts","Temporal knowledge weak: best model scores 11% on dates","Only 11% of facts pass date check for strongest LM","Language models crumble when asked to verify fact dates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single automatically generated question assigned to each fact is a faithful and neutral expression of that fact and that the knowledge base's recorded validity window is complete, so a fact with a slightly off wording or an incomplete validity window can score zero robustness even when the model has correct temporal knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Best LM keeps dates correct for only 11% of facts","Even top language model fails to date most facts","Temporal knowledge weak: best model scores 11% on dates","Only 11% of facts pass date check for strongest LM","Language models crumble when asked to verify fact dates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1349,"prompt_tokens":891,"completion_tokens":458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":507,"tokens_out":458,"duration_ms":4886,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:00:28.362718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the all-granularity robustness of gemma-2-27b-it on TimeStress using ten independently generated, human-verified questions per fact instead of one; if the aggregate robustness rises well above 11 percent, the low score is partly a verbalization artifact. A complementary check is to take 100 facts with manually verified validity intervals and see whether the same win-rate-by-distance curve and the persistent failures at relative distances of four or more still appear.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the preprocessed knowledge-base snapshot and the entity popularity index that TimeStress builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It contributes the prompt template adapted to generate the natural-language questions used in TimeStress."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It grounds the claim that popular facts are more likely to be learned, which backs the paper's upper-bound interpretation of its results."}],"review_version":1}