Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Around the World in 24 Hours: Probing LLM Knowledge of Time and Place

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models that are nearly perfect at clock arithmetic collapse when the same question crosses a time zone, not because they lack the time-zone facts but because they fail to retrieve and combine them.

desk verdict GeoTemp is a reusable benchmark with a real central result, but the paper overstates the confidence in the 'can't combine knowledge' diagnosis for smaller models. read the letter →

arxiv 2506.03984 v1 pith:RT7NI2PM submitted 2025-06-04 cs.CL

Carolin Holtermann , Paul Röttger , Anne Lauscher This is my paper
classification cs.CL
keywords geotemporalreasoningtimezoneslargelanguagemodelstemporalspatialevaluationbenchmarkknowledgeretrievalpromptsensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can reason about time and place together, using world time zones as the testbed. It introduces GeoTemp, a set of 320k synthetic prompts spanning 289 cities, 217 countries, and 37 time zones, and evaluates eight open chat models on four tasks that separate pure comprehension, pure clock arithmetic, and time-zone conversion. The central finding is a sharp gap: models are nearly perfect when adding hours to a clock in one location, but the best model reaches only 56% when a second city is involved and 33% when the question combines a time shift with a location change. Because direct probing shows the same models know the correct UTC offset for 90% of locations, the paper argues that the failure is not missing geographic facts but the inability to retrieve and combine knowledge during a compound task. The argument matters because it separates "does not know" from "cannot use what it knows," which changes where future work should aim.

What carries the argument

The central object is GeoTemp, a synthetic prompt dataset built from the Olson Time Zone Database, with 332,928 prompts that pair 289 precise locations across 37 UTC offsets and 217 countries and use randomized dates and times from 2023 as ground truth computed by the pytz library. The four task templates form the analytical ladder that carries the argument: VERIFICATION (repeat a given time), TIMETIME (add hours in one location), TIMEPLACE (same moment, two locations), and TIMETIMEPLACE (hour shift plus two locations). By comparing a model's accuracy across these rungs and against a direct UTC-offset probe, GeoTemp isolates whether a failure is due to comprehension, arithmetic, factual geographic knowledge, or the act of combining two knowledge types. A regex-based answer parser validated to at least 98% agreement with human labels is what makes open-ended evaluation of this size practical.

What would settle it

Give the same models the same TIMEPLACE problems as a forced choice between two candidate times: if accuracy rises to near TIMETIME levels, the open-ended generation format, not the combining of temporal and geographic knowledge, is the main obstacle, and the paper's central attribution would be falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLM geotemporal competence is fragmented. The four GeoTemp tasks form a ladder: VERIFICATION asks the model to repeat the time it was given; TIMETIME adds an hour offset in one place; TIMEPLACE converts between two places at the same moment; TIMETIMEPLACE applies both an hour shift and a place change. Accuracy collapses exactly at the two rungs that require linking a place to a UTC offset, and the pattern holds across Llama2, Llama3, and Qwen2 at multiple sizes. The collapse is not a simple knowledge deficit: when asked directly, models above the smallest size assign the correct UTC offset to at least 65% of locations, with the best at 90%, and injecting those offsets into the prompt recovers most of the lost performance. The paper therefore concludes that the limiting factor is the retrieval and joint activation of knowledge the model already has, not the absence of that knowledge.

Load-bearing premise

The evaluation assumes that near-chance accuracy on TIMEPLACE and TIMETIMEPLACE means the model cannot combine time and place reasoning, rather than that the synthetic prompt wording or the answer parser hides a correct answer.

Editorial extensions

If this is right

  • If correct, benchmarks that test temporal or geographic reasoning in isolation will overstate real-world LLM competence on questions that require both.
  • Scaling model size improves but does not close the gap: the largest open models remain far below ceiling on TIMEPLACE and TIMETIMEPLACE, arguing against an expectation that scale alone solves compound reasoning.
  • Prompt design is not a neutral detail: supplying time-zone offsets in the prompt substantially improves results, while forcing step-by-step explanations can degrade simple tasks.
  • Disparities in performance across locations track training-data exposure as proxied by perplexity more than country, income, or region, so location-specific fairness evaluations should account for exposure.
  • Knowledge injection works best where models already show direct time-zone knowledge, which suggests the effect is activation of existing memory rather than new information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, this retrieval-versus-knowledge split may generalize to other compound real-world tasks, such as combining exchange rates with dates or shipping times with daylight-saving rules, where each individual fact is known but joint use fails; a direct test would reuse GeoTemp's template with other fact pairs.
  • A natural next experiment is to force the model to externally verbalize both UTC offsets before computing, for example by asking it to first state the offset of each city and then answer; if that recovers accuracy as well as injecting offsets, the bottleneck is internal attention rather than missing knowledge.
  • The perplexity gradient suggests a caution for benchmark design: prompt sets built from many obscure places may underestimate genuine reasoning ability, since performance is partly a frequency artifact of training exposure.
  • Because the current tasks are synthetic, a useful validation would be to collect naturally occurring time-conversion questions from travel or scheduling forums and compare performance, in case real-world phrasings activate knowledge differently than the template.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces GeoTemp, a 320k-prompt benchmark spanning 289 locations, 217 countries, and 37 time zones, and evaluates eight open chat LLMs on four tasks (VERIFICATION, TIMETIME, TIMEPLACE, TIMETIMEPLACE). The central empirical claim is that models perform well on temporal-only reasoning and scale with size, but accuracy collapses on tasks that require combining temporal and geographic knowledge; the authors attribute this to a failure to retrieve and combine separately known facts rather than to missing time-zone knowledge. Secondary analyses examine regional bias, location-name perplexity, error categories, and the effect of injecting time-zone information.

Significance. If the central interpretation is correct, the finding that open models know individual geotemporal facts yet fail to compose them is an important, falsifiable benchmark result for the community. Strengths include the systematic construction from OTZD/pytz ground truth, the validated regex parser with human agreement, the breadth of eight models across three families, and the robustness checks against prompt variation. The dataset release is a useful resource. However, the strongest interpretation—that the accuracy gap measures knowledge-combination ability rather than prompt-adoption artifacts—requires additional controls that are currently missing.

major comments (3)
  1. [§5.1, Figure 4, Appendix D] The claim that TIMEPLACE/TIMETIMEPLACE accuracy measures the cost of adding geographic knowledge is not fully identified because the VERIFICATION control, which requires no time-zone knowledge and no arithmetic, is below ceiling for several models (e.g., Llama2-7B/13B and Qwen2-1.5B/7B in Figure 4). Appendix D shows concrete failures to adopt the prompt's hypothetical frame: Qwen2-72B refuses to answer because it cannot provide real-time information, and Llama2-70B converts the 'now' time to UTC under chain-of-thought. The Section 6.4 UTC probe is not a matched control: it asks for a static UTC offset without the 'Today is ... in 2023' frame and without a second location. To support the 'knows facts but cannot combine' conclusion, the authors should report TIMEPLACE and TIMETIMEPLACE accuracy conditioned on instances where the same model answers VERIFICATION correctly, and ideally administer the UTC probe in the same temporal frame.
  2. [§6.2 and §7] The abstract and Section 6.2 describe 'significant' correlations between location-name perplexity and accuracy, and Section 7 describes 'significant' performance improvements from time-zone injection, but no significance tests, confidence intervals, or effect-size statistics are provided. Figure 5 shows nontrivial variance across prompt variations, and the perplexity heatmaps in Appendix C.3 report raw percentages only. Since these claims support RQ2 and RQ3, the authors should quantify uncertainty (e.g., paired bootstrap over instances or models) before using the word 'significant'.
  3. [§6.3] The error analysis for Llama3-70B is based on 200 incorrect responses annotated by a single author, with no inter-annotator agreement for the error categories and no breakdown by instruction type. These categories are used to argue that the dominant failure is incorrect UTC retrieval rather than DST handling or arithmetic, so the authors should report annotation reliability and ideally have a second annotator score the same sample.
minor comments (5)
  1. [§8] The Discussion contains a typo: 'TEMPTEMP' should be 'TIMETIME'.
  2. [Figures 4, 14–16] Figure labels mix 'TimexTime', 'TimexPlace', and 'TimexTimexPlace' with the task names used in Table 1; please normalize the notation.
  3. [§4.2 and Table 5] Table 5 reports evaluation on 100 prompts per model, but Section 4.2 describes a 3,600-response annotation set; state explicitly that the regex validation subset is 100 responses per model.
  4. [§6.2] The perplexity proxy is acknowledged in the Limitations, but the paper should also state that perplexity is computed on a single sentence template, so the ranking may be sensitive to the template wording; a short robustness check with a second template would strengthen the claim.
  5. [§5.4] The robustness analysis samples 1,000 instances but does not report whether the sample is stratified by location, time zone, or task; a sentence describing the sampling procedure would clarify the interpretation of the error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GEOTEMP labels come from an external database, and the central empirical claims are measured rather than assumed.

full rationale

The paper's derivation chain is not circular. All ground-truth labels are computed from the external Olson Time Zone Database via pytz and datetime, not from any model output or fitted parameter, so the accuracy numbers are genuinely measured. The four tasks (VERIFICATION, TIMETIME, TIMEPLACE, TIMETIMEPLACE) are constructed from templates and sampled location pairs, but the difficulty ordering is not imposed by the construction; it is an empirical finding. The regex evaluator is validated against human annotations with at least 98% accuracy, so the evaluation pipeline is not a hidden restatement of the result. The perplexity analysis in Section 6.2 uses the models' own perplexity as an explanatory variable, but the paper only reports a correlation with measured accuracy rather than deriving accuracy from perplexity, so this is a mild self-referential covariate, not a circular derivation. Section 6.4's UTC probe is an additional independent measurement, not a fitted input renamed as a prediction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The skeptic's concern about the VERIFICATION task being below ceiling for some models is a genuine validity threat to the 'cannot combine' interpretation, but it concerns whether low accuracy reflects reasoning failure or prompt miscomprehension; it does not make any claim true by construction. Thus the paper is self-contained against external benchmarks and merits a circularity score of 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or new physical entities. It relies on an external time-zone database for labels, a validated regex parser, and a perplexity proxy for training exposure, all explicitly stated. The main free choices are dataset design decisions (location selection, prompt templates) rather than parameters fitted to make a claim true.

assumptions (3)
  • domain assumption The Olson Time Zone Database (OTZD) provides correct UTC offsets and DST rules for all locations and dates in the dataset.
    Ground truth labels are computed using pytz with OTZD data (Section 3.1, Appendix A).
  • domain assumption The regex matching algorithm extracts the model's intended answer with negligible error.
    Validated on 3,600 human-annotated responses, reaching at least 98% accuracy (Section 4.2, Appendix B).
  • ad hoc to paper The perplexity of 'I live in {city}, {country}' is a valid proxy for the model's training exposure to a location.
    Introduced in Section 6.2; the authors acknowledge in Limitations that this proxy 'is not fully reliable and may introduce inaccuracies'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Around the World in 24 Hours: Probing LLM Knowledge of Time and Place." pith.science (2026). https://pith.science/paper/RT7NI2PM

@misc{pith2026250603984,
  author       = {Pith},
  title        = {Pith review of: Around the World in 24 Hours: Probing LLM Knowledge of Time and Place},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RT7NI2PM}},
  note         = {Machine review of arXiv:2506.03984}
}
read the original abstract

Reasoning over time and space is essential for understanding our world. However, the abilities of language models in this area are largely unexplored as previous work has tested their abilities for logical reasoning in terms of time and space in isolation or only in simple or artificial environments. In this paper, we present the first evaluation of the ability of language models to jointly reason over time and space. To enable our analysis, we create GeoTemp, a dataset of 320k prompts covering 289 cities in 217 countries and 37 time zones. Using GeoTemp, we evaluate eight open chat models of three different model families for different combinations of temporal and geographic knowledge. We find that most models perform well on reasoning tasks involving only temporal knowledge and that overall performance improves with scale. However, performance remains constrained in tasks that require connecting temporal and geographical information. We do not find clear correlations of performance with specific geographic regions. Instead, we find a significant performance increase for location names with low model perplexity, suggesting their repeated occurrence during model training. We further demonstrate that their performance is heavily influenced by prompt formulation - a direct injection of geographical knowledge leads to performance gains, whereas, surprisingly, techniques like chain-of-thought prompting decrease performance on simpler tasks.

Figures

Figures reproduced from arXiv: 2506.03984 by the authors.

Figure 1
Figure 1. Combinations of temporal and geographic knowledge we test with GEOTEMP. Models struggle most with jointly reasoning over space and time. system, time zones based on geographic location, and daylight saving time to align activities across different regions (Hayden, 1987). These conven￾tions, deeply rooted in our understanding of the world, play a crucial role in decision-making, espe￾cially in today’s globalized work… view at source ↗
Figure 2
Figure 2. Geographical distribution of all 289 loca￾tions in GEOTEMP. Color indicates UTC time zone. <500 <50K <100K <500K <1M <5M >5M 0 5 10 15 20 25 % of all Locations [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of cities included in GEOTEMP according to their number of inhabitants. For each bin, we show the percentage of locations in our dataset that belong to this bin. by applying every task template of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Model accuracy on GEOTEMP. We show the percent of correct model responses (obtained via major￾ity vote across different instruction types) for different models for our four geotemporal tasks (VERIFICATION, TIMETIME, TIMEPLACE, TIMETIMEPLACE). agreement on the labeling …
Figure 5
Figure 5. Figure 5: Robustness analysis. We show the mean and standard deviation of model answers across three differ￾ent prompt variations. In (a), the results for the neutral instruction type across all models. In (b) we show the results obtained by Llama3-70B for each instruction type …
Figure 6
Figure 6. Figure 6: Model accuracy aggregated by target coun￾try for the tasks TIMEPLACE and TIMETIMEPLACE. We neither observe biases towards Western nor towards higher-income or population-dense countries. perplexity scores into four bins. Specifically, for each model, we determine the p…
Figure 8
Figure 8. Figure 8: Results of our error analysis. We show the proportion of model errors (in %) per error category. sample 200 of the model’s incorrect responses on the TIMETIMEPLACE task. One author then an￾notated this sample, assigning errors to one of five error categories ( [PITH_F…
Figure 9
Figure 9. Figure 9: Results of UTC knowledge injection on the TIMETIMEPLACE task. Colors respond to different amounts of knowledge injected. adding the time zone information, and Qwen2-72B of 53%. We also find that models do not perform as well when city information is replaced with time …
Figure 10
Figure 10. Figure 10: Histogram Country Population Low income Lower middle income Upper middle income High income Not classified 0 10 20 30 40 Distribution of City Income Level % of all Locations [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Histogram Country Income Level 0 Africa Asia Oceania South America North America Europe Antarctica 5 10 15 20 Distribution of Continents % of all Locations [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 13
Figure 13. Figure 13: Histogram UTC of Locations [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Model Accuracy neutral prompt 7b 13b 70b 8B 70B 1.5B 7B 72B 0 20 40 60 80 100 Verification TimexTime TimexPlace TimexTimexPlace Correct Responses in % Llama 2 Llama 3 Qwen 2 (a) Overall Results 7b 13b 70b 8B 70B 1.5B 7B 72B mini 0 20 40 60 80 100 Verification TimexTim…
Figure 15
Figure 15. Figure 15: Model Accuracy short prompt 7b 13b 70b 8B 70B 1.5B 7B 72B 0 20 40 60 80 100 Verification TimexTime TimexPlace TimexTimexPlace Correct Responses in % Llama 2 Llama 3 Qwen 2 (a) Overall Results 7b 13b 70b 8B 70B 1.5B 7B 72B mini 0 20 40 60 80 100 Verification TimexTime …
Figure 16
Figure 16. Figure 16: Model Accuracy CoT prompt 6 https://platform.openai.com/docs/overview [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Model accuracy aggregated by start country for the tasks TIMEPLACE and TIMETIMEPLACE. We neither observe biases towards Western nor towards higher-income or population-dense countries [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]
Figure 18
Figure 18. Figure 18: Llama2 accuracy distribution for different City Perplexities [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 19
Figure 19. Figure 19: Llama3 accuracy distribution for different City Perplexities [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 20
Figure 20. Figure 20: Qwen2 accuracy distribution for different City Perplexities [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Giuseppe Attanasio. 2023. S imple G eneration. https://github.com/MilaNLProc/simple-generation

    work page 2023

  5. [5]

    Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W

    Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. https://doi.org/10.1162/tacl_a_00459 Time-aware language models as temporal knowledge bases . Transactions of the Association for Computational Linguistics, 10:257--273

    doi:10.1162/tacl_a_00459 2022

  6. [6]

    Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2024. Faith and fate: limits of transformers on compositionality. In Proceedings of the 37th International Conferenc...

    work page 2024

  7. [7]

    Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. https://arxiv.org/abs/2303.10130 Gpts are gpts: An early look at the labor market impact potential of large language models . Preprint, arXiv:2303.10130

    arXiv 2023

  8. [8]

    Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2024. https://arxiv.org/abs/2406.09170 Test of time: A benchmark for evaluating llms on temporal reasoning . Preprint, arXiv:2406.09170

    arXiv 2024

Show all 35 references
  1. [9]

    Nathan Godey, \'E ric de la Clergerie, and Beno \^ t Sagot. 2024. https://aclanthology.org/2024.lrec-main.1087 On the scaling laws of geographical representation in language models . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Langua...

    2024

  2. [10]

    Wes Gurnee and Max Tegmark. 2024. https://arxiv.org/abs/2310.02207 Language models represent space and time . Preprint, arXiv:2310.02207

    2024 arXiv

  3. [11]

    Gregory Hayden

    F. Gregory Hayden. 1987. http://www.jstor.org/stable/4225926 Evolution of time constructs and their impact on socioeconomic planning . Journal of Economic Issues, 21(3):1281--1312

    1987

  4. [12]

    Smith, Yejin Choi, and Kentaro Inui

    Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2024. Realtime qa: what's the answer right now? In Proceedings of the 37th International Conference on Neural Informati...

    2024

  5. [13]

    Richard Landis and Gary G

    J. Richard Landis and Gary G. Koch. 1977. http://www.jstor.org/stable/2529310 The measurement of observer agreement for categorical data . Biometrics, 33(1):159--174

    1977

  6. [14]

    Ruotong Liao, Max Erler, Huiyu Wang, Guangyao Zhai, Gengyuan Zhang, Yunpu Ma, and Volker Tresp. 2024. https://doi.org/10.48550/arXiv.2409.20365 Videoinsta: Zero-shot long video understanding via informative spatial-temporal reasoning with llms

    2024 doi arXiv

  7. [15]

    Bastien Li \'e tard, Mostafa Abdou, and Anders S gaard. 2021. https://doi.org/10.18653/v1/2021.blackboxnlp-1.40 Do language models know the way to R ome? In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 510--517, Pu...

    2021 doi

  8. [16]

    Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D'Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-Mcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. https://proceedings.mlr.press/v...

    2022

  9. [17]

    Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon. 2024 a . https://arxiv.org/abs/2402.02680 Large language models are geographically biased . Preprint, arXiv:2402.02680

    2024 arXiv

  10. [18]

    Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, and Stefano Ermon. 2024 b . https://arxiv.org/abs/2310.06213 Geollm: Extracting geospatial knowledge from large language models . Preprint, arXiv:2310.06213

    2024 arXiv

  11. [19]

    Mazda Moayeri, Elham Tabassi, and Soheil Feizi. 2024. https://doi.org/10.1145/3630106.3658967 Worldbench: Quantifying geographic disparities in llm factual recall . In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, page 1211–12...

    2024

  12. [20]

    Kai Nylund, Suchin Gururangan, and Noah Smith. 2024. https://doi.org/10.18653/v1/2024.acl-long.141 Time is encoded in the weights of finetuned language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2024 doi

  13. [21]

    James Pustejovsky, Patrick Hanks, Roser Saurí, Andrew See, Rob Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. 2003. The timebank corpus. Proceedings of Corpus Linguistics

    2003

  14. [22]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations

    2024

  15. [23]

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. 2024. https://arxiv.org/abs/2409.12183 To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning . Prep...

    2024 arXiv

  16. [24]

    Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.acl-long.828 Towards benchmarking and improving the temporal reasoning capability of large language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023 doi

  17. [25]

    Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2024. https://doi.org/10.18653/v1/2024.findings-acl.374 Towards robust temporal reasoning of large language models via a multi-hop QA dataset and pseudo-instruction tuning . In Findings of the Association for Computational Linguistics ...

    2024 doi

  18. [26]

    Shivin Thukral, Kunal Kukreja, and Christian Kavouras. 2021. https://doi.org/10.18653/v1/2021.blackboxnlp-1.31 Probing language models for understanding of temporal expressions . In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks fo...

    2021 doi

  19. [27]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  20. [28]

    Siddharth Vashishtha, Adam Poliak, Yash Kumar Lal, Benjamin Van Durme, and Aaron Steven White. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.363 Temporal reasoning in natural language inference . In Findings of the Association for Computational Linguistics: EMNLP 2020,...

    2020 doi

  21. [29]

    Yuqing Wang and Yun Zhao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.382 TRAM : Benchmarking temporal reasoning for large language models . In Findings of the Association for Computational Linguistics ACL 2024, pages 6389--6415, Bangkok, Thailand and virtual meeting. ...

    2024 doi

  22. [30]

    Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.100 M enat QA : A new dataset for testing the temporal comprehension and reasoning abilities of large language models . In Fi...

    2023 doi

  23. [31]

    Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. https://aclanthology.org/2024.acl-long.563 Large language models can learn temporal reasoning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024

  24. [32]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  25. [33]

    Michael Zhang and Eunsol Choi. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.586 S ituated QA : Incorporating extra-linguistic contexts into QA . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371--7387, Online and Punta C...

    2021 doi

  26. [34]

    Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah A. Smith. 2024. https://arxiv.org/abs/2402.16797 Set the clock: Temporal alignment of pretrained language models . Preprint, arXiv:2402.16797

    2024 arXiv

  27. [35]

    Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. https://doi.org/10.18653/v1/D19-1332 `` going on a vacation '' takes longer than `` going for a walk '' : A study of temporal commonsense understanding . In Proceedings of the 2019 Conference on Empirical Methods in Na...

    2019 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.