REVIEW 10 cited by
A Dataset for Answering Time-Sensitive Questions
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Time is an important dimension in our physical world. Lots of facts can evolve with respect to time. For example, the U.S. President might change every four years. Therefore, it is important to consider the time dimension and empower the existing QA models to reason over time. However, the existing QA datasets contain rather few time-sensitive questions, hence not suitable for diagnosing or benchmarking the model's temporal reasoning capability. In order to promote research in this direction, we propose to construct a time-sensitive QA dataset. The dataset is constructed by 1) mining time-evolving facts from WikiData and aligning them to their corresponding Wikipedia page, 2) employing crowd workers to verify and calibrate these noisy facts, 3) generating question-answer pairs based on the annotated time-sensitive facts. Our dataset poses challenges in the aspect of both temporal understanding and temporal reasoning. We evaluate different SoTA long-document QA systems like BigBird and FiD on our dataset. The best-performing model FiD can only achieve 46\% accuracy, still far behind the human performance of 87\%. We demonstrate that these models are still lacking the ability to perform consistent temporal reasoning. Therefore, we believe that our dataset could serve as a benchmark to develop NLP models more sensitive to temporal shifts. The dataset and code are released in~\url{https://github.com/wenhuchen/Time-Sensitive-QA}.
Forward citations
Cited by 10 Pith papers
-
WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
LLMs answer simple waveform queries well but fail on multi-signal temporal reasoning, and event-time JSON waveforms beat standard VCD by 37–53% in accuracy, according to a new 360-question benchmark.
-
Chaining Event Spans for Temporal Relation Grounding
A new two-step model chains question-answer evidence across a question group to predict an event timeline, improving temporal reading comprehension and relation extraction.
-
ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models
Models leak future knowledge despite explicit temporal cutoffs, as quantified by the ExAnte benchmark across four tasks.
-
Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models
Temporal Attractor Steering resolves 29-57% of parametric temporal conflicts in open-weight LLMs while preserving 85-99% accuracy on non-conflict queries.
-
Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation
LLM judges frequently mark candidates wrong when the gold reference contradicts the model's own knowledge, even if the candidate exactly matches the provided reference.
-
Evaluating List Construction and Temporal Understanding capabilities of Large Language Models
A new benchmark shows LLMs give incomplete lists and inaccurate time intervals for temporal list questions, and retrieval helps only partly.
-
DailyQA: A Benchmark to Evaluate Web Retrieval Augmented LLMs Based on Capturing Real-World Changes
DailyQA, automatically built from Wikipedia revision logs, shows that LLMs with web retrieval answer only about half of time-sensitive questions correctly, and that reranking documents outperforms both snippets and ti...
-
Discrete Minds in a Continuous World: Do Language Models Know Time Passes?
LLMs can judge relative response lengths and shorten outputs under urgency, but the claim that they perceive physical time passage is not cleanly established.
-
Concept Incongruence: An Exploration of Time and Death in Role Playing
LLMs asked to role-play dead historical figures rarely abstain from answering post-death questions, and their factual accuracy drops due to poorly encoded death states and role-playing-induced shifts in temporal repre...
-
Reading Between the Timelines: RAG for Answering Diachronic Questions
TA-RAG uses LLM-extracted time intervals, time-filtered retrieval with averaged temporal query embeddings, and chronologically ordered context to beat standard RAG by 13-27 points on the new ADQAB benchmark of 525 mul...
Discussion (0). Sign in to comment.