Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

TimelineKGQA: A Comprehensive Question-Answer Pair Generator for Temporal Knowledge Graphs

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A generator that maps any temporal knowledge graph to categorized QA benchmarks.

desk verdict A genuinely useful QA-pair generator with public code and real coverage of under-served question types, but the difficulty evaluation is partly circular and the 'any TKG' claim overreaches; it deserves referee time because the flaws are fixable. read the letter →

arxiv 2501.04343 v1 pith:77PG5M4C submitted 2025-01-08 cs.LO cs.AIcs.CL

Qiang Sun , Sirui Li , Du Huynh , Mark Reynolds , Wei Liu This is my paper · ORCID
classification cs.LOcs.AIcs.CL
keywords temporalknowledgegraphquestionansweringquestion-answerpairgenerationreasoningdatasetcomplexitycategorizationTKGQAQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TimelineKGQA is a proposed Python package that turns any temporal knowledge graph into a question-answer benchmark, together with a four-way categorization of temporal questions based on timeline-context relationships. The paper's central claim is that question complexity can be systematically captured by the number of context facts, the factual-versus-temporal focus of the answer, the temporal relations involved, and four temporal capabilities: temporally constrained retrieval, timeline position retrieval, temporal semantic operations, and timeline arithmetic operations. A sympathetic reading is that this gives the field a unified way to generate custom TKGQA datasets, covering answer types from entities and timestamps to time ranges, durations, and yes/no, so that private-domain and harder benchmarks are no longer blocked by data scarcity. The paper demonstrates the generator on two knowledge sources and shows retrieval performance falls steadily from Simple to Medium to Complex questions, aligning difficulty with the proposed categorization.

What carries the argument

The load-bearing object is the timeline-context categorization, which treats time as a homogeneous line and classifies questions by |C| (number of context facts), answer focus, temporal relation type, and required temporal capability. The four capabilities are TCR (retrieve facts under a given time constraint), TPR (retrieve the time range of a given fact), TSO (compute a new time range from a temporal signal word), and TAO (apply set operations, Allen's thirteen relations, duration calculation, or ranking to time ranges). This categorization directly drives Generator Module 3's branching logic, so the framework is not just a taxonomy but the control structure of the generator.

What would settle it

Audit a random sample of the generated QA pairs by giving human annotators only the intended context facts and asking whether the paraphrased question is answerable from them and whether the recorded answer is the unique correct one; a substantial share of mismatches would refute the claim that the generator reliably produces valid QA pairs.

Watch

Extended reading notes

Core claim

The discovery is that a temporal question's complexity can be decomposed along four independent dimensions, and that every combination of these dimensions can be materialized as concrete QA pairs from any TKG. The generator first unifies the input graph by giving every fact a time range, samples one, two, or three context facts, then constructs questions according to the category path: simple factual or temporal questions, medium questions needing temporal semantic operations or timeline arithmetic operations, and complex questions adding ordinal ranking. Recategorizing CronQuestions under the framework exposes what is missing from existing benchmarks, notably the entire Medium.Temporal category, ranking beyond first/last, duration comparisons, and time range inference. The paper claims this is the first universal generator that incorporates all identified complexity dimensions.

Load-bearing premise

The pipeline assumes the sampled context facts are sufficient to answer each generated question and that LLM paraphrasing preserves both the intended meaning and the recorded answer, and neither assumption is verified in the paper.

Editorial extensions

If this is right

  • Any TKG, including private-domain graphs, can be turned into a TKGQA benchmark without manual annotation, enabling domain-specific fine-tuning.
  • Generated datasets can include question types that existing benchmarks lack, such as temporal aggregation, duration comparison, time range inference, and ordinal ranking.
  • Because answer types include time ranges, durations, and yes/no, retrieval systems must move beyond timestamp-or-entity outputs.
  • Applying the categorization to existing datasets reveals coverage gaps, so the framework doubles as an audit tool for benchmark completeness.
  • If difficulty aligns with context count, the generator can produce controllable difficulty curricula by mixing Simple, Medium, and Complex questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to audit the LLM paraphrase step: no verification is reported, so generated datasets may contain questions whose paraphrase changes the answer; measuring this drift rate would show how much of the pipeline's validity rests on the paraphraser.
  • The generator's sampling prioritizes temporally proximate facts and frequent entities, which likely oversamples mainstream entity pairs and undersamples rare or long-tail facts in generated benchmarks.
  • The same categorization could serve as an evaluation metric for any TKGQA system, reporting accuracy per capability (TCR, TPR, TSO, TAO) rather than one global score, which would localize which reasoning skill a model lacks.
  • Because the generator is template-driven before paraphrase, it may produce answer bias toward the sampled facts' ordering; future work could randomize fact order and check answer invariance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes TimelineKGQA, a Python package that generates temporal knowledge graph question-answer (QA) pairs from a given TKG. It introduces a categorization framework with four dimensions: context complexity (based on the number of context facts |C|), answer focus (factual vs. temporal), temporal relations (Allen relations, set operations, duration, ranking), and temporal capabilities (TCR, TPR, TSO, TAO). The generator has four modules: graph unification, fact sampling, template-based QA generation with three complexity levels, and LLM paraphrasing. The authors evaluate the generated datasets (from ICEWS and CronQuestion) with a retrieval-augmented generation baseline, reporting that retrieval performance degrades from Simple to Complex questions, which they interpret as evidence that their complexity categorization reflects question difficulty.

Significance. If the claims hold, TimelineKGQA would be a valuable resource for the TKGQA community: it is an open-source tool, it attempts a systematic multi-dimensional categorization of temporal questions, and it supports answer types beyond entities and timestamps (yes/no, ranges, durations). The paper also includes a useful reclassification of CronQuestions that highlights gaps in existing datasets. However, the empirical validation of the difficulty gradient is compromised by a circular metric, and the universality claim ("any TKG") is not backed by coverage guarantees or tests on sparse or non-temporal graphs. The lack of verification for LLM paraphrases further weakens confidence in the correctness of the generated QA pairs. The framework itself is a reasonable starting point, but the paper needs substantial additional evidence and clarification before its central claims can be accepted.

major comments (4)
  1. [Section 6, Table 5] The claim that the categorization framework "effectively reflects question difficulty" (Section 6) is circular. The complexity categories are defined by |C| (Section 4: Simple if |C|=1, Medium if |C|=2, Complex if |C|=3), and the evaluation metric requires all |C| context facts to be retrieved within the top |C|*K positions. Consequently, a question with larger |C| is harder to satisfy under this metric by construction, regardless of the semantic complexity of the question. The drop in Hits@1 from 0.660 (Simple) to 0.011 (Complex) is therefore an artifact of the definition of the metric and the categories, not independent evidence of a meaningful difficulty gradient. To support the claim, the authors should either evaluate with a metric that does not scale with |C| (e.g., require only one relevant fact to be retrieved, or directly measure answer correctness), or compare against a baseline where |C| is not the defining feature of the categories.
  2. [Section 5, Module 2 and Figure 2] The abstract and Section 1 claim that TimelineKGQA can "generate TKGQA datasets incorporating all these complexity dimensions from any given TKG." This universality claim is unsupported. Medium and Complex questions require two or three context facts that satisfy specific structural conditions: for Medium.Factual, an Allen relation between the facts; for Medium.Temporal, union/intersection, Allen relations, or duration; for Complex, ranking or multi-fact coordination. No property of the input TKG is stated that guarantees such configurations exist. For a sparse or disconnected TKG where facts do not share entities or overlap temporally, the sampler would only produce Simple questions, so the generator cannot fulfill the "all dimensions" promise. The authors should either restrict the claim to TKGs with sufficient density and connectivity, or provide a formal characterization of the conditions under which every complexity dimension is represented.
  3. [Section 5, Module 1] Module 1 states that the generator accepts "any knowledge graph" and converts it into a TKG by augmenting each fact with a time range (t_start, t_end). However, the manuscript provides no method or heuristic for deriving these time ranges when the source KG is non-temporal. Arbitrary or ungrounded time ranges would make the generated temporal questions and their answers meaningless. The paper only evaluates on ICEWS and CronQuestion, which are already temporal and dense, so the non-temporal case is not tested. The authors must specify and justify the augmentation procedure, or explicitly restrict the tool's scope to TKGs that already carry temporal annotations.
  4. [Section 5, Module 4] The LLM paraphrase step applies to every generated question without any verification that the paraphrase preserves the original meaning and the answer. A paraphrase that shifts a temporal constraint or an entity reference would silently produce an invalid QA pair, corrupting the generated dataset. No human evaluation, round-trip consistency check, or automated answer validation is reported. Given that the entire dataset quality rests on this step, the authors need to either add a verification mechanism (e.g., checking that the paraphrased question, when run through the original template's answer logic, yields the same answer) or provide a sample-based human evaluation demonstrating that meaning is preserved.
minor comments (6)
  1. [Section 2] The discussion of "suspicious works" and "potential academic integrity concerns" regarding references [8] and [9] is accusatory and outside the technical scope of the paper. It should be removed or rephrased as a neutral observation about benchmark saturation.
  2. [Section 3] The notation defines T as the set of timestamps, but facts are described with t_start and t_end. Please clarify whether timestamps are time points or intervals, and whether the generator handles both consistently.
  3. [Figure 1] Figure 1 contains typographical errors that distract from the content (e.g., "Staline", "Rossevelt", "Secrety", and "Genernal" if present). These should be corrected.
  4. [Section 6] The modified Hits@K metric (all |C| context facts must be retrieved within the top |C|*K positions) is described only in a sentence; please provide a formal definition with equations to avoid ambiguity.
  5. [Table 4] The counts for "Relation Duration" and "Relation Ranking" are much smaller than other categories; please verify that these numbers are consistent with the generation logic described in Module 3 and with the totals in Table 3.
  6. [General] The paper would benefit from a short discussion of the limitations of the retrieval-based evaluation: it measures evidence retrieval, not actual QA accuracy, so the reported numbers do not directly assess whether the generated questions are answerable or whether the answers are correct.

Circularity Check

1 steps flagged · score 6.0 of 10

The claimed empirical validation that question difficulty aligns with the categorization is by construction: categories are defined by |C| and the evaluation metric requires retrieving all |C| context facts in the top |C|·K positions.

  1. self definitional [Section 4 (Question Complexity Dimensions) and Section 6 (Dataset Generation, RAG baseline evaluation)]
    "Simple (if|C| = 1): ... Medium (if|C| = 2): ... Complex (if |C| = 3): ... all context facts must be retrieved within the top |C|· K positions."

    The categories being validated are defined by the same quantity the metric enforces. Section 4 makes Simple/Medium/Complex equivalent to |C|=1/2/3. Section 6 then defines Hits@K success as retrieving all context facts within the top |C|·K positions. For Hits@1, a Simple question succeeds when 1 fact is in the top 1; a Medium question requires 2 facts in the top 2; a Complex question requires 3 facts in the top 3. The probability of meeting the requirement necessarily falls as |C| grows for any retriever with imperfect ranking, so Table 5's decline from 0.660 to 0.128 to 0.011 is entailed by the metric definition, not by an independent measurement of temporal reasoning difficulty.

full rationale

One concrete circular step exists: the difficulty-validation result is self-definitional. Section 4 defines complexity by the number of context facts (|C|), and Section 6 evaluates difficulty by requiring all |C| context facts to appear in a window that scales with |C|. Consequently, the monotone decrease in Hits@1 from Simple to Medium to Complex is a mathematical consequence of the metric rather than evidence about the semantic difficulty of the generated questions. This does not invalidate the generator itself, which is largely self-contained: the template paths in Modules 2-4, the categorization taxonomy, and the dataset construction are independent contributions. The universality claim ('from any given TKG') is unsupported for sparse or non-temporal inputs, but that is a coverage/completeness gap, not circularity. The unverified LLM paraphrase step is a quality risk, not a circularity. No load-bearing self-citations or author-imported uniqueness theorems appear. Therefore the circularity score is 6: the central empirical 'prediction' about difficulty reduces by construction, while the main generator proposal retains independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The contribution is not a mathematical derivation; it is a dataset generator whose behavior depends on hand-chosen design parameters and unverified assumptions about question answerability and time-range availability.

free parameters (4)
  • context_size_threshold = |C| in {1,2,3}
    Hand-chosen boundary between Simple, Medium, and Complex; no justification that three facts is the right maximum.
  • fact_sampling_priorities = temporally proximate facts, frequent entities
    Module 2 biases generated questions toward common entities and nearby time ranges, shaping dataset difficulty and diversity.
  • questions_per_simple_fact = 5 (2 factual, 3 temporal/duration)
    Module 3 fixes the number of templates per fact without a stated reason.
  • LLM_paraphrase_configuration = not disclosed
    Module 4 uses an OpenAI-style LLM but does not specify model, prompt, or sampling parameters, so regeneration is unstable.
assumptions (5)
  • domain assumption Every fact in the input can be represented as (e1, r, e2, t_start, t_end) with meaningful durations.
    Section 3 defines TKG facts this way; instant events and missing end times are treated as special cases, but duration computations such as Obama's 8 years require complete ranges.
  • ad hoc to paper Question complexity is adequately captured by the number of context facts |C|.
    Section 4 defines Simple/Medium/Complex only by |C|=1,2,3; no evidence is given that this covers all temporal complexity.
  • ad hoc to paper LLM paraphrase preserves question meaning and answerability.
    Section 5 Module 4 paraphrases templates with no correctness check; the datasets assume this holds.
  • ad hoc to paper Retrieval of all sampled context facts is required to answer each generated question.
    Section 6 modifies MRR/Hits@K to require all |C| facts within a top window, making difficulty depend directly on the category definition.
  • ad hoc to paper Any input knowledge graph can be augmented with time ranges even if it is not already temporal.
    Module 1 says any KG is converted to a TKG by adding (t_start, t_end), but no source for those stamps is specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TimelineKGQA: A Comprehensive Question-Answer Pair Generator for Temporal Knowledge Graphs." pith.science (2026). https://pith.science/paper/77PG5M4C

@misc{pith2026250104343,
  author       = {Pith},
  title        = {Pith review of: TimelineKGQA: A Comprehensive Question-Answer Pair Generator for Temporal Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77PG5M4C}},
  note         = {Machine review of arXiv:2501.04343}
}
read the original abstract

Question answering over temporal knowledge graphs (TKGs) is crucial for understanding evolving facts and relationships, yet its development is hindered by limited datasets and difficulties in generating custom QA pairs. We propose a novel categorization framework based on timeline-context relationships, along with \textbf{TimelineKGQA}, a universal temporal QA generator applicable to any TKGs. The code is available at: \url{https://github.com/PascalSun/TimelineKGQA} as an open source Python package.

Figures

Figures reproduced from arXiv: 2501.04343 by the authors.

Figure 1
Figure 1. Categorization framework for Temporal Questions [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Question answer pair generation procedure in our TimelineKGQA Generator [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Temporal knowledge graph unification dimensions. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RTQA : Recursive Thinking for Complex Temporal Knowledge Graph Question Answering with Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    RTQA recursively decomposes complex temporal questions into sub-questions, solves them bottom-up with LLMs and retrieved TKG facts, and aggregates multiple answers, improving Hits@1 on MultiTQ and TimelineKGQA.

  2. TRAVELER: A Benchmark for Evaluating Temporal Reasoning across Vague, Implicit and Explicit References

    cs.CL 2025-05 conditional novelty 6.0 of 10

    TRAVELER provides a synthetic temporal question-answering benchmark and evaluation showing that LLM accuracy degrades from explicit to implicit to vague temporal references and as event-set length increases.

Reference graph

Works this paper leans on

9 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [1]

    Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A Dataset for Answering Time-Sensitive Questions. arXiv:2108.06314 [cs] http://arxiv.org/abs/2108.06314

    arXiv 2021

  2. [2]

    Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018. TempQuestions: A Benchmark for Temporal Question Answering. In Companion of the The Web Conference 2018 (Lyon, France, 2018). ACM Press, 1057–1062. https://doi.org/10.1145/3184558.3191536

    arXiv 2018

  3. [3]

    Zhen Jia, Philipp Christmann, and Gerhard Weikum. 2024. Faithful Temporal Question Answering over Heterogeneous Sources. In Proceedings of the ACM on Web Conference 2024. ACM, 2052–2063. https://doi.org/10.1145/3589334.3645547

    arXiv 2024

  4. [4]

    Zhen Jia, Philipp Christmann, and Gerhard Weikum. 2024. TIQ: A Benchmark for Temporal Question Answering with Implicit Time Constraints. In Companion Proceedings of the ACM on Web Conference 2024 . ACM, 1394–1399. https://doi.org/ 10.1145/3589335.3651895

    arXiv 2024

  5. [5]

    Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. Complex Temporal Question Answering on Knowledge Graphs. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM ’21). Association for Computing Machinery, 792–802. https://doi.org/10. 1145/3459637.3482416

    arXiv 2021

  6. [6]

    Yonghao Liu, Di Liang, Mengyu Li, Fausto Giunchiglia, Ximing Li, Sirui Wang, Wei Wu, Lan Huang, Xiaoyue Feng, and Renchu Guan. 2023. Local and Global: Temporal Question Answering via Information Fusion. InProceedings of the Thirty- Second International Joint Conference on Artificial Intelligence . 5141–5149. https: //doi.org/10.24963/ijcai.2023/571 Main Track

    doi:10.24963/ijcai.2023/571 2023

  7. [7]

    Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question An- swering Over Temporal Knowledge Graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Online, 2021-08). 6663–6676. https://doi.org/10.18653/v1/2021.acl-long.520

    doi:10.18653/v1/2021.acl-long.520 2021

  8. [8]

    Chao Xue, Di Liang, Pengfei Wang, and Jing Zhang. 2024. Question Calibration and Multi-Hop Modeling for Temporal Question Answering. Proceedings of the AAAI Conference on Artificial Intelligence 38, 17 (Mar. 2024), 19332–19340. https: //doi.org/10.1609/aaai.v38i17.29903

    work page doi:10.1609/aaai.v38i17.29903 2024

Show all 9 references
  1. [9]

    Zhiyuan Zha, Pengnian Qi, Xigang Bao, Mengyuan Tian, and Biao Qin. 2024. M3TQA: Multi-View, Multi-Hop and Multi-Stage Reasoning for Temporal Question Answering. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 10086–10090...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.