{"id":"400d852f-7dc4-4fb9-8af9-2767c4fec3be","arxiv_id":"2508.20828","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GDLLM improves event temporal relation extraction by feeding LLM-generated probability distributions into a graph attention network, achieving state-of-the-art micro-F1 scores on TB-Dense and MATRES.","lead":"This paper presents GDLLM, a system that combines a large language model with a graph attention network to classify the order of events in text, and reports the best scores on two standard datasets. The method uses the language model's probability guesses as graph features, which help the system learn both distant and close event relationships.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run F1 scores with no error bars or validation-protocol details leave the SOTA claim over MATRES (0.5 F1 over MulCo) vulnerable to seed variance and possible test-set tuning.","rationale":"I agree with the reader's overall CONDITIONAL verdict, but the most load-bearing concern is not the GAT's structural contribution per se. The w/o GD ablation shows a 12.8-point drop on MATRES, indicating the GAT adds substantial signal; thus the redundancy worry is already weakened. However, the paper's use of single runs and its silence on validation/tuning protocol directly threatens the quantitative SOTA claim, especially given the 0.5 F1 margin on MATRES. Without multiple seeds and a clear validation procedure, the reported advantage could be noise or overfitting. The reader's rationale does mention the lack of multiple runs, but the explicit weakest_assumption focuses on GAT attribution; hence partial agreement. The CONDITIONAL verdict is appropriate because the concerns are testable and addressable with additional experiments and code release.","tokens_in":12956,"tokens_out":9947,"duration_ms":96623,"concrete_test":"Run GDLLM with Llama3.1-8B on MATRES and TB-Dense using at least 5 different random seeds with identical hyperparameters; report mean±std micro-F1. Determine if the mean minus one std on MATRES exceeds 90.4 (MulCo's reported F1). If not, the SOTA claim is not established. Additionally, confirm whether hyperparameter search is conducted on a separate validation split; if not, rerun the full pipeline with a validation split and report test F1 for the best-validated model. Also, release the exact training configuration and inference code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is SOTA micro-F1 on TB-Dense (87.5) and MATRES (90.9). The MATRES margin over the previous SOTA (MulCo, 90.4) is only 0.5 points. The paper reports one run per configuration with no standard deviation, no seed count, no code release, and no hyperparameter details beyond 'HEBO'. Since neural models on MATRES typically show seed variance of ±0.5 F1 or more, a single run cannot establish that GDLLM is better than MulCo. Furthermore, the paper does not state whether HEBO optimizes on a held-out validation split or directly on the test set; if the latter, the reported numbers are optimistically biased and not comparable to baselines. These are load-bearing because the central assertion is a quantitative SOTA claim; if the margin is noise or the tuning is on test, the claim fails regardless of the model's internal design. The reader's concern about GAT redundancy is partially addressed by the w/o GD ablation (78.1 vs 90.9 on MATRES), so the more critical threat is statistical reliability and evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GDLLM, a two-stage method for event temporal relation extraction. A LoRA-fine-tuned LLM (Llama3.1-8B or Qwen2.5-7B) produces a probability distribution over relation classes for each event pair; these probability vectors are used as edge features in a two-layer, 8-head GAT built over event nodes in a document. The final classifier concatenates the GAT output representations with the LLM probability vector and classifies via a fully connected layer and softmax. The method is evaluated on TB-Dense and MATRES, reporting micro-F1 scores of 87.5 and 90.9 with Llama3.1-8B, which the paper claims are state-of-the-art. The paper also includes ablations (w/o LP, w/o GD, w/o PI), a distance-conditioned analysis, a zero-shot comparison, t-SNE visualizations, and a comparison with GNN-based benchmarks.","tokens_in":13232,"tokens_out":3412,"duration_ms":34738,"significance":"If the reported results are reliable, the core idea—using an LLM's probability distribution as soft edge features in a GAT to capture both long-distance and short-distance temporal dependencies—is a meaningful and well-motivated contribution. The design is sensible, the distance-conditioned analysis in Table 4 provides qualitative support, and the paper is transparent about some limitations, such as the Qwen model's weakness on the EQUAL class. The main significance is limited by the absence of statistical reliability evidence: the MATRES gain over the prior SOTA is only 0.5 micro-F1, and a single run cannot establish superiority. The TB-Dense gain of 1.9 points is larger but still needs variance estimates.","major_comments":[{"comment":"The central SOTA claim on MATRES rests on a 0.5-point micro-F1 improvement over MulCo (90.9 vs 90.4). The paper reports one run per configuration with no standard deviation, no number of seeds, and no significance test. Neural models on MATRES typically exhibit seed variance of at least ±0.5 F1, so this margin may be noise. Please report mean ± std over multiple seeds (e.g., 5) and a paired significance test for the comparisons with MulCo, and also for the TB-Dense results.","section":"§3.3, Table 1"},{"comment":"The hyperparameter optimization procedure is described only as 'we employ the HEBO algorithm.' It is not stated whether HEBO optimizes on a fixed validation split or directly on the test set, what the search space is, or which final hyperparameters were selected. If test-set performance was used for model selection, the reported numbers are optimistically biased and not directly comparable to baselines. Please specify the validation protocol and report the final hyperparameters (learning rate, weight decay, batch size, GAT hidden dimensions, etc.).","section":"§3.2, HEBO"},{"comment":"The 'w/o LP' ablation removes the entire LLM component, not just the probability generation: the GAT then receives only the dataset-derived node features. The resulting drop (e.g., from 90.9 to 68.7 on MATRES) shows that the LLM is essential, but it does not isolate the contribution of the probabilistic edge features. The 'w/o PI' ablation partly addresses this by replacing probabilities with hard labels, but the description of 'w/o LP' should be revised so readers do not interpret it as a controlled ablation of probability information.","section":"§3.5, Tables 2 and 3"},{"comment":"The evaluation protocol is not fully reproducible: data splits are said to follow prior work, but no split details are given, and no code or configuration is released. Given that the contribution is an empirical pipeline and the SOTA claim is quantitative, please provide a detailed setup (data splits, event marker preprocessing, LoRA configuration, and GAT layer specifics) and consider releasing code to allow independent verification.","section":"§3.1, §3.2"}],"minor_comments":[{"comment":"The notation in Eq. (1) is inconsistent: the numerator uses e^{z_r} with index r, while the denominator sums e^{z_n} over n=1..c. Please unify the index notation. Also, 'in the inference process of LoRA tuning' is awkward; the probability is generated after fine-tuning, not during inference of LoRA tuning.","section":"§2.2, Eq. (1)"},{"comment":"The node feature construction is vague: 'Each event Ei and its order and type information are both represented as a node' and 'node features are obtained from the dataset corresponding to the event.' Please specify how event order, event type, and any contextual embeddings are encoded into h_i^(0).","section":"§2.3, Graph feature construction"},{"comment":"The main claim says the method works 'without manually designed prompts or instructions,' but the zero-shot experiments use hand-crafted prompts. This is not contradictory for the main pipeline, but the text should clarify that the prompt-free claim applies to the fine-tuned GDLLM pipeline, not to the zero-shot setting.","section":"§3.7, Figure 5"},{"comment":"There are several typos and formatting issues: 'realtion', 'Manully' in Figure 5, 'V anilla', and inconsistent capitalization in figure/table captions. Please proofread.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core method is interesting and the paper is within scope for a NLP venue. My recommendation hinges on the statistical reliability of the SOTA claim: a single-run 0.5 F1 margin over the previous SOTA on MATRES is not sufficient. If the authors can provide multi-run variance estimates, significance tests, and a clear validation protocol for HEBO, the paper could be acceptable. I would not require a re-derivation of the method; the main need is evaluation rigor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible, well-motivated architecture paper with a real new idea, but the empirical evidence as reported is not enough to support the SOTA claim. The novelty is using LLM softmax probability vectors as edge features in a multi-head GAT for event temporal relation extraction. That specific combination is not in the cited literature, and the ablations suggest it matters: replacing soft probabilities with hard 0/1 labels (w/o PI) drops 8-11 F1, and removing the GAT entirely (w/o GD) drops 12-13. The distance-stratified analysis in Table 4 supports the claim that GAT helps on long-distance pairs, which is the mechanism they advertise. So the paper earns credit for a clear and interesting design and for testing its components.\n\nThe soft spots are mostly about statistical credibility. All results are single runs. MATRES gain over MulCo is 0.5 F1, which can easily be seed noise. The paper doesn't report variance, number of seeds, or whether HEBO selects hyperparameters on a validation split or directly on the test set. If the latter, the numbers are optimistically biased. No code is released. These issues directly undermine the SOTA claim. The w/o LP ablation (removing the LLM probabilities entirely) drops 22 points, but that is an expected sanity check rather than a meaningful comparison. Also, the final classifier directly concatenates the raw probability vector, so the GAT's structural contribution is not isolated from the value of having the probability as a feature. The paper does show w/o GD is much worse, which suggests the graph/event-node features matter, but an additional control (e.g., GAT node features without edge probabilities, or a non-graph aggregator) would sharpen the claim.\n\nOne wording misstep: they claim to work 'without manually designed prompts or instructions', but the event markers [EVi] are manually defined and the zero-shot experiments use a hand-built prompt. That's a minor overstatement.\n\nThe paper is worth refereeing, but not as-is. I'd ask for multiple seeds with standard deviations, a clear description of the tuning protocol, code or at least full hyperparameters, and an analysis that isolates the GAT's contribution beyond directly using the probability vector. With those, the core idea could be a solid contribution. For a reader working on structured prediction with LLMs, the architecture is worth knowing about.","headline":"Solid new idea—LLM softmax vectors as GAT edge features—but single-run results and a thin evaluation make the SOTA claim unproven.","tokens_in":13767,"tokens_out":3295,"would_cite":true,"duration_ms":32895,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GDLLM reports 87.5 micro-F1 on TB-Dense and 90.9 on MATRES by routing LLM probability distributions through a two-layer graph attention network.","keywords":["event temporal relation extraction","large language models","graph attention networks","soft probability edge features","long-distance event dependencies","minority relation classes","MATRES","TB-Dense"],"falsifier":"Take the trained GDLLM model on MATRES and delete the direct probability vector from the final classifier, keeping only the two graph node representations; if micro-F1 remains near 90.9, the graph is not adding information beyond the LLM probabilities. A complementary check is a graph-free baseline that feeds the LLM probability vector through a linear layer and compares its micro-F1.","tokens_in":12833,"feed_emoji":"🕰️","tokens_out":8188,"duration_ms":72918,"temperature":0.7,"pith_summary":"This paper aims to show that event temporal relation extraction improves when a large language model is used as a soft evidence generator rather than a hard reasoner. The proposed method, GDLLM, takes the probability distribution an LLM assigns to each event pair's temporal relation and uses it as edge features in a graph attention network, allowing information to propagate across events that are far apart in the text. On TB-Dense and MATRES it reports micro-F1 scores of 87.5 and 90.9, above the previous state of the art, and it reports that minority relation classes benefit the most. This matters because temporal relations in real text are imbalanced and long-distance event pairs are exactly the hard cases for prompt-based LLM approaches.","feed_headline":"LLM soft labels plus graph attention top MATRES and TB-Dense","feed_subtitle":"Probability distributions become edge features, lifting minority-class and long-distance event relations.","key_machinery":"The load-bearing mechanism is the distance-aware graph attention module with soft-inference edge features. Events are nodes; each node feature encodes the event's order and type. Every edge carries the LLM's probability distribution over relation classes instead of a hard predicted label. A two-layer multi-head graph attention network with eight heads computes attention coefficients that depend on the source node, target node, and this probability vector, allowing the network to weigh evidence from distant events. The second layer's head outputs are averaged, and the final classifier concatenates the two node representations with the original probability vector, splitting the global feature","core_discovery":"The paper's central claim: the bottleneck in event temporal relation extraction is not the language model's knowledge but how its outputs are used. GDLLM fine-tunes an LLM with LoRA to emit a probability vector over relation classes for each event pair, then uses those vectors as edge features in a two-layer, eight-head graph attention network over a document-level event graph. Attention weights combine node features with the probability vector, so soft evidence propagates between distant events while the same vector stays in the final classifier for short-distance pairs. With Llama3.1-8B this yields 87.5 micro-F1 on TB-Dense and 90.9 on MATRES, 1.9 and 0.5 points above the previous best, wi","pith_inferences":["A plausible testable consequence is that the base LLM matters less than the calibration of its probability outputs; a smaller, well-calibrated model might capture much of the gain at lower cost.","The paper does not isolate whether the graph's benefit comes from multi-head attention or from the probability edge features; replacing GAT with a distance-weighted transformer would separate structural propagation from attention.","Because the final classifier concatenates the raw probability vector alongside graph outputs, a stronger ablation would remove that direct path to show whether the graph genuinely adds information or just reweights the LLM's own distribution.","The distance-5 result shows only a small gain over the version without soft inference, suggesting very long event chains may need a different inductive bias such as hierarchical or recurrent graph layers."],"forward_implications":["LLM-based event temporal relation extraction can avoid hand-written prompt templates entirely; probability outputs become the interface between the language model and downstream structure.","Minority temporal relations improve enough that the micro-macro gap narrows, so downstream applications needing rare relations such as SIMULTANEOUS would see better recall.","The method transfers across LLMs: Qwen2.5-7B also beats the previous state of the art, suggesting the graph component is not tied to a single model.","Even in zero-shot mode, with no LoRA fine-tuning and fixed graph parameters, the approach outperforms earlier manually prompted zero-shot systems on MATRES.","Removing the graph module hurts more than removing the soft-probability edge features, indicating long-distance event pairs are the main source of error this architecture recovers."],"supporting_citations":[{"why":"Provides the MATRES dataset, one of the two benchmarks on which state-of-the-art performance is claimed.","marker":"(Ning et al., 2019)"},{"why":"Provides the TB-Dense dataset with dense, six-way temporal relations used for evaluation.","marker":"(Cassidy et al., 2014)"},{"why":"Supplies the LoRA parameter-efficient fine-tuning method used to adapt the LLM before probability generation.","marker":"(Hu et al., 2022)"},{"why":"MulCo, the previous state-of-the-art model that GDLLM outperforms on both datasets, also contributes GNN-based baselines.","marker":"(Yao et al., 2024)"},{"why":"CPTRE, the contrastive prototypical sampling baseline used for minority-class and macro-F1 comparisons.","marker":"(Yuan et al., 2024)"},{"why":"LLMERE, an LLM-based event relation extraction baseline with rationales on Llama3.1-8B.","marker":"(Hu et al., 2025)"},{"why":"MAQInstruct, an instruction-based unified event relation extraction baseline using Llama2-7B.","marker":"(Xu et al., 2025)"},{"why":"TIMERS, a document-level graph-based baseline that motivates the GNN comparison.","marker":"(Mathur et al., 2021)"}],"fun_headline_variants":["LLM soft labels as graph edges top MATRES and TB-Dense","Graph attention with LLM soft labels boosts rare event relation classes","Distance-aware graph edges from LLM probabilities improve event temporal F1","LLM probability vectors as edge features: long-distance event relations improved","Soft probability edges in graph nets give top event temporal F1"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the LLM's probability distribution over relation classes carries reliable relational evidence and that the graph attention module adds non-redundant structure on top of it; if the graph output merely echoes those probabilities, the reported gains would collapse.","fun_headline_variants_meta":{"raw":{"variants":["LLM soft labels as graph edges top MATRES and TB-Dense","Graph attention with LLM soft labels boosts rare event relation classes","Distance-aware graph edges from LLM probabilities improve event temporal F1","LLM probability vectors as edge features: long-distance event relations improved","Soft probability edges in graph nets give top event temporal F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001127,"raw_usage":{"total_tokens":4534,"prompt_tokens":766,"completion_tokens":3768,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":3688}},"tokens_in":510,"tokens_out":3768,"duration_ms":23948,"temperature":1.0,"reasoning_tokens":3688,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:45:46.329596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained GDLLM model on MATRES and delete the direct probability vector from the final classifier, keeping only the two graph node representations; if micro-F1 remains near 90.9, the graph is not adding information beyond the LLM probabilities. A complementary check is a graph-free baseline that feeds the LLM probability vector through a linear layer and compares its micro-F1.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MATRES dataset, one of the two benchmarks on which state-of-the-art performance is claimed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the TB-Dense dataset with dense, six-way temporal relations used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CPTRE, the contrastive prototypical sampling baseline used for minority-class and macro-F1 comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TIMERS, a document-level graph-based baseline that motivates the GNN comparison."}],"review_version":1}