{"id":"bf9422be-14f4-4e78-b321-dc21fb2cb0f0","arxiv_id":"1909.00429","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A bi-LSTM model with ELMo/BERT embeddings, a Siamese common sense encoder, and ILP inference improves temporal relation extraction accuracy by roughly 10 points over CogCompTime on MATRES and TCR.","lead":"This paper builds a neural system for temporal relation extraction that combines LSTM encoders, contextualized word embeddings, and a common sense encoder trained on an existing temporal knowledge base. It reports about 10 absolute point accuracy gains over the previous state of the art on two benchmark datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CSE gain is not separable from CogCompTime's own errors because TEMPROB was reconstructed with CogCompTime as the base system.","rationale":"The reader's weakest assumption matches my concern exactly: TEMPROB was reconstructed with CogCompTime, the same system used as the comparison baseline, so the CSE contribution is not cleanly attributable to external knowledge. This is the most load-bearing issue because the headline claim has two parts: a strong neural baseline versus CogCompTime, and a specific CSE common-sense contribution. The first part is well supported by Table 2's Concat rows (BERT 69.1 vs 61.6 accuracy, 74.4 vs 66.6 F1) and does not depend on CSE. The second part, including the claim in Section 4.2 that approximately 2 points come from CSE, depends entirely on TEMPROB being a reliable external resource. Since TEMPROB's reconstruction inherits CogCompTime's extraction behavior, the gain could partially measure the model's ability to imitate the baseline's outputs on the test set, especially given the small MATRES test set (837 relations). This does not undermine the main empirical finding, so the verdict should remain CONDITIONAL rather than REJECT: the authors should either strengthen the CSE attribution by testing with an independent TEMPROB construction, provide the code and reconstructed resource to allow inspection of leakage, or soften the common-sense attribution. The paper shares a code URL but no commit hash or direct data link is included in the manuscript, and only single-run results are reported, which supports a conditional rather than unconditional acceptance.","tokens_in":9701,"tokens_out":1616,"duration_ms":14434,"concrete_test":"Rebuild TEMPROB using an independent base extractor with no shared lineage with CogCompTime (for example, the original Ning et al. 2018b base method, or a high-quality end-to-end neural scorer) while holding the Siamese architecture and all other training choices fixed; then retrain CSE and rerun the Concat+CSE rows of Table 2. If the margin over Concat (approximately 2 points) collapses or reverses when CogCompTime's reconstruction is replaced, the CSE attribution is an artifact of baseline leakage; if the margin persists, the common-sense contribution is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim of about 10% absolute improvement over CogCompTime is independently supported by the Concat rows in Table 2 (BERT 69.1 vs 61.6 accuracy; F1 74.4 vs 66.6), so the headline comparison does not hinge on CSE. What is not cleanly supported is the subclaim that CSE contributes approximately 2 points of real common-sense knowledge. Section 3.2 states that the TEMPROB used to train CSE was reconstructed with the base method changed to CogCompTime, which is also the comparison baseline. TEMPROB aggregates automatically extracted relation frequencies; if CogCompTime's systematic errors or label biases are baked into those frequencies, then matching those frequencies means partly imitating the baseline's errors. The observed gain from adding CSE could then reflect the model learning to re-absorb CogCompTime's output distribution rather than external temporal knowledge. The paper provides no analysis of TEMPROB quality, no comparison of the reconstructed TEMPROB with the original Ning et al. (2018b) version, and no ablation showing CSE's gain persists when trained on a TEMPROB version built with an independent base method. The Concat-only results still establish the main neural-baseline claim, but the paper overstates the common-sense attribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a neural baseline for temporal relation extraction based on a bidirectional LSTM over word embeddings, with event positions handled either by XML position indicators or by concatenating the hidden states at the event positions (Concat). The model is evaluated on MATRES and TCR with five embedding types (word2vec, GloVe, FastText, ELMo, BERT), and is further augmented with a Siamese 'common sense encoder' (CSE) trained on TEMPROB, plus ILP-based global inference. The authors report that Concat with ELMo or BERT plus CSE outperforms the feature-based CogCompTime by about 10% absolute accuracy and F1 on MATRES and by similar margins on TCR, and they argue that the results support a strong neural baseline for future work.","tokens_in":9976,"tokens_out":4569,"duration_ms":39067,"significance":"If the results hold, the paper makes a useful contribution by showing that a relatively standard LSTM with contextualized embeddings can substantially beat a feature-based system on the MATRES benchmark, and by providing a systematic comparison of position-indicator versus concatenation handling, five embedding choices, and a common-sense injection method. The paper also reports statistical significance tests (McNemar, paired t-test) and states that the system is public, which supports reproducibility. The main caveat is that the reported contribution of the CSE component is entangled with the baseline used to reconstruct TEMPROB; this does not undermine the core neural-versus-feature-based finding, but it needs to be addressed before the common-sense attribution can be accepted.","major_comments":[{"comment":"The CSE is trained on TEMPROB 'reconstructed using the same method described in Ning et al. (2018b) with the base method changed to CogCompTime.' Since CogCompTime is also the comparison baseline in Tables 2 and 3, the approximately 2-point gain attributed to CSE could partly reflect the model learning to reproduce CogCompTime's own output distribution rather than acquiring independent common-sense knowledge. The authors should compare against the original TEMPROB (Ning et al., 2018b) or a TEMPROB built with an independent base extractor, and/or analyze which errors CSE fixes to show that they are not simply CogCompTime's systematic errors. This is load-bearing for the paper's claim that CSE contributes real temporal common sense.","section":"Section 3.2"},{"comment":"The sentence 'Roughly speaking, the 8% gain is contributed by LSTMs for 2%, contextualized embeddings for 4%, and CSE for 2%' is not supported by the reported ablations; Table 2 shows only Concat versus Concat+CSE, not a controlled decomposition into LSTM, embedding, and CSE contributions. Moreover, the Concat-only rows (e.g., BERT accuracy 69.1 versus CogCompTime 61.6) already show a substantial gain, so the 'about 10%' headline depends on the CSE increment that is subject to the concern raised above. The authors should either present the decomposition as a rough estimate with a supporting ablation, or reframe the main claim to rely on the Concat-only result.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"The sentence 'We think it is important for to understand' contains a typo: 'for to' should be 'for us to'.","section":"Section 1"},{"comment":"The compound 'T EMPROB' is rendered with an inconsistent space in several places (e.g., 'T EMPROB' and 'T EMPROB'); it should be 'TEMPROB' consistently.","section":"Section 3.2"},{"comment":"The column header 'Emb.' lists ELMo and BERT as entries, but the system column only says 'Concat+CSE'; it would be clearer to label the rows explicitly as 'Concat+CSE (ELMo)' and 'Concat+CSE (BERT)'.","section":"Table 3"},{"comment":"The sentence 'The third metric came into use since the TempEval3 workshop (UzZaman et al., 2013), which involves graph closure and reduction on top of the second metric, hoping to better capture how useful a TempRel system is..' contains a doubled period at the end.","section":"Appendix A"},{"comment":"The paper reports only p<0.001 for the main McNemar tests; reporting exact p-values or effect sizes, and ideally confidence intervals for the metric differences, would strengthen the significance claims, especially given the small test set (837 TempRels in MATRES PT).","section":"Section 4.2 and Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The comparison against CogCompTime comes from the same research group that built CogCompTime, and the CSE training resource is reconstructed using that same system. This is a self-referential element that deserves particular scrutiny in revision: the authors should either make the CSE ablation independent or soften the causal attribution to common sense. The Concat-only results, however, already support the paper's main neural-baseline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a useful baseline paper, and the main claim holds up. The Concat+BERT row alone (69.1 acc / 74.4 F1 on MATRES) beats CogCompTime (61.6 / 66.6) by roughly 10 absolute points, with p<0.001 by McNemar. That conclusion is independent of the more exotic CSE component. The paper also does a fair job comparing position-handling methods and embedding types, and the TCR sanity check is a nice bonus.\n\nWhat's actually new: the systematic comparison of simple hidden-state concatenation against the XML-tag position-indicator trick, and the common sense encoder (CSE) idea—using a Siamese network to smooth TEMPROB statistics for rare or unseen verb pairs. The CSE is a plausible way to generalize a sparse knowledge base, and the paper reports consistent gains from it under both ELMo and BERT. The authors also report significance tests, which is more than many papers in this area do.\n\nNow the soft spots, which are localized but real. Section 3.2 says the TEMPROB used to train CSE was reconstructed with CogCompTime as the base method—the same system used as the comparison baseline. That means the 2-point CSE gain could partly reflect the model learning to re-absorb CogCompTime's own output distribution rather than acquiring external temporal common sense. The paper gives no analysis of TEMPROB quality, no comparison with the original Ning et al. (2018b) version, and no ablation with an independently built resource. The framing \"common sense\" is stronger than the evidence supports, and the rough 2%/4%/2% decomposition of gains into LSTM/embeddings/CSE is not accompanied by error bars. The neural results also appear to be single-run, and the submission links to a publication page rather than a code repository with a commit hash. These issues matter for a paper whose selling point is being a strong baseline: reproducibility is the whole point.\n\nNone of this sinks the paper. The Concat rows independently establish the principal contribution, and the CSE concern is a request for better attribution, not evidence of a broken method. This is a paper for anyone working on temporal relation extraction or event ordering, and it deserves a serious referee. My recommendation: send it to peer review, and ask the authors to clarify the CSE training-data provenance, ideally with an ablation using an independently built TEMPROB, and to release code with a version identifier.","headline":"A solid neural baseline paper whose headline result survives scrutiny; the CSE component's 2-point gain is real but not cleanly attributable to external common sense because it was trained on CogCompTime's own output.","tokens_in":10497,"tokens_out":2619,"would_cite":true,"duration_ms":26311,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fairly standard LSTM, given contextualized word embeddings and a Siamese encoder of a temporal common-sense knowledge base, beats the previous best feature-based system on temporal relation extraction by about 10…","keywords":["temporal relation extraction","TempRel","MATRES","common sense encoder","Siamese network","contextualized embeddings","integer linear programming","LSTM baseline"],"falsifier":"Rebuild TEMPROB with a different base extractor, or with no extractor at all, then retrain the Siamese encoder and evaluate Concat+CSE on MATRES and TCR; if the margin over plain Concat disappears or becomes insignificant, the CSE gain is an artifact of the baseline's own supervision rather than generalizable temporal knowledge.","tokens_in":9515,"feed_emoji":"🕰️","tokens_out":4908,"duration_ms":43072,"temperature":0.7,"pith_summary":"Temporal relation extraction — deciding whether one event happens before, after, or simultaneously with another — has resisted neural methods, partly because training data has been small and noisy. This paper argues that a standard LSTM, combined with contextualized word embeddings (ELMo or BERT) and a Siamese encoder trained on a corpus-derived temporal knowledge base, outperforms the previous best feature-based system by about 10 percent absolute accuracy and F1, a 25 percent error reduction. The gains hold on two benchmark datasets, MATRES and TCR. The paper's larger point is that neural approaches were previously limited less by architecture or data size than by annotation quality, and the new system can serve as a strong baseline.","feed_headline":"Standard neural model beats top temporal-relation system by 10%","feed_subtitle":"A standard LSTM plus common-sense knowledge cuts errors by 25 percent on two temporal-ordering benchmarks.","key_machinery":"The system combines four components: (1) an LSTM that encodes each event by the hidden state at its position and concatenates the two event hidden states to represent an event pair; (2) a Siamese network trained to fit TEMPROB, an automatically induced table of typical temporal relations between verb pairs, which generalizes to unseen or rare verb pairs through learned verb embeddings; (3) a feedforward network that turns the concatenated LSTM and CSE representations into confidence scores over before, after, equal, and vague; and (4) integer linear programming that enforces global transitivity constraints on the predicted temporal graph.","core_discovery":"On MATRES, the Concat+CSE system reaches about 71.7 accuracy, 76.7 F1, 66.0 awareness, and 71.5 three-metric average, versus CogCompTime's 61.6, 66.6, 60.8, and 63.0, with p < 0.001 under McNemar's test. The gains are roughly composed of 2 points from the LSTM formulation itself, 4 points from contextualized embeddings, and 2 points from the common sense encoder. On TCR, the same system outperforms CogCompTime by roughly 8 percent on the three-metric average. The paper also finds that simply concatenating the LSTM hidden states at the two event positions works as well as inserting XML position indicators, and that ELMo and BERT give statistically significant improvements over static embeddings while not differing significantly from each other.","pith_inferences":["Because TEMPROB was reconstructed with CogCompTime as the base method, the CSE's contribution may partly encode the baseline's own biases; a cleaner test would train CSE on a knowledge base built with an independent or randomized extractor and check whether the 2-point gain persists.","The paper does not directly probe CSE's generalization to unseen verb pairs; a focused evaluation comparing CSE outputs on rare tuples against human temporal judgments would test whether the Siamese encoder truly supplies common sense or merely interpolates familiar statistics.","If the data-quality interpretation is correct, then collecting larger MATRES-style same-axis annotations should yield further neural gains without changing the architecture, a prediction that future dataset releases could settle.","The reported decomposition into LSTM, contextualized embeddings, and CSE contributions is specific to these benchmarks; transferring the same system to clinical or multilingual temporal corpora would reveal which components travel."],"forward_implications":["A standard LSTM with contextualized embeddings can serve as a strong baseline for temporal relation extraction, replacing feature-based systems without architectural innovation.","The finding that simple concatenation of event-position hidden states matches XML position indicators simplifies future neural designs for this task.","The sizable gains from ELMo and BERT over static embeddings suggest contextualized representations carry information essential for temporal ordering.","The roughly 2-point contribution from the common sense encoder indicates that external temporal knowledge, when injected compactly, adds measurable value beyond context.","If the bottleneck was annotation quality rather than architecture, further improvements should come from more high-quality multi-axis temporal annotations."],"supporting_citations":[{"why":"Supplies MATRES, the high-quality multi-axis temporal relation dataset whose official split the experiments use for training and testing.","marker":"Ning et al. (2018c)"},{"why":"Provides CogCompTime, the feature-based state-of-the-art system that serves as the primary baseline and as the base method for reconstructing TEMPROB.","marker":"Ning et al. (2018d)"},{"why":"Defines TEMPROB and the induction method that the Siamese common sense encoder is trained to fit.","marker":"Ning et al. (2018b)"},{"why":"Supplies the TCR dataset used as the second test set for comparing CogCompTime against Concat+CSE.","marker":"Ning et al. (2018a)"},{"why":"Supplies ELMo contextualized word embeddings, one of the two embedding types that yield the large reported gains.","marker":"Peters et al. (2018)"},{"why":"Supplies BERT contextualized word embeddings, the alternative to ELMo in the experiments.","marker":"Devlin et al. (2018)"},{"why":"Provides the position-indicator (XML markup) event-position handling method that the paper compares against simple concatenation.","marker":"Dligach et al. (2017)"},{"why":"Supplies the Siamese network architecture that the common sense encoder uses to fit TEMPROB and generalize to unseen verb pairs.","marker":"Bromley et al. (1994)"}],"fun_headline_variants":["Neural model with common sense beats temporal baseline by 10%","Temporal relation errors cut 25% by neural model with common sense","Contextual embeddings plus common sense trim temporal errors by 25%","New neural baseline outperforms top temporal system by 10 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The common sense encoder is trained on statistics produced by the very system it is measured against, so if those statistics carry CogCompTime's systematic errors or label biases, the reported 2-point CSE gain is not a clean measurement of external temporal common sense.","fun_headline_variants_meta":{"raw":{"variants":["Neural model with common sense beats temporal baseline by 10%","Temporal relation errors cut 25% by neural model with common sense","Contextual embeddings plus common sense trim temporal errors by 25%","New neural baseline outperforms top temporal system by 10 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1660,"prompt_tokens":845,"completion_tokens":815,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":740}},"tokens_in":461,"tokens_out":815,"duration_ms":6619,"temperature":1.0,"reasoning_tokens":740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:52:26.442341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild TEMPROB with a different base extractor, or with no extractor at all, then retrain the Siamese encoder and evaluate Concat+CSE on MATRES and TCR; if the margin over plain Concat disappears or becomes insignificant, the CSE gain is an artifact of the baseline's own supervision rather than generalizable temporal knowledge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Siamese network architecture that the common sense encoder uses to fit TEMPROB and generalize to unseen verb pairs."}],"review_version":1}