{"id":"494606a5-66fd-4901-8a82-cb6b034a6429","arxiv_id":"2608.13024","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A timestamp-gated retrieval and memory framework (TIEM) with a new holdout benchmark claims consistent gains over ten baselines on five event-driven financial forecasting datasets.","lead":"TIEM is a timestamp-gated framework that combines hypergraph evidence retrieval, case-based skill memory, and joint fusion to forecast stock direction from corporate events, and the authors report it beats ten baselines on five benchmarks. It also introduces a recent-period A-share holdout and a Name-Date Probe to detect training-data contamination that can inflate reported accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baselines may not receive the focal event text; TIEM alone sees the announcement, so its lead could reflect input asymmetry rather than method superiority.","rationale":"The paper's central empirical claim is that TIEM outperforms ten baselines on five benchmarks. For that claim to hold, the baselines must be forecasting from the same event information that TIEM is given. The Appendix B HEFR prompt provides the announcement text in a dedicated block; NDP shows this text is informative. The Fig. 12 case study shows all eight displayed baselines reasoning without the announcement ('event content is unavailable'; an old earnings article is mistaken for the catalyst), while TIEM retrieves the decision-day document via Eq. (1). Because the case study is offered as an illustration of the system's reasoning path, it is in-scope evidence about how the evaluation harness was built. If this is the actual harness, the comparison is not head-to-head: the baselines are handicapped by missing the single most informative input. This concern is independent of the timestamp-leakage issue the reader flags; even with perfect temporal gating, the headline numbers would not establish TIEM's advantage. I would therefore keep a CONDITIONAL verdict, but the condition must include an input-control rerun, and the paper should explicitly state whether every baseline prompt contains the focal-event block. The temporal audit in Appendix H, the NDP, and the three-gate CIP are good-faith contamination controls, and the publicly announced GitHub release is a plus; they do not, however, cure a possible asymmetry in the comparison itself.","tokens_in":28639,"tokens_out":13633,"duration_ms":156919,"concrete_test":"Run a two-condition baseline control on the same 128-instance test sets and both backbones. Condition A reproduces the submitted baseline prompts; Condition B prepends the HEFR '[The event / announcement to react to] {focal_event}' block to every baseline prompt (MemGPT, Mem0, A-MEM, Vanilla RAG, HippoRAG, GraphRAG, LightRAG, HyperGraphRAG, zero-shot, CoT) while leaving each method's retrieval/memory mechanism unchanged. Report Acc/MCC/F1 per condition. If adding the focal-event text lifts any baseline by roughly the TIEM-vs-best-baseline gap, or if TIEM is no longer first on every benchmark under Condition B, the headline comparison is confounded by input asymmetry and must be re-run before the outperformance claim can be accepted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing assumption is not only that timestamps are correct; it is that the baselines are given the same task input as TIEM. The HEFR prompt (Appendix B, Fig. 9) contains a dedicated '[The event / announcement to react to] {focal_event}' block, and Section 5.3 (Fig. 5) reports that this focal-event text contributes positive predictive signal (Delta_content > 0). Yet in the paper's own case study (Fig. 12), every displayed baseline is reasoning without that block: MemGPT/Mem0/A-MEM explicitly say 'The event content for Alphabet Inc. ... is unavailable', Vanilla RAG and HippoRAG treat an old earnings-beat article as 'the catalyst', and GraphRAG/LightRAG/HyperGraphRAG fall back on prior positive headlines because the decision-day event is absent. TIEM alone retrieves the decision-day document through the focal stream Phi(c) (Eq. 1). If this trace reflects the actual evaluation harness, TIEM's large reported lead (e.g., Astock 65.62 vs 60.94, FinPURE 69.53 vs 60.94) may be an input-asymmetry artifact: the other methods are being asked to forecast an event whose text they never see. Section E says baselines share 'test instances, targets, output format' but does not state that all prompts contain the focal event; 'evidence access follows each method's mechanism' is not a substitute. This is a fairness confound independent of timestamp leakage.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"TIEM is a timestamp-gated framework for event-driven catalyst-outcome forecasting. It combines an Event-Evidence Hypergraph (EEH) with Day, Episode, and Theme tiers; a Case-based Skill Memory (CSM) with EVOKE, REVISE, MERGE, and RETIRE primitives; and Heterogeneous Evidence-Experience Fusion Reasoning (HEFR) for a single-call fusion of retrieved evidence and skills. The paper also introduces FinPURE, a recent-period A-share holdout benchmark, a three-gate Contamination Inspection Protocol (CIP), and a Name-Date Probe (NDP) for per-model contamination sensitivity. Experiments on five benchmarks with two backbone LLMs report consistent gains for TIEM over ten baselines, together with ablations, cross-dataset shift analyses, token-efficiency analyses, and cross-backbone consistency analyses. The authors release code and data.","tokens_in":28963,"tokens_out":5169,"duration_ms":54210,"significance":"If the empirical claims hold, TIEM is a valuable contribution to trustworthy financial LLM forecasting: it directly addresses temporal leakage and contamination, which are central concerns in this area; the FinPURE benchmark and the Name-Date Probe are reusable evaluation resources; and the framework is modular and clearly specified. The three-gate protocol and the ancestry-based skill gating are sensible design choices, and the consistency of the reported gains across datasets and backbones is encouraging. The release of code and data is a concrete reproducibility strength. However, the empirical support is currently weakened by the absence of uncertainty quantification on the 128-instance test sets and by an unresolved fairness question about whether baselines receive the same event text as TIEM. These issues are load-bearing for the headline claim and need to be addressed before the paper can be accepted.","major_comments":[{"comment":"Table 1 reports only point estimates (means over three runs) for test sets of 128 instances, with no confidence intervals, standard deviations, or significance tests. For example, on Astock with GPT-5.4-mini the gap between TIEM (65.62) and HyperGraphRAG (61.72) is 3.9 percentage points; with n=128 a paired proportion difference has a standard error in the 4-6 percentage-point range, so several reported differences are not distinguishable from sampling noise. Please report per-run variability and paired bootstrap or McNemar tests for the headline comparisons, and apply the same uncertainty quantification to the ablation results in Table 2.","section":"§5.2, Table 1"},{"comment":"The evaluation may be unfair to the baselines: the HEFR prompt in Fig. 9 includes a dedicated focal-event block, and §5.3 reports that focal-event text contributes positive signal (Delta_content > 0 in Fig. 5), yet the case study in Fig. 12 shows MemGPT, Mem0, and A-MEM stating that 'The event content for Alphabet Inc. ... is unavailable' and Vanilla RAG and HippoRAG using an older earnings article as the catalyst. If this reflects the actual evaluation harness, the baselines were not given the decision-day event text while TIEM received it through Phi(c) in Eq. (1), so the reported lead could be an input-asymmetry artifact rather than evidence of method superiority. Please state explicitly whether all baseline prompts contain the focal event text, and if not, rerun the comparison with identical focal-event input to all methods.","section":"Appendix E and Fig. 12"},{"comment":"The temporal-leakage-free claim rests on timestamp granularity and end-of-day semantics, but the audit in Appendix H is self-reported over one run and does not describe how timestamp correctness was verified against an independent source. Since the CIP protocol and the entire contribution are motivated by leakage prevention, please clarify the provenance and verification of timestamps for each dataset, and quantify sensitivity to timestamp noise (e.g., shifting date-only records by one day) in the main results.","section":"§3 and Appendix H"}],"minor_comments":[{"comment":"Propositions 1-3 are stated in the main text with 'Proof', but the proofs are existential witnesses: Eq. (18) assumes X_j = zeta_j Y_c + epsilon_j with zeta_j >= 0, so the error reduction Pe(I1) < Pe(I0) follows by construction. The paper acknowledges this in the appendix; I recommend moving these to a clearly labeled 'Illustrative sufficiency analysis' and removing the 'Proof' marker from the main text.","section":"Appendix A.1-A.3"},{"comment":"Table 2 reports ablations on Astock only; since the test set is 128 instances, the small differences (e.g., 65.62 vs 64.84) should be interpreted with the same uncertainty quantification requested in the major comment on Table 1.","section":"Table 2"},{"comment":"Figure 12 is a single case study; the paper already states it is not aggregate evidence. Please add a sentence in the main text pointing readers to the aggregate results so the case is not read as representative.","section":"Fig. 12"},{"comment":"Some notation is inconsistent: Eq. (1) uses Phi(c) for the focal text while Eq. (14) uses F(c); H_c in Eq. (5) is clear from context but not defined before use. A table of symbols would help.","section":"Section 4"},{"comment":"The appendix lists many hyperparameters; a small sensitivity study over the most influential ones (e.g., B_char, k1, tau) would strengthen the claim that results are not tuned per dataset.","section":"Table 3 and Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical scope and useful artifacts, but I would not accept it without resolving the baseline-input asymmetry and providing uncertainty quantification for the headline results; both are fixable within a revision. The theoretical appendix should be relabeled as illustrative rather than presented as proofs in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper does something genuinely useful on the evaluation side: the Name-Date Probe, CIP, and FinPURE target a real problem, and the three-gate audit design is a step forward for this subfield. Second, the headline comparison with baselines looks compromised. In the paper's own case study, every displayed baseline reasons without the focal event text—MemGPT, Mem0, and A-MEM explicitly say the event content is unavailable, and the RAG baselines fall back on older, positive headlines—while TIEM receives the decision-day announcement in a dedicated prompt block. Section 5.3 shows focal-event text carries positive predictive signal (Delta_content > 0), so if the baselines never saw that text, a meaningful part of TIEM's reported lead is input asymmetry, not method quality. Appendix E says baselines share test instances, targets, and output format but never states that all prompts contain the event. This needs to be checked against the actual harness; if confirmed, the 2-9 percentage point gains are uninterpretable.\n\nWhat is new and good: the combination of timestamp-gated hypergraph retrieval (EEH), ancestry-gated skill memory (CSM), and budgeted single-call fusion (HEFR) is not in the cited literature, and the framework is described in unusual detail. The ablations show consistent component contributions, and the cross-backbone consistency analysis is a nice touch. The Appendix A proofs are not proofs of efficacy; they construct toy Gaussian or binary-symmetric witnesses and assume outcome-informative components, so they are essentially existence arguments. To the paper's credit, it labels them as sufficient-condition witnesses rather than general guarantees, but they add little.\n\nSecondary soft spots: each test set is only 128 instances, with no confidence intervals or significance tests on the main table; the temporal audit in Appendix H is self-reported and only covers one run rather than the exact test set; and date-only end-of-day semantics remains a real, acknowledged leakage risk. The citation pattern looks reasonable and the baseline coverage is appropriate. The math, such as it is, is correct but not informative.\n\nWho this is for: people building temporal LLM evaluation protocols and event-driven financial forecasting systems. It deserves a serious referee despite my skepticism about the empirical claim, because the evaluation artifacts and the framework itself are worth engaging with. My recommendation: ask for the code and the exact prompts, make the focal event available to all methods (or explicitly justify withholding it), add error bars to the main table, and reframe the appendix theorems as toy illustrations. If the asymmetry is real, the paper should be repositioned as an evaluation-tooling contribution rather than a new state-of-the-art forecasting method.","headline":"Worth a referee and possibly a reading group, but the main empirical claim is suspect until the baseline prompt asymmetry is resolved.","tokens_in":29546,"tokens_out":3282,"would_cite":true,"duration_ms":38600,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TIEM claims that timestamp-gating all evidence and stored skills — admitting only what was truly available before the decision — improves event-driven financial forecasting and beats ten baselines on five benchmarks.","keywords":["event-driven financial forecasting","temporal leakage","timestamp gating","hypergraph retrieval","skill memory","contamination audit","name-date probe","FinPURE benchmark"],"falsifier":"Independently audit all 640 test instances: for each catalyst, verify against wire-level or exchange-level timing that every focal and concurrent record's timestamp precedes its decision time, that every retrieved Episode and Theme ends strictly before it, and that every skill ancestor's resolve time does too — rather than trusting the corpus's own stamps or the paper's self-reported audit of 1,280 cases from an unspecified subset. If a single test instance contains a record that became available after its decision time, or if recomputing FinPURE labels directly from the stored price windows changes any direction, the leakage-free claim fails as stated.","tokens_in":28430,"feed_emoji":"📈","tokens_out":18315,"duration_ms":152325,"temperature":0.7,"pith_summary":"The paper's aim is to close the 'Evidence Chasm' in event-driven financial forecasting: the gap between reported benchmark accuracy and genuine predictive skill that opens when training-data contamination and temporal leakage let a model peek at information it should not have. It proposes TIEM, a timestamp-gated framework in which every piece of evidence — focal event text, prior news, multi-day episodes, cross-stock themes, and stored 'skills' distilled from past outcomes — must be verifiably available before the decision time, with each skill rejected unless its complete provenance ancestry resolves earlier. On five benchmarks spanning Chinese A-share and U.S. equity markets across two LLM backbones, TIEM reports the strongest accuracy, Matthews correlation, and macro-F1 among eleven methods, and keeps its lead under cross-dataset, cross-time, and recent-period settings. The paper also contributes FinPURE, a recent-period A-share earnings holdout whose labels can be recomputed from stored price windows, and a Name-Date Probe whose diagnostics indicate that names and dates alone carry no reliable predictive advantage. A sympathetic reader should care because the claim is not just that more context helps forecasting, but that a strict information-availability discipline improves scores precisely where contamination would otherwise inflate them.","feed_headline":"Timestamp-gated framework beats 10 baselines in market forecasts","feed_subtitle":"A strict availability gate admits only pre-decision evidence, lifting accuracy, MCC, and F1 across five benchmarks.","key_machinery":"The load-bearing mechanism is the temporal availability gate, made operational by three coordinated components. The Event-Evidence Hypergraph (EEH) organizes timestamped text into Day-tier atomic facts, Episode-tier windows (bounded multi-day causal chains per stock), and Theme-tier clusters (cross-stock co-movements), each record carrying an explicit temporal extent [τs, τe], so retrieval at decision time T admits only facts with ts ≤ T and Episodes/Themes with τe < T. The Case-based Skill Memory (CSM) stores outcome-derived IF-THEN skills tagged with domain, an EMA-updated advantage score, a stability state, and complete source ancestry A; its ancestry gate requires every ancestor's resolve time to precede T and fails closed otherwise, which is what keeps reused experience temporally honest. Heterogeneous Evidence-Experience Fusion Reasoning (HEFR) then packs five budgeted streams — focal, prior, Episode, Theme, experience — plus a budget-exempt concurrent block into one joint LLM call, using a character-allocation scheme that conserves the total budget across the streams. The gate is the active ingredient: it converts retrieval from pure similarity matching into similarity restricted to records whose availability is verifiable, and the temporal-gate ablation in the paper shows that removing it costs roughly six accuracy points on one backbone and ten on the other.","core_discovery":"The central claim is that temporal leakage and training-data contamination, not model capability, are the binding constraint on trustworthy event-driven catalyst-outcome forecasting, and that the constraint can be addressed by construction rather than by hoping a model's training cutoff protects the evaluation. TIEM enforces at every step that nothing entering the prediction context postdates the decision: focal and same-day concurrent evidence may be dated at or before T, prior day records, Episodes, and Themes must end strictly before T, and each retrieved skill carries a complete ancestry of the outcomes it was distilled from, all resolving before T, with missing or late ancestry failing closed. On this gated substrate, evidence is organized at three temporal scales — atomic Day facts, per-stock Episode chains, and cross-stock Theme clusters — and fused with retrieved skills by Heterogeneous Evidence-Experience Fusion Reasoning in a single budgeted LLM call. The reported results are that TIEM achieves the strongest average accuracy, Matthews correlation, and macro-F1 on both of two backbone LLMs across all five benchmarks, leads in cross-dataset and recent-period transfer, and that ablations attribute the gain to each of the three components in turn. The authors frame the contribution as a transferable recipe: timestamped evidence and provenance-tracked experience remain effective across markets, periods, and model choices, under a contamination-inspection protocol that does not assume a universal training cutoff.","pith_inferences":["My inference: the three-gate audit discipline — admit evidence only if verifiably available, require complete ancestry, fail closed on missing metadata — transfers beyond finance to any time-sensitive prediction task (clinical outcomes, geopolitical risk, supply-chain disruption), where the honest metric is accuracy after an availability audit, not accuracy on a static benchmark.","My inference: the paper does not isolate whether cross-dataset transfer comes from evidence transfer or skill transfer; ablating CSM specifically on the CMIN-US, EDT, and CSMD shift benchmarks would settle whether outcome-derived skills carry the cross-market gain or the hypergraph evidence does.","My inference: the appendix proofs are existence witnesses — a Gaussian signal-detection setting for EEH, a stability analysis of the EMA advantage for CSM, and a repetition-code argument for HEFR — so they show each component can help under favorable conditions, not that it must help on arbitrary natural-language evidence; the five-benchmark results are the primary evidence."],"forward_implications":["If TIEM's results hold, timestamp-gated evidence with provenance-tracked skill reuse is a transferable recipe rather than a dataset-specific trick: it leads on two distinct LLM backbones (DeepSeek-V4-Flash and GPT-5.4-mini) and across Chinese and U.S. equity benchmarks with three- and five-day horizons.","The lead persists under cross-market, cross-time, and recent-period evaluations, including the FinPURE holdout whose labels can be recomputed from stored price windows, so the reported advantage is not confined to in-distribution examples.","Component ablations attribute the gain to all three pieces — EEH's multi-tier temporal organization, CSM's outcome-informed skill reuse, and HEFR's single-call joint fusion — implying the three mechanisms do complementary work rather than one carrying the result.","The Name-Date Probe shows names and dates alone stay at the base rate while event-content gains are positive, supporting the paper's claim that the forecasts are content-grounded rather than driven by identity shortcuts.","The accuracy gain is not bought with unbounded context: TIEM sits on the accuracy–token Pareto frontier and leads the log-adjusted and per-1k-token trade-offs across the five benchmarks."],"supporting_citations":[{"why":"HyperGraphRAG is the strongest structured-retrieval baseline TIEM must beat, and supplies the hypergraph evidence representation that EEH extends with explicit temporal extents and gates.","marker":"(Luo et al., 2025)"},{"why":"Mem0 is the production memory baseline whose flat persistent recall motivates CSM's source-tagged, ancestry-gated skill memory.","marker":"(Chhikara et al., 2025)"},{"why":"A-MEM represents the agentic memory baseline with linked structured notes that TIEM contrasts against and outperforms.","marker":"(Xu et al., 2025)"},{"why":"GraphRAG establishes the graph-community retrieval paradigm in the baseline family TIEM is compared with.","marker":"(Edge et al., 2024)"},{"why":"HippoRAG supplies the graph-based memory-retrieval baseline and the provenance-aware association structure that CSM's ancestry gate extends to time.","marker":"(Gutiérrez et al., 2024)"},{"why":"MemGPT is the tiered memory-management baseline anchoring the memory-agent family in the comparison.","marker":"(Packer et al., 2023)"},{"why":"Astock is the primary in-distribution Chinese A-share benchmark carrying the main results and ablations.","marker":"(Zou et al., 2022)"},{"why":"CMIN-US is the U.S. equity benchmark that tests whether TIEM's lead transfers across markets.","marker":"(Luo et al., 2023)"}],"fun_headline_variants":["Beat 10 baselines with a timestamp gate against leakage","TIEM: Time-locking evidence for trustworthy market forecasts","Stop the leak: TIEM gates evidence for accurate predictions","TIEM: Where temporal leakage ends and accuracy begins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that timestamps are trustworthy: the paper assumes that record timestamps, and the end-of-day semantics applied to date-only records, accurately reflect when information truly became available, so that the temporal gates genuinely exclude future evidence. If timestamps are missing, coarse, or misassigned, the leakage-free claim collapses and the reported gains could be contamination rather than forecasting skill.","fun_headline_variants_meta":{"raw":{"variants":["Beat 10 baselines with a timestamp gate against leakage","TIEM: Time-locking evidence for trustworthy market forecasts","Stop the leak: TIEM gates evidence for accurate predictions","TIEM: Where temporal leakage ends and accuracy begins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3396,"prompt_tokens":985,"completion_tokens":2411,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2345}},"tokens_in":601,"tokens_out":2411,"duration_ms":19587,"temperature":1.0,"reasoning_tokens":2345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:19:49.218431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently audit all 640 test instances: for each catalyst, verify against wire-level or exchange-level timing that every focal and concurrent record's timestamp precedes its decision time, that every retrieved Episode and Theme ends strictly before it, and that every skill ancestor's resolve time does too — rather than trusting the corpus's own stamps or the paper's self-reported audit of 1,280 cases from an unspecified subset. If a single test instance contains a record that became available after its decision time, or if recomputing FinPURE labels directly from the stored price windows changes any direction, the leakage-free claim fails as stated.","supporting_citations":[],"review_version":1}