{"id":"99affb4a-ac3c-4c50-9a87-46fb3a751192","arxiv_id":"2507.14160","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new public benchmark of 16 survival prediction tasks derived from 7.6 million DeFi lending records, with baseline results for six survival and six classification models.","lead":"This paper releases FinSurvival, a set of 16 survival-analysis benchmark datasets built from 7.6 million Aave cryptocurrency lending transactions. It is meant to give AI and statistics researchers a large, public, high-censoring testbed for time-to-event prediction, a resource that is scarce outside medicine.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark metrics treat user-repeated index events as independent; the temporal split leaves the same users in train and test, so reported C-indices/AUCs may not reflect generalization to unseen users.","rationale":"I read the paper as a dataset and benchmark contribution. The dataset construction from Aave transactions is described in detail, the release appears feasible, and the authors' prior DeFi survival work supports the domain expertise. For the central evaluation claim, that the 16 tasks are challenging and that current methods perform as reported, the most load-bearing assumption is that the roughly 7.6 million records can be scored as independent survival observations. Section 2.2 creates a temporal split, but the subject definition in Section 2.1 is user and coin, and Table 3 reports only 114,861 unique users for 7,698,497 records. Therefore, the test set contains many index events from users whose earlier events fall in the training window. The user-history features in Tables 12-13 are legitimate time-varying covariates, but when the same users appear on both sides of the split, the benchmark conflates predicting a known user's next event from their own history with predicting for a user never observed. This is not an internal inconsistency, and it does not invalidate the dataset; it means the benchmark numbers cannot yet support the generalization claim attached to them. A user-disjoint or user-clustered evaluation is the natural remedy. The RMST threshold concern in Section 4.1 and the sub-0.5 GBM C-indices are real and should be fixed, but they are narrower: the first affects only the classification threshold construction, and the second is a single implementation red flag. The independence and nesting issue cuts across all 16 tasks and both task families, which is why I select it as the load-bearing concern. The paper itself flags competing risks as a limitation in Section 6 but does not flag user-level dependence, so this is a missing limitation rather than a disclosed one. My recommendation is CONDITIONAL, consistent with the reader's verdict, with the condition being a user-aware evaluation that either confirms the reported metrics or leads to revised claims.","tokens_in":21837,"tokens_out":9185,"duration_ms":104404,"concrete_test":"Recompute the Table 7 C-indices and Table 8 AUCs for all 16 tasks under two alternative splits using the released pipeline: (i) cluster-bootstrap by user, resampling the 114,861 users with replacement and keeping all of their records, and report a 95% confidence interval for each metric; (ii) a user-disjoint test set containing only records from users whose first observed transaction is after July 1, 2022, while training on all records before the cutoff. If the aggregate metrics, such as the XGBoost mean C-index of 0.783, shift by more than 0.03, or if the model rankings change, the 'challenging tasks' claim must be revised to specify the evaluation regime. If the metrics are unchanged, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's 'challenging tasks' and model-comparison claims rest on C-index and AUC numbers computed over 7,698,497 records, but those records are not independent: each user contributes many index events (Section 2.1, Table 3 reports 114,861 unique users), and the July 1, 2022 cutoff in Section 2.2 does not exclude users who have index events on both sides of the split. Because the 128 features include user-history covariates (Section 2.1, Tables 12-13) computed from events up to the index time, test records of active users carry information about behavior observed during the training window. The reported metrics therefore mix predicting a user's future event from their own past history with predicting for a user never seen in training, and the paper does not distinguish these regimes. This is not a circularity or an internal inconsistency; it is a correctness risk for the benchmark claim. If the benchmark is meant to test survival models that generalize to new users or to compare methods fairly, a user-clustered or user-disjoint evaluation is required. The dataset release itself can remain valid, but the evaluation claims need to be scoped to the repeated-events setting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinSurvival, a suite of 16 survival analysis tasks constructed from Aave V2 Ethereum DeFi lending transactions, totaling 7,698,497 records and 128 features, with a mean censoring rate of 81.26%. The tasks are formed by pairing four index event types (borrow, deposit, repay, withdraw) with four outcome event types per index, with subjects defined as (user, coin) pairs. The paper also creates 16 binary classification tasks by thresholding the survival time at a per-task restricted mean survival time (RMST), and benchmarks six survival models (Cox, AFT, GBM, XGBoost, DeepSurv, DeepHit) using C-index and six classification models (LogReg, Decision Tree, XGBoost, Elastic Net, DeepHit, Neural Net) using AUC. It claims this is the first large-scale, publicly available financial survival dataset derived from DeFi transactions and that existing methods find the tasks challenging.","tokens_in":21956,"tokens_out":7615,"duration_ms":80352,"significance":"If the dataset and evaluation are sound, this is a valuable public resource: it is orders of magnitude larger than existing public survival datasets, has high-dimensional features, heavy censoring, and real financial behavior, and the open-source pipeline and public raw data are reproducibility strengths. The benchmark results, once the evaluation issues below are addressed, would provide a useful testbed for survival models under repeated-event and high-censoring conditions. The paper makes no machine-checked proof or parameter-free derivation claims, but the dataset release itself is a concrete, falsifiable contribution that can be independently validated.","major_comments":[{"comment":"The classification labels are defined by an RMST threshold, but the paper never states whether the RMST is computed on the training set, the full dataset, or the test set. Section 4.1 says 'We calculated RMST(τ) for increasing values of τ in one-day increments and stopped when the change in RMST(τ) was less than 5%' and Section 2.2 says the buffer length was selected 'in conjunction with the restricted mean survival times (RMSTs) for each task.' If the RMST is estimated from the full data (including the post-July-2022 test period), then test labels are defined using test outcomes, which is label leakage and undermines the temporal-split logic. Please specify the estimation scope, and if thresholds were derived from full data, recompute them on the training set only and re-run the classification benchmark in Table 8.","section":"Section 4.1 (and Section 2.2)"},{"comment":"The benchmark treats all 7,698,497 records as independent, but records are repeated events nested within 114,861 users (subjects are user-coin pairs; Section 2.1), and the July 1, 2022 split does not exclude users who appear in both training and test sets. Because the features include user-history aggregates computed from events before the index time (Tables 12-13), test records of active users contain information about behavior observed during the training window. The reported C-indices and AUCs therefore mix within-user prediction (using the user's own past) with prediction for never-seen users, and may overstate generalization to new users. Please provide a user-disjoint temporal split or a cluster-robust evaluation (e.g., metrics reported separately for users appearing only in test versus users overlapping with training), and scope the 'challenging tasks' claim accordingly.","section":"Sections 2.1, 2.2, 3.2, 4.3 (Tables 7 and 8)"},{"comment":"The GBM model achieves C-index values near or below 0.3 on nearly all tasks, which is far below the 0.5 random-guessing baseline. This consistent anti-predictive pattern strongly suggests a bug in the implementation or evaluation (e.g., inverted predictions or a survival-objective coding error) rather than a genuine property of the GBM. Since the paper uses these results to conclude that existing methods are not well addressed by the tasks, the authors should verify the GBM predictions (e.g., check concordance orientation on a subsample, or flip predictions to see if C-index rises) and either correct the implementation or explicitly report and explain the anomaly.","section":"Section 3.2 and Table 7"}],"minor_comments":[{"comment":"The novelty claim ('to the best of our knowledge, this is the first large-scale, publicly available financial survival dataset derived from DeFi transactions') should be supported by a brief description of the search process (databases, queries, keywords) or softened to 'we are not aware of', since it is not a systematic claim.","section":"Section 1"},{"comment":"The text refers to 'section 2.1 for a visualization' and to Kaplan-Meier curves 'in section 2.1'; these should reference Figure 1 and Figure 2 respectively.","section":"Section 2.1"},{"comment":"The RMST calculation method ('monotonic spline method') is cited to a journal article rather than to a specific software implementation; please clarify the package/software used for the spline and the convergence check, and report whether the 5% change is measured as an absolute day difference or a relative change.","section":"Section 4.1"},{"comment":"The classification construction drops censored observations with censoring time less than RMST; the paper should report how many records are dropped per task (the difference between the survival and classification counts in Table 4) and discuss any selection bias this introduces.","section":"Section 4.1 and Table 4"},{"comment":"The benchmark tables do not report variance or confidence intervals for the C-index and AUC scores; given that several model differences are small (e.g., LogReg vs Elastic Net mean AUC 0.795 vs 0.793), the Borda-rank-based ordering should be interpreted with caution.","section":"Sections 3 and 4 (Tables 7 and 8)"},{"comment":"The claims that Aave is 'inherently fair' and that the possibility of unethical or fraudulent use is 'very small' are unsupported opinions and should be removed or heavily qualified, as they are outside the technical analysis.","section":"Broader Impact Statement"},{"comment":"DeepSurv and DeepHit are trained on subsamples (40,000/20,000 rows) while other models use the full data; this asymmetry should be stated in the main text near Table 7 to avoid unfair comparisons.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset is a potentially useful contribution and the public release is commendable. The main obstacle is the evaluation: the user-dependence and RMST-scope issues need to be resolved before the benchmark claims can be fully trusted. If the authors provide user-disjoint or cluster-robust evaluation results and clarify the RMST estimation scope, the paper would be substantially stronger and in scope for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nFinSurvival is a genuinely useful resource: a public, 16-task survival benchmark from Aave DeFi transactions, 7.6M records, 128 features, with code and pipeline released. That is new relative to the prior literature, and the dataset release itself looks solid. The authors document the raw data source, the rolling-join construction, the feature themes, and the censoring structure, and they ship code to reproduce it. Credit where due: this is the kind of artifact the survival community needs.\n\nThe evaluation claims are softer than the dataset. Three things stand out.\n\nFirst, the temporal split does not separate users. Records are nested in 114,861 users, many of whom have index events on both sides of July 1, 2022, and user-history features are computed up to the index time. So the reported C-indices and AUCs mix \"same user, new event\" with \"never-seen user\" and likely overstate generalization to new users. A user-clustered or user-disjoint split, or at least a per-user evaluation, is needed to scope the \"challenging tasks\" claim.\n\nSecond, the RMST threshold for the classification labels: the paper never states whether the RMST is computed on the training split only or the full data. If it is the full data, the threshold itself is informed by test outcomes, which is a mild label leakage. This is easy to fix and should be stated.\n\nThird, the GBM baseline producing c-index values around 0.3 across many tasks is flagged by the authors as worse than random, but they do not investigate whether it is an implementation inversion (e.g., predicting risk instead of survival time, or a sign flip). Leaving that unexplained undermines the model-comparison table. It is not fatal to the dataset, but it should be resolved or the GBM results removed.\n\nThe circularity burden is low: the only fitted quantity is the per-task RMST threshold, and the leakage concern is addressable. The underlying data is public and the code is released, so these are fixable evaluation issues, not load-bearing flaws in the resource itself.\n\nWho is this for: benchmarking deep survival models, especially high-censoring, high-dimensional financial data. I would cite it if I work in that area, and it deserves a serious referee. A conditional accept with those three evaluation fixes would be appropriate.","headline":"A genuinely useful public DeFi survival benchmark whose evaluation claims are weakened by user leakage and two addressable methodological ambiguities.","tokens_in":22603,"tokens_out":1888,"would_cite":true,"duration_ms":19804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62N01","62N02"],"pacs":[],"model":"deepseek-v4-flash","headline":"FinSurvival turns 7.7 million public DeFi lending transactions into 16 survival-model benchmarks, and shows existing methods, especially deep learning, struggle on them.","keywords":["survival analysis","benchmark","decentralized finance","DeFi","time-to-event data","censored data","Aave","machine learning evaluation"],"falsifier":"Recompute the C-index and AUC after holding out entire wallets: train on transactions from one set of users and evaluate only on transactions from users never seen in training. If the mean C-index of XGBoost and the mean AUC of logistic regression drop toward 0.5, the paper's claim that the task is learnable across users would be weakened; if scores stay at the reported levels, the independence concern is empirically resolved.","tokens_in":21549,"feed_emoji":"📊","tokens_out":3359,"duration_ms":41103,"temperature":0.7,"pith_summary":"The paper aims to close a gap in survival analysis: researchers in finance, medicine, and industry need large, realistic, freely available time-to-event datasets, but most public ones have only thousands of records and few features. It constructs FinSurvival, a suite of 16 survival tasks derived from public transaction data of the Aave V2 lending protocol, with 7,698,497 records, 128 features, and a mean censoring rate of 81.26%. Each task answers a concrete behavioral question, such as how long a user takes to repay a crypto loan after borrowing it. The paper also converts each task into a binary classification problem by thresholding survival time with the restricted mean survival time, creating 16 companion classification benchmarks. Benchmarking six survival models and six classification models shows the tasks are challenging and not well solved by current methods, making FinSurvival a practical testbed for future AI survival modeling.","feed_headline":"7.7M DeFi transactions become 16 survival benchmarks","feed_subtitle":"First large public financial time-to-event suite; 81% censoring makes it a hard test for AI models.","key_machinery":"The central mechanism is the automated index-event/outcome-event pipeline: it takes raw Aave V2 transactions, groups events by user and coin, performs a rolling join so each index event is matched to the first subsequent outcome event by the same subject, and records the elapsed time, censoring any index event that never sees an outcome before the observation window ends. Two design choices carry the argument: the subject definition (user plus reserve) keeps the survival semantics aligned with actual lending behavior, and the 128 features combine raw transaction fields with user-history, market-history, and cyclic time encodings. A secondary mechanism is the restricted mean survival time threshold, computed by increasing the truncation horizon until the estimated RMST changes by less than five percent, which turns each survival task into a binary classification task with a short horizon of roughly 17 to 21 days.","core_discovery":"The central claim is that a large-scale, realistic financial survival benchmark can be built entirely from public blockchain transaction data, without any private or paid information, and that doing so reveals important limitations in current survival models. The authors define survival records by pairing an index event (borrow, deposit, repay, or withdraw) with a later outcome event (the same four types plus account liquidation), using the pair of user and coin as the subject so that, for example, the time from borrowing Wrapped Bitcoin to repaying Wrapped Bitcoin is tracked correctly. They report that no existing large public financial survival dataset is derived from DeFi transactions, and their pipeline produces 16 datasets with a mean censoring rate of 81.26%, including highly imbalanced tasks where class 1 rates are as low as 0.2%. On the time-to-event tasks, XGBoost achieves the highest concordance index with a mean of 0.783, followed by accelerated failure time regression at 0.747, while DeepHit and DeepSurv score near or below random guessing; on the classification tasks, linear models lead with mean AUC around 0.795, and XGBoost underperforms. These results are the paper's evidence that the suite offers a demanding, realistic evaluation environment spanning large scale, high censoring, rich covariates, and skewed outcomes.","pith_inferences":["The reported concordance indices and AUC scores likely overstate generalization to unseen users, because each of the 114,861 wallets contributes many index events and the July 2022 cutoff can place records from the same wallet in both training and testing sets; a clustered evaluation holding out entire users would be a sterner test and could change model rankings.","The short RMST horizons of roughly 17 to 21 days mean the classification tasks mostly reward predicting whether a behavior occurs within three weeks, so high AUC there may not imply good long-horizon risk discrimination; combining the classification and time-to-event results would give a fuller picture.","Because liquidations compete with repayments as possible outcomes after a borrow, the single-outcome design likely oversimplifies real DeFi dynamics; explicitly modeling competing risks, which the paper lists as future work, could alter both task difficulty and the best-performing methods.","The pipeline should transfer to other public financial event logs, such as decentralized exchange trades or crowdsourced lending records, so even without new data collection one could check whether the observed model rankings generalize across financial domains."],"forward_implications":["If the benchmark is correct, deep survival models can finally be stress-tested at a scale far beyond existing public datasets, since FinSurvival contains roughly 150 times more records than the largest SurvSet entry.","The RMST-based classification companion tasks make the suite usable by standard classifiers, not only specialized survival methods, broadening who can benchmark against the data.","The demonstrated strength of XGBoost and AFT over deep survival models on these tasks implies that model choice depends heavily on data regime, and that existing deep survival approaches have headroom for improvement on heavily censored, high-dimensional financial data.","The release of raw transaction data and the pipeline itself means new DeFi protocols and exchanges can be turned into new survival benchmarks as the ecosystem grows, without the pipeline being rebuilt from scratch."],"supporting_citations":[{"why":"Defines the Aave protocol whose V2 Ethereum transactions are the raw data source for all 16 datasets.","marker":"[Boado, Ernesto (2020)]"},{"why":"Prior DeFi survival analyses that motivate the index/outcome framing and provide domain knowledge for the 128 engineered features.","marker":"[Green et al. (2022, 2023)]"},{"why":"SurvSet repository used as the comparison baseline showing existing public survival datasets are much smaller.","marker":"[Drysdale (2022)]"},{"why":"Supplies the Cox proportional hazards model used as a benchmark survival method.","marker":"[Cox (1972)]"},{"why":"XGBoost is the top-performing time-to-event model with mean C-index 0.783.","marker":"[Chen and Guestrin (2016)]"},{"why":"DeepHit is the deep learning survival baseline used in both prediction and classification benchmarks.","marker":"[Lee et al. (2018)]"},{"why":"Provides the restricted mean survival time method the paper uses to threshold survival times into classification labels.","marker":"[Uno et al. (2014)]"},{"why":"DeepSurv is the Cox-based neural network baseline whose poor performance motivates the paper's conclusions about deep survival models.","marker":"[Katzman et al. (2018)]"}],"fun_headline_variants":["7.5M DeFi records power 16 survival benchmarks","New financial survival suite: 16 tasks from DeFi","DeFi transactions craft 16 survival modeling challenges","Crypto loans provide 16 tough survival time tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark treats each of the 7.6 million records as an independent observation even though 114,861 users each generate many records, and the temporal train/test split does not exclude the same user from both sides, so without a clustered or user-level analysis the reported scores may not measure generalization to new users.","fun_headline_variants_meta":{"raw":{"variants":["7.5M DeFi records power 16 survival benchmarks","New financial survival suite: 16 tasks from DeFi","DeFi transactions craft 16 survival modeling challenges","Crypto loans provide 16 tough survival time tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1541,"prompt_tokens":1103,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":719,"tokens_out":438,"duration_ms":4922,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:36:12.166651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the C-index and AUC after holding out entire wallets: train on transactions from one set of users and evaluate only on transactions from users never seen in training. If the mean C-index of XGBoost and the mean AUC of logistic regression drop toward 0.5, the paper's claim that the task is learnable across users would be weakened; if scores stay at the reported levels, the independence concern is empirically resolved.","supporting_citations":[{"cited_title":"AAVE Protocol Whitepaper","cited_arxiv_id":null,"evidence_quote":"Defines the Aave protocol whose V2 Ethereum transactions are the raw data source for all 16 datasets."}],"review_version":1}