{"id":"33f2fa0e-4416-4659-9803-fd51076deb96","arxiv_id":"2511.18613","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On the tested stock-price data, LSTM forecasts are several times more accurate than baseline KAN, which trains faster but is far less accurate.","lead":"This paper benchmarks standard Kolmogorov-Arnold Networks against LSTMs on stock-price forecasting and finds LSTMs far more accurate, with KANs faster to train. It frames KAN as promising for resource-constrained or interpretability-focused settings, but the quantitative claims rest on single-run comparisons without error bars.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 10x LSTM advantage is unsupported: Table 3's LSTM/KAN RMSEs contradict the model-selection tables (Table 1 and Table 2), so the headline magnitude is not verifiable from the paper's own results.","rationale":"The paper's contribution is the empirical quantification of a plausible architectural difference. The central claim is not merely 'LSTM is better'—it is the specific 6.5–10x margin, which is repeated in abstract, results, discussion, and conclusion. The reader's conditional verdict already flags test-set selection as a concern. My stress-test found a more direct problem: the paper's own tables do not agree. Table 2's best KAN RMSE is 0.152 and its worst is 0.331; Table 3 uses 0.390. Table 1's LSTM 100u-linear RMSE is 0.0829; Table 3 uses 0.039. Unless the tables are from different datasets/splits/horizons (not documented), the headline comparisons are not derivable from the reported experiments. The duplicated rows in Table 2 reinforce the impression of reporting errors. This does not imply the broad conclusion is false—LSTM is known to be strong on sequential data—but it makes the paper's quantitative contribution unreliable as written. The honest non-paranoid reading is that the direction of the finding may be right, but the magnitude should be re-verified with a reproducible pipeline. Thus the reader's CONDITIONAL verdict is appropriate; I would not upgrade or downgrade it without that verification.","tokens_in":9315,"tokens_out":4394,"duration_ms":38215,"concrete_test":"Reproduce Table 3's 1-day Normal row from the paper's stated best configurations (LSTM 4 layers/100 units/linear; KAN Config 5/6) using the same train/test split and the code intended at the GitHub URL. If LSTM RMSE is ≈0.083 and KAN ≈0.152, then the reported 0.039 vs 0.390 is not reproducible and the advantage drops from 10x to ~1.8x. Also inspect whether Table 2's duplicate rows (Configs 1/2, 5/6) arise from copy-paste errors. If exact reproduction is impossible because the dataset is not described, treat the quantitative claims as unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest quantitative claim—Table 3's 10x LSTM advantage, e.g. LSTM 0.039 vs KAN 0.390 for 1-day Normal—is the load-bearing result, and it is not internally consistent with the paper's own model-selection tables. Table 2 reports every KAN test RMSE as 0.152–0.331, with the stated best config at 0.152 (Config 5/6); none is 0.390. Table 1 reports the LSTM configuration Table 3 labels 'best' for 1-day (4 layers, 100 units, linear) as having test RMSE 0.0829, not 0.039. Thus the 6.5–10x ratios are computed from numbers that do not appear in the reported experiments, or from an undescribed different data split/protocol. Additionally, Table 2 configs 1/2 and 5/6 have identical train/test RMSEs despite differing k/grid or neuron counts, suggesting duplicated/misreported rows. The 200-day rows list LSTM as N/A while the abstract/text claims LSTM superiority 'across all tested prediction horizons.' Until the source of these discrepancies is resolved, the central magnitude claim lacks support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled empirical comparison of baseline Kolmogorov-Arnold Networks (KAN, implemented with PyKAN) and Long Short-Term Memory (LSTM) networks for forecasting stock price data. Accuracy is measured by RMSE in normalized feature space under a direct multi-step forecasting protocol with 20-day look-back windows. The central claim is that LSTM achieves 6.5–10x lower RMSE across 1-day, 2-day, 100-day, and (where available) 200-day horizons, while KAN trains about 2.1x faster. The paper further argues for complementary roles: LSTM for accuracy-critical short-term forecasting and KAN for interpretability, speed, and extended-horizon forecasts. The scope is explicitly limited to baseline KAN and baseline LSTM, excluding specialized temporal KAN variants.","tokens_in":9575,"tokens_out":3231,"duration_ms":32641,"significance":"If the quantitative claims were reliable, the paper would provide a useful benchmark for baseline KAN on stochastic financial time series and would reinforce the established view that LSTM remains the default for accuracy-critical sequential forecasting. Strengths include the direct architecture-to-architecture comparison, open code links, and an explicit scope limitation to baseline variants. However, the current manuscript's central magnitude claim is not verifiable from its own tables: the LSTM/KAN RMSEs in Table 3 do not match the model-selection tables, and the configuration selection protocol is invalid for generalization claims. These issues make the headline 7–10x advantage unsupported as presented.","major_comments":[{"comment":"The central RMSE values underlying the 6.5–10x advantage are not present in the experiments reported earlier. Table 3 lists LSTM 1-day RMSE as 0.039 with '100u-linear' as best config, but Table 1 reports this exact configuration (4 layers, 100 units, linear) with test RMSE 0.0829 and reports the overall best LSTM test RMSE as 0.0745 (2 layers, 10 units, tanh). Similarly, Table 3 lists KAN 1-day RMSE as 0.390, while Table 2 reports all KAN test RMSEs in the range 0.152–0.331. The paper must reconcile these numbers or state the exact data split, normalization, and evaluation protocol that produced Table 3; otherwise the headline accuracy advantage is unsupported by the paper's own evidence.","section":"§5.4, Table 3 vs Tables 1 and 2"},{"comment":"The 'Best Config' column in Table 3 is selected from test RMSE values, with no validation set, no seed averaging, and no confidence intervals. For example, the 1-day LSTM config is '100u-linear' despite Table 1 showing this config with higher test RMSE than the 10u-tanh config; the selection appears to be an a posteriori choice based on the test set. This makes the reported 7–10x advantage an upper-bound artifact of test-set fitting rather than a fair architectural comparison. A proper protocol should use a validation split for model selection and report mean and variance over multiple random initializations or, at minimum, a clear statement of how configurations were chosen without access to test labels.","section":"§5.4 and §4.1–4.2"},{"comment":"There is an internal contradiction about the 200-day horizon. Table 3 reports LSTM RMSE as N/A for all 200-day rows, while the abstract and §5.4 claim LSTM is superior 'across all tested prediction horizons,' and §5.5 lists LSTM as 'best overall' without excluding 200-day. The discussion also states that LSTM 'faces computational constraints beyond 100-day predictions' (§5.6), but no runtime or memory evidence for this claim is provided. Either include LSTM results for the 200-day horizon or explicitly restrict all superiority claims to horizons where LSTM was actually run.","section":"§5.4, §5.5, §5.6, and Abstract"},{"comment":"Table 2 contains duplicate rows: Config 1 (grid=3, k=6) and Config 2 (grid=3, k=2) have identical train/test RMSE (0.274/0.188), and Configs 5 and 6 (differing only in number of neurons) also have identical RMSEs (0.274/0.152). This suggests misreported or insufficiently described experiments. Additionally, the manuscript does not specify the stock, the time period, the train/test split size, or how the 'Normal/Volatile/Trending' market conditions are defined and labeled. These details are essential to reproduce or interpret any of the tables.","section":"§5.2, Table 2; §3.2"}],"minor_comments":[{"comment":"The term 'non-deterministic' is used loosely for stochastic financial data; consider using 'stochastic' or 'noisy' consistently, since true non-determinism is a stronger mathematical claim.","section":"§1 and §5.6"},{"comment":"Runtime comparisons (35.12s vs ~75s) lack hardware details, number of runs, and standard deviations. The claim of '82.6% of training instances completing under 60 seconds' needs a denominator and a clear description of how it was measured.","section":"§5.3 and Figure 7"},{"comment":"The GitHub repository name 'grand challenges 2024' and the Google Docs supplementary link are not stable or archival; the code and additional results should be deposited in a permanent repository (e.g., Zenodo) with a version/DOI.","section":"§7 Appendix"},{"comment":"Minor typographical issues: 'T rain RMSE' in Table 1, and 'KANs Only' in Table 3's Best Config column. Also, Table 3 does not define '100u-linear' vs '10u-tanh-100'—these abbreviations should be explained in the caption or text.","section":"Tables 1 and 3"},{"comment":"Some references are listed twice ([14] and [15] are the same paper) and some are not consistently formatted. A careful proofreading pass is needed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central quantitative claim is currently unsupported by its own tables, and the model-selection protocol is methodologically inadequate for the stated generalization. I recommend major revision because the issues are fixable by rerunning the experiments with a proper validation protocol and by reconciling Tables 1–3. However, if the authors cannot produce the raw results or code that reproduces Table 3, the editor should consider rejection, as the discrepancy would then be a load-bearing reporting error rather than a fixable methodological gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's central result—LSTM beats vanilla KAN by 6.5-10x in RMSE—does not hold up against the paper's own tables. Table 1 shows the best LSTM test RMSE at 0.0745; Table 3 reports 0.039 for the same 1-day normal market. Table 2 shows the best KAN test RMSE at 0.152; Table 3 reports 0.390. The 10x ratio comes from numbers that appear nowhere in the experimental record. There are also duplicated rows in Table 2 (configs 1/2 and 5/6 have identical train/test RMSE despite different hyperparameters), and the 200-day LSTM rows are N/A while the abstract claims superiority 'across all tested prediction horizons' and the discussion says LSTM faces computational constraints—without showing any experiment.\n\nTo be fair, the paper does some things well. It is explicit that the comparison is only between baseline architectures, and it correctly notes that specialized KAN variants are out of scope. The qualitative direction—LSTM outperforms standard KAN on sequential data—is consistent with prior work the authors cite, so the conclusion isn't surprising. The 2.1x training speed advantage for KAN is a modest practical point, though it's based on runtime averages without much detail.\n\nThe soft spots are load-bearing. No error bars or seed averaging; the 'best config' per horizon is selected using test RMSE, so the reported advantage is partly a test-set fit. The dataset is not described (which stock? what period? how many points?), and the code URL contains a space and is presumably broken. The paper itself concedes in the introduction that standard KAN has shown limited effectiveness for sequential data, citing a critique, so the main finding is not new.\n\nGiven these problems, the paper is a small empirical replication with unreliable numbers. It could become a useful baseline if the authors fix the tables, add validation-based selection and variance reporting, and describe the data. As it stands, I wouldn't cite it, and I wouldn't send it to a serious referee—the internal contradictions would waste everyone's time. For a reading group, it's a decent case study in how benchmark comparisons can go wrong, but not for its positive results.","headline":"The 7-10x LSTM advantage in Table 3 is not backed by the paper's own numbers; the qualitative claim is plausible but the magnitudes are unverified.","tokens_in":10116,"tokens_out":3847,"would_cite":false,"duration_ms":34133,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62M10","91G80"],"pacs":[],"model":"deepseek-v4-flash","headline":"LSTM beats baseline KAN by 6.5–10x on stock-price forecasts, the paper shows.","keywords":["time series forecasting","LSTM","Kolmogorov-Arnold Networks","stock price prediction","RMSE","interpretability","forecast horizon","deep learning"],"falsifier":"Retrain both architectures on the same dataset with hyperparameters chosen on a held-out validation set, average over at least five random seeds, and report test RMSE with confidence intervals. If LSTM's advantage over KAN falls below about 2x or is not statistically significant, the claimed 6.5–10x superiority is an artifact of test-set selection.","tokens_in":9151,"feed_emoji":"📈","tokens_out":4678,"duration_ms":45090,"temperature":0.7,"pith_summary":"This paper tries to establish a performance baseline between two neural architectures on stochastic financial time series: Long Short-Term Memory networks and standard Kolmogorov-Arnold Networks. Using a direct multi-output forecasting protocol with 20-day look-back windows and RMSE in normalized space, the authors report that LSTM achieves 6.5 to 10 times lower error than KAN across 1-day, 2-day, and 100-day horizons and three market conditions. They also find KAN trains about 2.1 times faster and offers theoretical interpretability through the Kolmogorov-Arnold representation theorem, but this does not offset its accuracy gap. The claim matters because it informs architecture selection for accuracy-critical forecasting and motivates specialized temporal KAN variants rather than standard KAN.","feed_headline":"LSTM beats KAN by 6.5–10x on stock forecasts","feed_subtitle":"KAN trains 2x faster and is interpretable, but its RMSE gap makes LSTM the accuracy-first choice.","key_machinery":"The comparison is carried by the two architectures themselves: LSTM, a gated recurrent network that mitigates vanishing gradients and captures long-term dependencies, and KAN, which replaces fixed node activations with learnable univariate spline functions grounded in the Kolmogorov-Arnold representation theorem, where a multivariate function decomposes into sums of univariate functions. The theorem supplies KAN's interpretability rationale but does not model temporal dependencies, which the paper identifies as the root of KAN's sequential-data weakness. The evaluation machinery is a direct multi-output forecasting protocol: 20-day look-back windows, iterative single-step prediction, and RMS","core_discovery":"The central discovery, on the paper's own terms, is that baseline KAN, implemented in its standard form, is not competitive with LSTM for stock price forecasting. On normalized RMSE, LSTM's best configurations achieve 0.039 (1-day normal market) versus KAN's 0.390, a 10-fold advantage; across all horizons and market types the advantage stays between 6.5 and 10 times. KAN's training-time advantage (averaging 35.12 seconds vs about 75 seconds) and interpretability are confirmed, and KAN can produce forecasts at 200-day horizons where the paper reports no LSTM result, but the accuracy gap dominates. The authors conclude that LSTM should be the default for practical financial forecasting, while","pith_inferences":["The reported 6.5–10x advantage may be inflated because the best configuration per horizon was selected directly on the test set, without a validation set or repeated-seed averaging; a fairer protocol could shrink the gap.","The paper's phrasing that KAN can forecast 200 days where LSTM fails likely reflects that LSTM was not trained at that horizon rather than an inherent limitation; a head-to-head at 200 days would settle it.","Since KAN uses a different optimizer (LBFGS) and smaller parameter budgets in the reported runs, its '2.1x faster training' may partly reflect optimizer and capacity differences, not a fundamental architectural advantage.","If the accuracy gap is real, then for non-stationary financial series, KAN's spline activations may be too rigid to track regime changes; a testable extension is to feed KAN with differenced or volatility-adjusted inputs."],"forward_implications":["For accuracy-critical financial forecasting, LSTM remains the evidence-supported default until specialized KAN variants change the picture.","Standard KAN may still fit rapid prototyping and resource-constrained deployments, where its ~2x faster training matters more than accuracy.","The results bound the value of theoretical interpretability: transparency alone does not make an architecture practically useful for sequential prediction.","The gap motivates mathematical research into free-knot polynomial spline approximation, since KAN's optimization difficulty is cited as the main blocker.","Specialized temporal KAN variants (e.g., Time-Frequency KAN) are outside this study's scope but are the natural next targets for evaluation."],"fun_headline_variants":["LSTM beats KAN by up to 10x on stock forecasts","KAN trains 2x faster but LSTM is 10x more accurate","Accuracy gap: LSTM 10x better than KAN on forecasts","LSTM wins over KAN in stock forecasting accuracy","For accurate stock forecasts, LSTM beats KAN by 10x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison's validity depends on the premise that the per-horizon 'Best Config' values in Table 3 are fair representations of each architecture, rather than artifacts of test-set-based model selection.","fun_headline_variants_meta":{"raw":{"variants":["LSTM beats KAN by up to 10x on stock forecasts","KAN trains 2x faster but LSTM is 10x more accurate","Accuracy gap: LSTM 10x better than KAN on forecasts","LSTM wins over KAN in stock forecasting accuracy","For accurate stock forecasts, LSTM beats KAN by 10x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3032,"prompt_tokens":811,"completion_tokens":2221,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2137}},"tokens_in":555,"tokens_out":2221,"duration_ms":16528,"temperature":1.0,"reasoning_tokens":2137,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:39:40.916602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain both architectures on the same dataset with hyperparameters chosen on a held-out validation set, average over at least five random seeds, and report test RMSE with confidence intervals. If LSTM's advantage over KAN falls below about 2x or is not statistically significant, the claimed 6.5–10x superiority is an artifact of test-set selection.","supporting_citations":[],"review_version":1}