{"id":"41de7c54-addc-4ff6-84ed-0923bafc725e","arxiv_id":"2501.01010","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"CryptoMamba applies a Mamba state space model to Bitcoin price prediction, reporting better test-set RMSE and trading returns than baselines, but the generalization claim is undermined by validation results and missing error bars.","lead":"CryptoMamba is a Mamba-based neural network for predicting next-day Bitcoin prices, tested against LSTM, GRU, and transformer baselines. The authors claim better accuracy and efficiency, but their own validation results do not support the broad generalization claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own validation results contradict its core generalization claim: in Table 4 validation, CryptoMamba-v is beaten by LSTM-v, Bi-LSTM-v, and GRU-v across all three trading strategies, so 'consistent outperformance across market conditions' rests on a single test period with no error bars…","rationale":"The paper's central claim is superior predictive accuracy plus generalization across market conditions (abstract, Sections 5.2 and 6). For generalization, the validation period is the only second market condition; the test period is one year. Table 4 shows that on validation, CryptoMamba-v's final balances are below LSTM-v, Bi-LSTM-v, and GRU-v in Vanilla, Smart, and Extended Smart strategies. The text in Section 6 nonetheless asserts consistent outperformance 'in both steady and volatile intervals' and 'in both the validation and test periods'. This is an internal contradiction, not merely a missing robustness check. If validation counts, the claim is false. If validation does not count, then the regression claims in Table 3 have no error bars, no multiple seeds, and only a single test year; the RMSE gaps to S-Mamba are 4.3 (0.25%) without volume and 53.5 (3.2%) with volume, so the differences could reflect noise. Finally, the trading experiment lacks a buy-and-hold baseline and transaction costs, so 'highest returns in real-world trading' is not anchored to a meaningful benchmark. The reader's concern about a single split is valid, but the more damaging issue is that the paper's own validation results directly undermine the generalization claim, making the strongest critique sharper than the reader's weakest-assumption formulation.","tokens_in":11113,"tokens_out":5658,"duration_ms":53220,"concrete_test":"Re-run the complete pipeline with 10 random seeds and report mean ± std for the validation and test trading final balances as well as the Table 3 regression metrics; also compute the buy-and-hold final balance over the same periods. If CryptoMamba-v's validation balances remain below LSTM-v/Bi-LSTM-v/GRU-v, or if its test RMSE advantage over S-Mamba-v is less than one standard deviation, or if buy-and-hold beats its test final balance, then the 'consistently outperforms' and 'highest returns' claims are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 and the abstract claim that CryptoMamba 'consistently' outperforms baselines and generalizes across market conditions are contradicted by the paper's own Table 4. On the validation period, CryptoMamba-v's best final balance ($124.09 in Vanilla) is below LSTM-v ($136.93), Bi-LSTM-v ($156.98), and GRU-v ($153.57); under Smart and Extended Smart, several baselines also beat it. The validation period is not an aside: it is the paper's only second market condition, since the test period is a single year. The text nonetheless states that CryptoMamba 'performs consistently well... in both the validation and test periods' and 'outperforming state-of-the-art baselines in both validation and test periods.' That is an internal contradiction. If validation counts, the claimed consistency is false; if validation is treated as merely illustrative, then the test-period regression claims in Table 3 have no error bars, no multiple seeds, and tiny RMSE gaps to S-Mamba (CryptoMamba 1713.0 vs S-Mamba 1717.4; CryptoMamba-v 1598.1 vs S-Mamba-v 1651.6). Additionally, the trading evaluation lacks a buy-and-hold baseline and does not deduct transaction costs despite the stated threshold rationale. The central claim of consistent superiority and enhanced generalization is therefore not supported by the paper's own data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CryptoMamba, a Mamba-based state space model for daily Bitcoin price prediction, and evaluates it against LSTM, Bi-LSTM, GRU, iTransformer, and S-Mamba on a September 2018--September 2024 Yahoo Finance dataset. The authors report test-set regression metrics (RMSE, MAPE, MAE), parameter counts, and training/inference efficiency, and they run three trading strategies (Vanilla, Smart, Extended Smart) on the validation and test years to translate predictions into financial outcomes. The central claims are that CryptoMamba consistently outperforms all baselines across all evaluation metrics and generalizes across different market conditions, with the volume-inclusive variant achieving the best test RMSE of 1598.1 and the highest test-period trading balances.","tokens_in":11437,"tokens_out":4192,"duration_ms":39153,"significance":"If the claims were fully supported, this would be a useful empirical contribution: a compact SSM (136k parameters) beating recurrent and transformer baselines on a recent Bitcoin dataset would strengthen the case for Mamba-style models in financial time series. The paper provides a public codebase, a clearly defined train/validation/test split, three transparent trading protocols, and efficiency measurements, all of which are positive features. However, the evidence as presented does not support the stated generalization and consistency claims: the validation results in Table 4 contradict the text, the test-set regression gaps are small and come from a single split with no uncertainty quantification, and the trading evaluation lacks a buy-and-hold baseline and transaction costs. These are load-bearing deficiencies for the paper's central message.","major_comments":[{"comment":"The claim that CryptoMamba \"outperforming state-of-the-art baselines in both the validation and test periods\" is directly contradicted by Table 4. In the validation period, CryptoMamba-v's Vanilla final balance is $124.09, below LSTM-v ($136.93), Bi-LSTM-v ($156.98), and GRU-v ($153.57); under the Smart strategy, LSTM-v reaches $149.95 and Bi-LSTM $141.36 versus CryptoMamba-v's $127.12. The text must either be corrected to reflect that validation results do not support consistent superiority, or if validation is discounted as illustrative, the paper must justify why only the single test year supports the generalization claim. As written, this is an internal contradiction that directly undermines the abstract and conclusion.","section":"Section 6, Table 4"},{"comment":"The test-period superiority claim rests on a single train/validation/test split with a fixed random seed, and no confidence intervals, multiple seeds, or statistical significance tests are reported. The RMSE gap between CryptoMamba and S-Mamba is only 4.4 (1713.0 vs 1717.4, 0.26%), and between CryptoMamba-v and S-Mamba-v is 53.5 (1598.1 vs 1651.6, 3.2%), so the reported ordering could easily be within run-to-run noise. Without repeated runs or a significance test, the statement that CryptoMamba \"consistently outperforms all baseline models\" is not supported by the evidence in Table 3.","section":"Section 5.2, Table 3"},{"comment":"The trading evaluation does not include a buy-and-hold baseline, and transaction costs are never deducted even though the Vanilla threshold is explicitly justified as accounting for transaction fees (\"This safeguard is introduced to account for transaction fees\"). Since the trading results are the basis for the claim of \"real-world\" utility, the absence of a buy-and-hold comparison and of fee modeling means the final balances cannot be interpreted as economic value added. For example, in the test Vanilla setup iTransformer-v reaches $241.50 versus CryptoMamba-v's $246.58; with realistic fees the ranking could change.","section":"Section 6, Algorithms 1--3"},{"comment":"The hyperparameter comparison is not shown to be fair. CryptoMamba uses hand-specified hyperparameters (3 C-Blocks, 4 CMBlocks, sequence lengths 14/16/32, d_state 64), while the baselines are configured with values described as following \"best practices,\" with no evidence that comparable per-model tuning was performed on the validation set. Since the paper's efficiency and accuracy claims depend on this comparison, the authors should report a hyperparameter search budget or sensitivity analysis for both CryptoMamba and the baselines.","section":"Section 5.1, Setup"}],"minor_comments":[{"comment":"The phrase \"Our findings signal a huge advantage for SSMs\" is an overstatement relative to the single-dataset, single-split evidence; please temper this to match the actual statistical support.","section":"Abstract"},{"comment":"The table entry \"V olume\" contains a typo and should read \"Volume.\"","section":"Section 4.1, Table 2"},{"comment":"The data preprocessing is not described: it is unclear whether features were normalized, detrended, or transformed, and how the 14-day input context was handled at the boundary of each split. This information is needed for reproducibility.","section":"Section 5.1"},{"comment":"Figure 1 is very schematic; a formal description of the tensor shapes through the C-Blocks and the Merge block would clarify the architecture and support the parameter count of 136k.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe genuinely new thing here is applying Mamba/SSM to Bitcoin daily price prediction with a compact hierarchical block design. The model has 136k parameters, trains in under half an hour on a consumer GPU, and on the test year its volume-inclusive variant edges out S-Mamba (RMSE 1598 vs 1652). The code is public, and the paper compares against a reasonable set of baselines. That's real effort, and the efficiency numbers are believable.\n\nThe problem is the central claim. The abstract and Section 5.2 say CryptoMamba 'consistently outperforms' all baselines and generalizes across market conditions. But Table 4, the paper's own validation results, shows CryptoMamba-v finishing at $124.09 in the Vanilla strategy, behind LSTM-v ($136.93), Bi-LSTM-v ($156.98), and GRU-v ($153.57). Similar shortfalls appear in the Smart and Extended Smart strategies. The text explicitly mentions 'both the validation and test periods' as evidence, which the table contradicts. That's not a minor wording issue; it undercuts the central generalization story.\n\nBeyond that, the test-set improvements over S-Mamba are small (1713 vs 1717 without volume), and there are no error bars or multiple seeds, so we don't know if that gap is noise. The trading simulation has no buy-and-hold baseline and does not actually deduct transaction costs, despite the threshold being justified as a safeguard for fees. Hyperparameters like sequence lengths, d_state, risk percentage, and max-short are hand-picked, with no sensitivity analysis.\n\nIn short, the architecture is a reasonable incremental application of Mamba to a specific asset, and the paper is honest about its dataset and code. But as written, the headline claim is not supported by its own evidence. It needs a major revision: correct the validation overclaim, add uncertainty quantification, include a buy-and-hold and transaction-cost baseline, and ideally test on more than one year.\n\nWould I bring it to reading group? Maybe, as a case study in how financial ML papers overclaim. Would I cite it? Not in the next year. But it deserves a serious referee: the topic is relevant and the code is reusable. A good referee would catch exactly this contradiction and force the authors to fix it.\n\nRecommendation: send to peer review, but expect heavy revision.\n\nThat's my take.","headline":"A compact Mamba variant for Bitcoin forecasting with real code and a believable test-set edge, but the paper's own validation table contradicts its central 'consistent outperformance' claim.","tokens_in":11984,"tokens_out":2948,"would_cite":false,"duration_ms":26392,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CryptoMamba, a Mamba-based state space model, predicts Bitcoin prices more accurately than LSTM, Bi-LSTM, GRU, iTransformer, and S-Mamba baselines.","keywords":["Bitcoin price prediction","state space models","Mamba","time series forecasting","cryptocurrency","trading simulation","deep learning","long-range dependencies"],"falsifier":"A reader could check this by running the same models on a different out-of-sample year, such as the 2022 bear market, and seeing whether CryptoMamba's RMSE remains below S-Mamba-v and iTransformer-v; if the gap narrows or reverses, the claim of consistent outperformance would fail.","tokens_in":10928,"feed_emoji":"📈","tokens_out":6619,"duration_ms":48012,"temperature":0.7,"pith_summary":"Predicting Bitcoin's next-day price is hard because the market is volatile and non-stationary. The paper claims that CryptoMamba, a Mamba-based state space model, handles this better than existing recurrent, transformer, and state space baselines by capturing long-range dependencies and adapting to regime shifts. On daily Bitcoin data from 2018 to 2024, it reports the lowest RMSE, MAPE, and MAE among all tested models, with the volume-inclusive variant reaching an RMSE of 1598.1. The paper also shows that feeding these predictions into simple trading algorithms yields the highest simulated final balances, suggesting that more accurate forecasts can translate into real financial utility.","feed_headline":"Beats LSTM, GRU, iTransformer, and S-Mamba on Bitcoin forecasting","feed_subtitle":"A 136k-parameter state space model beats much larger recurrent and attention baselines on daily Bitcoin data.","key_machinery":"The load-bearing mechanism is the C-Block, a hierarchical building block made of several CMBlocks—each a normalization layer followed by a Mamba selective state space block—and a final MLP linear layer that resizes the sequence. Mamba's input-dependent dynamics make the system time-varying, so the model can adjust its state transitions to the data, which the paper argues is key for adapting to volatility and regime shifts. Stacking multiple C-Blocks and merging their outputs with a linear layer lets the model refine features at different scales, capturing both short- and long-range dependencies. The paper also treats trading volume as an input feature and shows it improves accuracy for most models.","core_discovery":"The central claim is that a custom Mamba-based architecture tailored to financial time series consistently outperforms established baselines in Bitcoin price prediction. The authors design CryptoMamba as a stack of C-Blocks, each containing several CMBlocks (normalization plus a Mamba selective state space layer) and a linear projection, followed by a Merge block that produces the next-day close price. They report that the volume-inclusive model achieves an RMSE of 1598.1, a MAPE of 2.034, and an MAE of 1120.7 on a one-year held-out test period, beating LSTM, Bi-LSTM, GRU, iTransformer, and S-Mamba on every metric, and that even without volume input CryptoMamba outperforms the volume-inclusive baselines. In simulated trading with three strategies, CryptoMamba-v ends with the highest final balance ($246.58 Vanilla, $213.20 Smart, $262.78 Extended Smart) while keeping lower maximum drawdowns than most baselines.","pith_inferences":["The paper tests only a single validation year and a single test year; a natural extension is to evaluate CryptoMamba across multiple non-overlapping test periods such as bull and bear regimes to see whether the performance gap persists.","Because the baselines are configured with their commonly used hyperparameters rather than a systematic per-model search, part of the reported gap may reflect tuning effort; an independent fair-tuned comparison would clarify this.","The three trading algorithms are simple heuristics; combining CryptoMamba's predictions with position sizing, transaction-cost models, or portfolio-level risk management could change the financial outcomes, a question the paper leaves open.","The compact size of CryptoMamba suggests it could be deployed in edge devices for real-time trading, but latency and memory under streaming data are not evaluated."],"forward_implications":["If the reported results hold, Mamba-based state space models become a strong candidate architecture for cryptocurrency and other financial time-series forecasting, not just for natural language processing and vision.","The finding that volume improves accuracy across most architectures suggests trading volume should be a standard input feature in Bitcoin price prediction tasks.","The model's small parameter count (136k) and fast training and inference imply that high-accuracy forecasting does not require large transformer-scale models, which matters for real-time or resource-constrained trading systems.","The trading simulations show that forecast accuracy can be converted into higher final balances and lower drawdowns under simple rule-based strategies, supporting the use of such models in automated trading."],"supporting_citations":[{"why":"Defines Mamba, the selective state space block that CryptoMamba's CMBlocks are built on.","marker":"[12]"},{"why":"Introduces S-Mamba, the advanced time-series baseline that CryptoMamba is compared against and improves upon.","marker":"[27]"},{"why":"Supplies the LSTM, Bi-LSTM, and GRU baselines and their hyperparameter configurations.","marker":"[23]"},{"why":"Provides the iTransformer baseline, representing attention-based forecasting.","marker":"[24]"},{"why":"Motivates SSMs as a framework for capturing long-range dependencies, the core rationale for the architecture.","marker":"[11]"}],"fun_headline_variants":["State space model beats LSTM, GRU, iTransformer on Bitcoin","136k-parameter SSM beats much larger models on Bitcoin","CryptoMamba: Small SSM tops Bitcoin prediction baselines","Mamba SSM wins Bitcoin forecast with higher trading balance","CryptoMamba SSM beats baselines and boosts trading gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim of consistent superiority rests on the assumption that the single one-year test period is representative of Bitcoin's various market conditions, and that the hand-chosen hyperparameters for CryptoMamba are fairly matched against the baselines.","fun_headline_variants_meta":{"raw":{"variants":["State space model beats LSTM, GRU, iTransformer on Bitcoin","136k-parameter SSM beats much larger models on Bitcoin","CryptoMamba: Small SSM tops Bitcoin prediction baselines","Mamba SSM wins Bitcoin forecast with higher trading balance","CryptoMamba SSM beats baselines and boosts trading gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3929,"prompt_tokens":907,"completion_tokens":3022,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2933}},"tokens_in":523,"tokens_out":3022,"duration_ms":18304,"temperature":1.0,"reasoning_tokens":2933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:36:36.357992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could check this by running the same models on a different out-of-sample year, such as the 2022 bear market, and seeing whether CryptoMamba's RMSE remains below S-Mamba-v and iTransformer-v; if the gap narrows or reverses, the claim of consistent outperformance would fail.","supporting_citations":[{"cited_title":"Forecasting cryptocurrency prices using lstm, gru, and bi-directional lstm: a deep learning approach.Fractal and Fractional, 7(2):203, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM, Bi-LSTM, and GRU baselines and their hyperparameter configurations."},{"cited_title":"itransformer: Inverted transformers are effective for time series forecasting","cited_arxiv_id":null,"evidence_quote":"Provides the iTransformer baseline, representing attention-based forecasting."}],"review_version":1}