{"id":"2fc1cdac-e584-4437-a253-75b415087b52","arxiv_id":"2502.15757","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A dual-attention transformer and a simple MLP both outperform prior limit order book trend prediction models across FI-2010, Tesla/Intel, and Bitcoin datasets, with apparent decline in predictability between 2012 and 2015.","lead":"The paper introduces TLOB, a transformer with separate temporal and spatial attention, plus a simpler MLP variant, for predicting price trends from limit order book data. It reports state-of-the-art F1 scores on the FI-2010 benchmark, on Tesla and Intel stocks, and on Bitcoin, while also arguing that market predictability has declined over time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparisons are not input-matched: TLOB/MLPLOB use 104 extra FI-2010 handcrafted features and order-message augmentation on TSLA/INTC, so the claimed SoTA gains may reflect input differences rather than architecture.","rationale":"The reader's weakest-assumption analysis points to the same load-bearing issue: the baseline comparison is not input-matched. The paper's own Section 6.4 concedes that the extra features and message augmentation improve F1 by roughly 1 and 1.5 points, respectively, and several of the reported advantages over BiNCTABL and DeepLOB are of exactly that magnitude. Because the FI-2010 baselines were taken from another benchmark rather than retrained with the 104 features, and because no evidence shows TSLA/INTC baselines received the order-message augmentation, the central claim of universal SoTA outperformance rests on an unverified input-equality assumption. The released code makes this testable, which is a point in the paper's favor, but the paper as written does not resolve the asymmetry. The historical trend claim is also thin (one trading day per period), but it is secondary to the headline performance claim and does not change the verdict. I did not find grounds to reject the paper outright; the appropriate disposition is the same CONDITIONAL verdict the reader reached, pending the input-matched comparison.","tokens_in":14616,"tokens_out":2849,"duration_ms":29529,"concrete_test":"Ablate the input augmentation rather than the architecture. On FI-2010, retrain and test TLOB and MLPLOB using only the raw LOB snapshot features (without the 104 handcrafted features) and compare against the Prata et al. [31] baseline numbers, or equivalently retrain BiNCTABL/DeepLOB with the 104 features added. On TSLA/INTC, retrain TLOB and MLPLOB without concatenating order messages, and/or retrain DeepLOB and BiNCTABL with the same order-message augmentation. If either model no longer exceeds all baselines on all four horizons under input-matched conditions, the state-of-the-art claim as stated is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TLOB outperforms SoTA on every dataset and horizon depends on comparisons in which the proposed models receive more input information than the baselines. Section 6.4 states: for FI-2010, the authors 'used [the 104 handcrafted features] in both our models,' reporting an F1 improvement of approximately 1; for Tesla and Intel, they augmented LOB snapshots by concatenating order messages, reporting an improvement of approximately 1.5. The FI-2010 baseline numbers in Table 3 are not recomputed with those features: Section 7.1 says they are 'extracted from the benchmark of Prata et al. [31]'. DeepLOB and BiNCTABL on TSLA/INTC were trained locally, but the paper gives no indication that these baselines received the order-message augmentation. The reported winning margins are small in several cells: TLOB vs BiNCTABL on FI-2010 is +0.45, +11.18, +2.33, +0.71 F1 for h=10,20,50,100, and TSLA h=10 is +1.81 over BiNCTABL. If 1.0–1.5 points of the gain come from extra inputs, the SoTA claim fails on FI-2010 h=10 and h=100 and on TSLA h=10. This is an external-validity risk, not an internal inconsistency, but it is load-bearing for the paper's headline contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two deep learning models for price trend prediction from limit order book (LOB) data: MLPLOB, an MLP with separate feature-mixing and temporal-mixing layers, and TLOB, a transformer with dual temporal and spatial attention, a bilinear normalization layer, and an MLP-based feed-forward block. The models are evaluated on FI-2010, a NASDAQ dataset (TSLA and INTC), and a Bitcoin dataset for horizons 10, 20, 50, and 100, using F1-score. The authors claim that TLOB outperforms state-of-the-art baselines on every dataset and horizon, and that MLPLOB also surpasses the baselines except for the longest horizons on some datasets. Additional contributions include a labeling method that decouples the smoothing window from the prediction horizon, an empirical claim that stock price predictability declined by 6.68 F1-score between 2012 and 2015 for Intel, and an exploration of setting the classification threshold to the average spread. The paper also promises to release code.","tokens_in":14929,"tokens_out":7862,"duration_ms":69818,"significance":"If the reported comparisons are correct, the paper would be a useful contribution to the LOB forecasting literature: it would demonstrate that a well-tuned MLP can compete with more complex architectures, and that dual attention provides gains on longer horizons. The code release and the ablation study are positive elements. However, the central state-of-the-art claim hinges on a baseline comparison that may not be input-matched, because the proposed models use 104 additional handcrafted features on FI-2010 and order-message enrichment on TSLA/INTC, while the baselines appear not to have received the same inputs. The margins in several key cells are smaller than the stated effect of these extra inputs, so the significance of the architectural contribution is not yet established beyond reasonable doubt.","major_comments":[{"comment":"The FI-2010 comparison is not input-matched. Section 6.4 says that 'the FI-2010 dataset also contains 104 handcrafted features derived from the LOB, we used them in both our models' and that this choice improved F1-score by approximately 1. Section 7.1 says the baseline numbers in Table 3 are 'extracted from the benchmark of Prata et al. [31] since the settings are equal for the FI-2010 dataset.' These two statements are in tension: if the settings were truly equal, the baselines would also have used the 104 features, but then the reported gain of approximately 1 from using them would be ill-defined. In any case, the reader cannot verify that the baselines received the same 144-dimensional input. The margins of TLOB over BiNCTABL at h=10 (+0.45) and h=100 (+0.71) are smaller than the stated feature-induced gain, so the SoTA claim on FI-2010 is not established. Please re-run the baselines on the identical input representation, or provide explicit evidence from [31] that the baselines used the same features.","section":"Section 6.4, Section 7.1, Table 3"},{"comment":"For TSLA and INTC, the paper states in Section 6.4 that the authors 'augmented the LOB snapshots by concatenating them with the corresponding orders' for their own models, yielding 'an approximate improvement of 1.5 in the F1-score.' DeepLOB and BiNCTABL were trained locally on these datasets, but the paper never specifies whether these baselines received the same order-message enrichment. The TSLA h=10 improvement of TLOB over BiNCTABL is +1.81, which is comparable to the stated augmentation gain, so the comparison may reflect the additional input information rather than the model architecture. The authors must specify the exact feature representation used by each baseline and, ideally, re-run the baselines with the same enrichment before claiming state-of-the-art performance on TSLA/INTC.","section":"Section 6.4, Tables 4-5"},{"comment":"The abstract's claim that 'stock price predictability has declined over time, -6.68 in F1-score' is supported only by Table 7, which compares a single day of INTC from 2012 with a single day in 2015, using only TLOB, with no variance estimates or statistical test. A one-day comparison is far too fragile to support a general claim about market efficiency. Please expand the experiment to multiple days and models, or considerably soften the claim and move it out of the abstract.","section":"Section 7.4, Table 7"},{"comment":"The footnote says 'if we had taken the results reported in the individual papers, MLPLOB and TLOB would have still outperformed all the other models,' but this is not demonstrated with numbers. Because the original papers differ in data splits, feature sets, and evaluation protocols, this assertion does not substitute for a re-computed comparison. Please either provide the corresponding table or remove the claim.","section":"Section 7.1, footnote 8"}],"minor_comments":[{"comment":"The BTC dataset is described as sampled at 250 ms in Section 6.2, but footnote 4 says it is sampled every 100 ms; please reconcile these values.","section":"Section 6.2, footnote 4"},{"comment":"The DLA result at h=100 (52.2) is much lower than at the other horizons; please verify this value and add a note if it is due to a failed run.","section":"Table 3"},{"comment":"The precision-recall curves are referenced as 'Fig. ??', indicating a missing cross-reference; please fix the figure numbering.","section":"Appendix B"},{"comment":"The assertion that TLOB and MLPLOB converge in 'less than half the epochs required by BiNCTABL and DeepLOB' is not supported by any figure or training curve; please add evidence or soften the statement.","section":"Section 7.2"},{"comment":"The MLPLOB learning rate is listed as 0.003, which is surprising given the TLOB value of 0.0001; please check whether this is a typo.","section":"Table 10"},{"comment":"The abstract states that the code is released at a GitHub URL, while the last paragraph of the Introduction says 'We commit to releasing the code publicly'; please unify these statements to avoid ambiguity about availability.","section":"Abstract and Introduction"},{"comment":"The sentence 'TLOB and MLPLOB have a higher number of parameters compared to SoTA LOB-based models' is confusing because the next sentence says they are still smaller than standard deep learning models; consider rewording for clarity.","section":"Section 6.4"}],"recommendation":"major_revision","confidential_remarks":"The baseline numbers for FI-2010 are taken from Prata et al. [31], a benchmark paper co-authored by the first author of this manuscript. This creates a conflict-of-interest concern: the authors are relying on numbers from their own previous work without independent reproduction. I recommend that the editor require the re-computation described in Major Comment 1 rather than accepting the extracted numbers at face value. I also suggest checking the novelty of the two new models relative to the authors' prior benchmark study, since some architectural components (e.g., bilinear normalization) are already present in [31, 37]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is not a lazy paper: three datasets spanning FI-2010, NASDAQ, and Bitcoin, four horizons, code released, an ablation, and a frank limitations section. The MLPLOB result, in particular, is a legitimate and interesting data point—a simple MLP-Mixer-style stack matching and in places beating BiNCTABL, the strongest prior model. Second, the central claim that TLOB/MLPLOB \"outperform SoTA in every dataset and horizon\" is not currently supported, because the proposed models received extra input information that the baselines likely did not.\n\nThe stress-test note is right. Section 6.4 says the authors added 104 handcrafted FI-2010 features and got ~1 F1 point; they also concatenated order messages for TSLA/INTC and got ~1.5 points. The FI-2010 baseline numbers are taken from the Prata et al. benchmark, not re-run with those features. On TSLA/INTC, DeepLOB and BiNCTABL were re-trained, but the paper never states whether they saw the same augmented inputs. Look at the margins: against BiNCTABL on FI-2010, TLOB's lead is +0.45 at h=10 and +0.71 at h=100; MLPLOB's is +0.54 and +0.52. If the feature bump is ~1 point, those cells flip. So the SoTA claim is fragile exactly on the benchmark that matters most.\n\nThe historical predictability section is the other soft spot. Table 7 compares one day of Intel from 2012 against one day from 2015, only with TLOB, only at h=50, and the abstract headlines the -6.68 F1. That is a single pair of days, with no variance, no other stocks, no other models. It is suggestive at best, and the paper's causal interpretation about market efficiency is well beyond what the data can support.\n\nWhat the paper does well: the cross-market comparison is genuinely broader than most work in this subfield; the spread-based threshold experiment is a good idea even though it only worked on Tesla; and the authors are unusually honest that the model is not production-ready. The writing is clear, and the code release matters.\n\nBottom line: this deserves a serious peer review, but the referee must require the authors to either re-run all baselines with the same input features or soften the SoTA claim accordingly. As it stands, I would not cite it for the performance numbers, but I would bring it to a reading group to discuss the input-matching pitfall in LOB benchmarking.","headline":"A useful empirical breadth and an honest write-up, but the SoTA claim is currently riding on input-feature differences that the baselines did not get.","tokens_in":15461,"tokens_out":2393,"would_cite":false,"duration_ms":26758,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-attention transformer for limit order book data outperforms state-of-the-art price-trend models on every dataset and horizon tested, with average F1 gains of 3.7 on FI-2010, 1.3 on Tesla, 7.7 on Intel, and 1.1 on Bitcoin.","keywords":["price trend prediction","limit order book","dual attention","transformer","F1-score","market efficiency","labeling method","mid-price prediction"],"falsifier":"Retrain DeepLOB and BiNCTABL using exactly the augmented inputs TLOB and MLPLOB receive (104 handcrafted features on FI-2010, concatenated order messages on TSLA-INTC) and compare F1 on all horizons; the central SoTA claim would be weakened if the gaps fall below the reported 3.7, 1.3, 7.7, and 1.1 points.","tokens_in":14398,"feed_emoji":"📈","tokens_out":6954,"duration_ms":68959,"temperature":0.7,"pith_summary":"This paper argues that a transformer that applies attention separately to time and to order-book features—TLOB—outperforms existing price-trend models on every dataset and horizon tested, with average F1 gains of 3.7 on the FI-2010 benchmark, 1.3 on Tesla, 7.7 on Intel, and 1.1 on Bitcoin. It also shows that a deliberately simple MLP architecture reaches similar or better results on short horizons, suggesting that complex architectures are not strictly necessary for this task. The paper introduces a labeling scheme that decouples the smoothing window from the prediction horizon, removing a bias it identifies in earlier labels, and reports that stock predictability declined between 2012 and 2015 for Intel, consistent with growing market efficiency. When the trend threshold is set to the average bid-ask spread, reported F1 scores deteriorate, which the paper reads as evidence that classification accuracy does not automatically translate into trading profitability.","feed_headline":"Dual-attention model beats all price-trend baselines in every test","feed_subtitle":"A transformer that reads order-book snapshots two ways also shows market predictability has slipped since 2012.","key_machinery":"The machinery is dual self-attention inside each TLOB block: temporal attention computes attention across the sequence of LOB snapshots, spatial attention computes attention across the price-volume feature dimension, and an MLPLOB block (two fully connected layers with GeLU activation, applied once across features and once across time) stands in for the transformer feed-forward network. A bilinear normalization layer adapts to batch statistics to handle non-stationarity, sinusoidal positional embeddings preserve snapshot order, and the proposed labeling rule uses separate window length $k$ and horizon $h$ parameters to define the trend from smoothed future and past mid-prices, removing the horizon bias of earlier labels.","core_discovery":"On its own terms, the paper's central discovery is that a dual-attention transformer—one attention pass over LOB snapshots for temporal dependencies and a second over LOB features for spatial dependencies, followed by an MLP mixing block and preceded by bilinear normalization—produces the highest F1 scores among the compared methods on all four datasets (FI-2010, Tesla, Intel, Bitcoin) across all four horizons. The same study finds that a plain MLP with feature-mixing and temporal-mixing layers often matches or beats more elaborate baselines, and that the gap between the two proposed models is small on short horizons but grows on long horizons in favor of TLOB. The paper additionally claims that predictability is not static: on Intel data, F1 falls by 6.68 points from 2012 to 2015, and that replacing the class-balancing threshold with a spread-based threshold lowers performance, highlighting the difference between predictive accuracy and practical profitability.","pith_inferences":["A controlled test that feeds the same handcrafted features and order messages to the baseline models would isolate how much of the reported gain comes from the architecture versus the extra input information; the paper does not run that test.","The new labeling method's benefit could be quantified by applying it to an existing baseline such as DeepLOB while keeping the architecture fixed.","The predictability-decline result rests on one stock at two dates; repeating it over many stocks and years would show whether the trend is a market-wide effect."],"forward_implications":["Short-horizon price-trend prediction can be handled by a simple MLP; the transformer's advantage is concentrated at horizons 50 and 100.","The decoupled smoothing window and horizon in the new labeling method can be applied to any model, so existing baselines may improve without architectural changes.","F1 scores on older data overstate current predictability; the measured 6.68-point drop on Intel between 2012 and 2015 means recent data is needed for realistic evaluation.","Replacing the class-balancing threshold with the average spread lowers F1, so classification accuracy on balanced labels is not the same as profitability after transaction costs."],"supporting_citations":[{"why":"Supplies the FI-2010 LOB dataset and its original trend labels used in the benchmark experiments.","marker":"[29]"},{"why":"Provides the state-of-the-art baseline results and hyperparameters that TLOB and MLPLOB are compared against.","marker":"[31]"},{"why":"Defines DeepLOB, one of the two main deep-learning baselines retrained on Tesla, Intel, and Bitcoin.","marker":"[45]"},{"why":"Defines BiNCTABL, the other main baseline, and the bilinear normalization layer reused in TLOB.","marker":"[37]"},{"why":"Supplies the transformer architecture and sinusoidal positional encoding that TLOB adapts.","marker":"[41]"},{"why":"MLP-mixer, the design source for MLPLOB's feature- and temporal-mixing layers.","marker":"[35]"}],"fun_headline_variants":["Transformer with dual attention tops all LOB trend baselines","Simple MLP rivals fancy transformer on order-book trends","Order-book transformer beats baselines but costs eat profits","Market predictability slips 6.7 F1 points, new model takes lead","Dual attention net wins price trend tests on 4 datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the baseline models were given the same input information as TLOB and MLPLOB, but the paper adds 104 handcrafted FI-2010 features and NASDAQ order messages only to its own models.","fun_headline_variants_meta":{"raw":{"variants":["Transformer with dual attention tops all LOB trend baselines","Simple MLP rivals fancy transformer on order-book trends","Order-book transformer beats baselines but costs eat profits","Market predictability slips 6.7 F1 points, new model takes lead","Dual attention net wins price trend tests on 4 datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2648,"prompt_tokens":1028,"completion_tokens":1620,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1536}},"tokens_in":644,"tokens_out":1620,"duration_ms":12267,"temperature":1.0,"reasoning_tokens":1536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T05:25:01.958161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain DeepLOB and BiNCTABL using exactly the augmented inputs TLOB and MLPLOB receive (104 handcrafted features on FI-2010, concatenated order messages on TSLA-INTC) and compare F1 on all horizons; the central SoTA claim would be weakened if the gaps fall below the reported 3.7, 1.3, 7.7, and 1.1 points.","supporting_citations":[{"cited_title":"Ntakaris, M","cited_arxiv_id":null,"evidence_quote":"Supplies the FI-2010 LOB dataset and its original trend labels used in the benchmark experiments."},{"cited_title":"Prata, G","cited_arxiv_id":null,"evidence_quote":"Provides the state-of-the-art baseline results and hyperparameters that TLOB and MLPLOB are compared against."},{"cited_title":"Zhang, S","cited_arxiv_id":null,"evidence_quote":"Defines DeepLOB, one of the two main deep-learning baselines retrained on Tesla, Intel, and Bitcoin."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BiNCTABL, the other main baseline, and the bilinear normalization layer reused in TLOB."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MLP-mixer, the design source for MLPLOB's feature- and temporal-mixing layers."}],"review_version":1}