Research on Optimizing Real-Time Data Processing in High-Frequency Trading Algorithms using Machine Learning

Lei Fu; Liyang Wang; Yu Cheng; Yuxiang Wang; Yuxin Fan; Zhuohuan Hu

REVIEW 5 major objections 6 minor 10 references

Research on Optimizing Real-Time Data Processing in High-Frequency Trading Algorithms using Machine Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pruned convolutional network with dynamic feature selection keeps high-frequency trading forecasts accurate at 10 ms latency.

desk verdict A well-intentioned engineering combo that is too garbled to evaluate; the pruning equation is inverted and the experiments lack any reproducible protocol. read the letter →

arxiv 2412.01062 v1 pith:4KHT4YDN submitted 2024-12-02 cs.LG q-fin.CP

Yuxin Fan , Zhuohuan Hu , Lei Fu , Yu Cheng , Liyang Wang , Yuxiang Wang This is my paper · ORCID

classification cs.LGq-fin.CP

keywords high-frequencytradingreal-timedataprocessingdynamicfeatureselectionlightweightneuralnetworkconvolutionalmodelpruninglatencymachinelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a machine-learning pipeline can satisfy the accuracy and latency demands of high-frequency trading at the same time. It combines a dynamic feature-selection stage, which re-picks input features as market conditions shift, with a deliberately small convolutional network whose weights are pruned to keep inference cheap. The empirical claim is that this pipeline maintains consistent prediction error and explanatory power across changing data windows and feature-selection thresholds, while running at 35 ms average execution time and 10 ms average latency, beating five baseline methods on both accuracy and speed. If correct, the contribution is a practical architecture for adaptive, low-latency forecasting in a setting where a few extra milliseconds can change the profitability of a trade.

What carries the argument

The central machinery has two parts. The first is the dynamic feature selector: it clusters data vectors (improved $k$-means++), assigns each feature a weight equal to the reciprocal of its probability-weighted within-cluster variance, and keeps the features whose mutual information with the prediction target is large, with the joint density estimated online by kernel density estimation. That component is what is supposed to track regime changes and revise the input set in real time. The second is the lightweight predictor: a modular convolutional network with small kernels ($3\times3$ and $5\times5$) whose multi-scale outputs are weighted and summed, plus a pruning step meant to remove parameters to cut computation while a Lagrangian term keeps training error low. Together these components carry the paper's claim that accuracy and speed can be achieved without a full-size deep model.

What would settle it

On the same 50-ETF data, replace the Eq. 3 reciprocal-variance weights with weights estimated from mutual information alone and re-run the reported comparisons; if RMSE and $R^2$ do not worsen, the reciprocal-variance assumption is not what carries the accuracy, and if they do worsen, that assumption is confirmed as load-bearing.

Watch

Extended reading notes

Core claim

The paper argues that a complete high-frequency trading forecasting pipeline can be both adaptive and fast when feature selection and model size are co-designed for speed. The dynamic feature selector clusters incoming market data with an improved $k$-means++ rule, scores each feature by the reciprocal of the probability-weighted within-cluster variance, and retains features with large mutual information with the target, estimated online by kernel density estimation. The retained features feed a lightweight network built from small convolutional modules with $3\times3$ and $5\times5$ kernels whose outputs are weighted and summed, and pruning keeps the parameter count down. On 50 ETF datasets the method is reported to outperform five comparison methods (DL-AN, VE-FTS, T-TS, DF-QA, RL-EMF) on RMSE and $R^2$ across window sizes and mutual-information thresholds, with the lowest average execution time of 35 ms and latency of 10 ms.

Load-bearing premise

The load-bearing premise is that features with low variance inside each cluster are the informative ones, because Eq. 3 defines feature importance as the reciprocal of that weighted variance; the paper offers no independent justification for this weighting, so if it is wrong the dynamic selector can discard precisely the signals the network needs.

Editorial extensions

If this is right

HFT systems that need to update their feature sets online could run this pipeline inside the decision loop without waiting for a full retraining cycle.
Prediction quality that stays stable when data windows or feature-selection thresholds change would make the system less sensitive to hyperparameter tuning than the compared baselines.
The 35 ms execution / 10 ms latency figures provide a concrete budget: any comparable adaptive feature-selection method must meet this speed to be usable in the same setting.
The small convolutional kernels and pruning imply the model can run on commodity hardware, reducing the infrastructure cost of low-latency forecasting.
If the consistency across market conditions is real, the approach could reduce the frequency of strategy failures during volatile periods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reciprocal-variance weighting in Eq. 3 is an assumption, not a derived result; a direct comparison against mutual-information-only feature ranking would show whether the clustering step is actually responsible for the accuracy gain.
The same recipe of dynamic feature selection plus a pruned small network could be transferred to other latency-sensitive prediction settings, such as online advertising bidding or network traffic classification, where input distributions drift quickly.
The reported 10 ms latency is a system-level figure; decomposing it into feature computation, network inference, and queueing would reveal which component the design actually optimises.
Reproducing the parameter-reduction result requires resolving the pruning criterion as written: Eq. 7 puts weights with $|w| > \tau$ into the pruning set, the opposite of the usual small-weight pruning, so the intended rule should be confirmed before comparing parameter counts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper does not hold together as written. The high-level idea—dynamic feature selection feeding a pruned lightweight CNN for low-latency HFT prediction—is a reasonable engineering combination, and the authors cite the right neighborhood of prior work. But the manuscript is too broken to evaluate, and the key pruning rule is inverted.

What is actually new: not much in principle. Clustering, mutual information, KDE, CNNs, and magnitude pruning are all standard. The only twist is weighting features by inverse within-cluster variance (Eq. 3), which is an ad hoc variant of variance-based feature selection. Credit where due: the design intent is clear, the component choices are defensible, and the latency/accuracy tradeoff is a real problem worth working on. The paper reads like a sincere attempt to combine known pieces.

The soft spots are not minor. Equations 1 through 9 have symbols replaced by question marks; as printed, Eq. 7 defines the pruning set as {w | |w| > τ}, which removes the large weights and destroys the network, and Eq. 8 penalizes small weights, pushing in the opposite direction. Even if these are typos, the experiments do not support the claims: no train/test split is described, the dataset is ambiguous, no code or data are shipped, hardware details are absent, and the abstract promises "revenue enhancement" but no profitability metric appears anywhere. The feature weighting in Eq. 3 rests on an unstated assumption that low within-cluster variance implies informativeness, which is false for near-constant features. The error shading in the figures has no described protocol behind it, so the "Ours wins" story is not independently checkable.

I agree with the stress-test note: the pruning inversion is a load-bearing defect. Even granting every reported number, the described method cannot produce the claimed lightweight network. This is not a case of a small flaw in an otherwise solid paper; it is a central mechanism that is internally inconsistent.

Who is this for? As it stands, no one. A careful reader cannot reproduce the method or verify the results. If the authors fixed the equations, released code and data, and provided an honest out-of-sample evaluation with a real latency measurement, the paper could be worth a second look—but that is a different submission.

Recommendation: desk-reject this version. It does not deserve referee time until the technical writing and experimental protocol are repaired.

Referee Report

5 major / 6 minor

Summary. The paper proposes a machine-learning pipeline for real-time high-frequency trading data processing, consisting of a dynamic feature-selection module (k-means clustering with inverse-variance feature weighting, mutual information, and kernel density estimation) and a lightweight CNN with pruning. The authors report experiments on 50ETF data comparing their method ('Ours') with five baselines on RMSE, R², execution time, and latency, claiming the lowest average execution time of 35 ms and latency of 10 ms while maintaining accuracy across market conditions.

Significance. If substantiated, the claimed combination of adaptive feature selection and a pruned low-latency CNN would be a practically relevant contribution to HFT systems. The paper addresses a timely problem and deserves credit for comparing against five named baselines and for reporting both accuracy and latency metrics. However, the manuscript as written does not support these claims: the core equations are garbled, the pruning rule is internally inconsistent, and no out-of-sample evaluation or measurement methodology is presented. The contribution cannot be independently assessed or reproduced from the supplied text.

major comments (5)

[III-A, Eqs. (1)-(5)] Equations (1)-(5) are rendered with placeholder '?' symbols in place of all mathematical notation, making the dynamic feature-selection procedure unreadable and unverifiable. Since every experimental claim depends on this procedure, the method as printed cannot be implemented or checked; this alone blocks reproducibility.
[III-B, Eqs. (7)-(8)] Equation (7) defines the pruning set as P_pruned = {w_{i,j} ∈ W | |w_{i,j}| > τ}, which is the reverse of standard magnitude pruning: it selects large-magnitude weights for removal. If implemented as written, the operation would delete the dominant convolutional features in Eq. (6) and destroy the network's learned representation; if the intended condition was |w_{i,j}| ≤ τ, then the equation is miswritten and the mechanism is not reproducible. Equation (8) compounds the inconsistency by penalizing the indicator 1(w_i ≤ τ), which encourages weights to move away from the sparse region rather than toward it. The paper's central speed and lightweight-network claims rest on this defective step.
[IV-A and IV-B] No train/test split, temporal walk-forward validation, or any held-out period is described. Feature weights, cluster assignments, network weights, and hyperparameters (k=10, τ=0.01, λ=0.1, learning rate 0.001, batch size 128) are all selected on the same data, so the reported RMSE and R² values may simply reflect in-sample fitting. The abstract's claim of 'consistent performance in varying market conditions' is therefore not supported by the experiments as reported.
[IV-B and Table I] The execution-time and latency results in Table I are reported as single numbers with no hardware/software environment, no measurement repetitions, and no statistical uncertainty. The text refers to 'error range shading' in Figures 2 and 3, but the actual plots are not available in the reviewed version, so the claimed stability and error bounds cannot be checked. In addition, neither the abstract's nor the conclusion's mention of 'revenue enhancement' is backed by any profitability, return, or cost metric in the experiments.
[III-A, Eq. (3)] The feature-weight definition in Eq. (3) is the reciprocal of the weighted within-cluster variance, so features with low intra-cluster variance are treated as most important. The paper provides no theoretical justification, empirical validation, or ablation for this inverse-variance assumption; if it is wrong, the dynamic feature selection will discard predictive features and the reported advantages collapse. This assumption is load-bearing for the proposed method and is left completely untested.

minor comments (6)

[I] The introduction says machine learning can formulate predictions and act 'in a matter of seconds,' which is inconsistent with HFT's microsecond-scale latency requirements; rephrase to reflect the actual time scales.
[IV-A] 'We selected 50 ETF datasets' and 'The 50ETF dataset contains...' are ambiguous; clarify whether 50 ETFs or the 50ETF contract is used, and describe the feature set precisely.
[II] The text attributes reference [9] to 'Li et al.,' but the reference list gives Xu, Ruohao et al.; correct the attribution.
[III-B, Eq. (6)] The notation in Eq. (6) is garbled (e.g., '? represents the size of the convolution kernel' and the bias term expression); define all variables and rewrite the convolution formula cleanly.
[IV-B] The prediction target (next-tick price, direction, return, or volatility) is never defined; specify it and the exact input features used in the experiments.
[V] The conclusion claims the method offers 'advantages in terms of processing speed and revenue enhancement,' but no revenue-related metric appears in Table I or the text; this claim should be removed or supported.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported prediction metrics reduce to the training objective because no held-out evaluation is described; the central performance claim is in-sample fit.

fitted input called prediction [Section IV-A and IV-B, Eq. 8, Figures 2-3]
"The dynamic feature selection module uses the k-means clustering algorithm, the number of clusters k is set to 10, and the feature weights are adjusted once in each training cycle. ... Figure 2 shows the comparison of the root mean square error of different methods, reflecting the prediction performance of the model with different parameter settings by increasing the size of the data window. It can be seen that the Ours method exhibits the lowest RMSE across all parameters, indicating its superiority in prediction accuracy and stability."

Equation 8 defines the only training objective as L_train(W) plus a pruning penalty. The experimental section then reports RMSE and R^2 for the 'Ours' method as 'prediction performance', but no train/test or validation split is specified anywhere in the paper. The feature weights (including Eq. 3), network weights, and hyperparameters are all selected on the same 50 ETF datasets, so the reported RMSE and R^2 are in-sample values of the optimized objective rather than out-of-sample predictions. The claimed superiority in accuracy is therefore a restatement of the fit, not independent predictive evidence.

full rationale

The paper contains no self-citation or imported uniqueness argument; its references are external, so the self-citation categories do not apply. The central circularity is empirical: the 'prediction performance' in Figures 2 and 3 and Table I is computed on the same data used to fit the dynamic feature weights, the CNN weights, and the hyperparameters. No held-out split, rolling-origin evaluation, or external benchmark with independent labels is described. Thus the reported RMSE/R^2 advantages reduce to the training objective in Eq. 8, and the latency claim, while separately questionable because the pruning condition in Eq. 7 is inverted relative to magnitude pruning, is not itself a circular step. Because the paper's main claimed contribution is the demonstrated accuracy and speed of the proposed pipeline, and that demonstration rests on in-sample fitting, the central claim is partially circular. Score 6.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a collection of hand-chosen hyperparameters and several unvalidated modeling assumptions. The feature weighting in Eq 3 is the most fragile: it asserts that low cluster variance implies high feature importance. The experimental section provides no sensitivity analysis and no held-out data, so all fitted quantities are entangled with the reported metrics.

free parameters (8)

Number of clusters k = 10
Set by hand in Section IV-A for k-means clustering in dynamic feature selection.
Pruning threshold τ = 0.01
Set in Section IV-A; determines which weights survive and directly affects accuracy and speed.
Regularization parameter λ = 0.1
Set in Section IV-A; controls pruning intensity in Eq 8.
Adam initial learning rate = 0.001
Set in Section IV-A; decays by half every 10 training cycles.
Batch size = 128
Set in Section IV-A for training.
Convolution kernel sizes F = [3x3, 5x5]
Chosen in Section IV-A for multi-scale feature extraction.
KDE bandwidth h = not specified
Required in Eq 5 for mutual information estimation; no value or selection method is given.
Network depth L = not specified
Number of convolution modules in Eq 9 is not given, so the architecture is incomplete.

assumptions (5)

domain assumption K-means++ cluster centers on market data yield stable feature weights.
Used in Eqs 1-3 to define feature weights; no stationarity or cluster-stability evidence is given.
ad hoc to paper Feature importance is the reciprocal of weighted within-cluster variance.
This is an unmotivated modeling choice in Eq 3; low-variance features are assumed more predictive, which is not generally true and is not validated.
domain assumption KDE with bandwidth h estimates the joint distribution for mutual information reliably in real time.
Used in Eq 5; no bandwidth selection or convergence analysis is provided.
domain assumption Pruning weights below threshold τ preserves prediction accuracy.
Standard pruning assumption, but the displayed inequality '|w| > τ' in Eq 7 appears to invert the intended pruning set.
domain assumption The 50 ETF dataset(s) represent HFT market conditions and contain learnable signal.
No data source, date range, or ticker is given; the text conflates '50 ETF datasets' with the '50ETF' instrument.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Research on Optimizing Real-Time Data Processing in High-Frequency Trading Algorithms using Machine Learning." pith.science (2026). https://pith.science/paper/4KHT4YDN

@misc{pith2026241201062,
  author       = {Pith},
  title        = {Pith review of: Research on Optimizing Real-Time Data Processing in High-Frequency Trading Algorithms using Machine Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KHT4YDN}},
  note         = {Machine review of arXiv:2412.01062}
}

read the original abstract

High-frequency trading (HFT) represents a pivotal and intensely competitive domain within the financial markets. The velocity and accuracy of data processing exert a direct influence on profitability, underscoring the significance of this field. The objective of this work is to optimise the real-time processing of data in high-frequency trading algorithms. The dynamic feature selection mechanism is responsible for monitoring and analysing market data in real time through clustering and feature weight analysis, with the objective of automatically selecting the most relevant features. This process employs an adaptive feature extraction method, which enables the system to respond and adjust its feature set in a timely manner when the data input changes, thus ensuring the efficient utilisation of data. The lightweight neural networks are designed in a modular fashion, comprising fast convolutional layers and pruning techniques that facilitate the expeditious completion of data processing and output prediction. In contrast to conventional deep learning models, the neural network architecture has been specifically designed to minimise the number of parameters and computational complexity, thereby markedly reducing the inference time. The experimental results demonstrate that the model is capable of maintaining consistent performance in the context of varying market conditions, thereby illustrating its advantages in terms of processing speed and revenue enhancement.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages

[1]

Algorithmic Trading Strategies: Real-Time Data Analytics with Machine Learning

Devan, Munivel, Kumaran Thirunavukkarasu, and Lavanya Shanmugam. "Algorithmic Trading Strategies: Real-Time Data Analytics with Machine Learning." Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online) 2.3 (2023): 522-546

work page 2023
[2]

Efficient Continuous Space Policy Optimization for High-frequency Trading

Han, Li, et al. "Efficient Continuous Space Policy Optimization for High-frequency Trading." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023

work page 2023
[3]

Novel modelling strategies for high-frequency stock trading data

Zhang, Xuekui, et al. "Novel modelling strategies for high-frequency stock trading data." Financial Innovation 9.1 (2023): 39

work page 2023
[4]

Major Issues in High-frequency Financial Data Analysis: A Survey of Solutions

Zhang, Lu, and Lei Hua. "Major Issues in High-frequency Financial Data Analysis: A Survey of Solutions." Available at SSRN 4834362 (2024)

work page 2024
[5]

HFTViz: Visualization for the exploration of high frequency trading data

Yaali, Javad, Vincent Grégoire, and Thomas Hurtut. "HFTViz: Visualization for the exploration of high frequency trading data." Information Visualization 21.2 (2022): 182-193

work page 2022
[6]

Deep adaptive input normalization for time series forecasting

Passalis, Nikolaos, et al. "Deep adaptive input normalization for time series forecasting." IEEE transactions on neural networks and learning systems 31.9 (2019): 3760-3765

work page 2019
[7]

Forecasting stock index futures intraday returns: Functional time series model

Fu, Yizheng, et al. "Forecasting stock index futures intraday returns: Functional time series model." Journal of Advanced Computational Intelligence and Intelligent Informatics 24.3 (2020): 265-271

work page 2020
[8]

Non-stationary transformers: Exploring the stationarity in time series forecasting

Liu, Yong, et al. "Non-stationary transformers: Exploring the stationarity in time series forecasting." Advances in Neural Information Processing Systems 35 (2022): 9881-9893

work page 2022

Show all 10 references

[9]

Dynamic feature selection algorithm based on Q- learning mechanism

Xu, Ruohao, et al. "Dynamic feature selection algorithm based on Q- learning mechanism." Applied Intelligence (2021): 1-12

2021
[10]

Reinforcement Learning based Evolutionary Metric Filtering for High Dimensional Problems

Ali, Bassel, et al. "Reinforcement Learning based Evolutionary Metric Filtering for High Dimensional Problems." 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 2020

2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Algorithmic Trading Strategies: Real-Time Data Analytics with Machine Learning

Devan, Munivel, Kumaran Thirunavukkarasu, and Lavanya Shanmugam. "Algorithmic Trading Strategies: Real-Time Data Analytics with Machine Learning." Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online) 2.3 (2023): 522-546

work page 2023

[2] [2]

Efficient Continuous Space Policy Optimization for High-frequency Trading

Han, Li, et al. "Efficient Continuous Space Policy Optimization for High-frequency Trading." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023

work page 2023

[3] [3]

Novel modelling strategies for high-frequency stock trading data

Zhang, Xuekui, et al. "Novel modelling strategies for high-frequency stock trading data." Financial Innovation 9.1 (2023): 39

work page 2023

[4] [4]

Major Issues in High-frequency Financial Data Analysis: A Survey of Solutions

Zhang, Lu, and Lei Hua. "Major Issues in High-frequency Financial Data Analysis: A Survey of Solutions." Available at SSRN 4834362 (2024)

work page 2024

[5] [5]

HFTViz: Visualization for the exploration of high frequency trading data

Yaali, Javad, Vincent Grégoire, and Thomas Hurtut. "HFTViz: Visualization for the exploration of high frequency trading data." Information Visualization 21.2 (2022): 182-193

work page 2022

[6] [6]

Deep adaptive input normalization for time series forecasting

Passalis, Nikolaos, et al. "Deep adaptive input normalization for time series forecasting." IEEE transactions on neural networks and learning systems 31.9 (2019): 3760-3765

work page 2019

[7] [7]

Forecasting stock index futures intraday returns: Functional time series model

Fu, Yizheng, et al. "Forecasting stock index futures intraday returns: Functional time series model." Journal of Advanced Computational Intelligence and Intelligent Informatics 24.3 (2020): 265-271

work page 2020

[8] [8]

Non-stationary transformers: Exploring the stationarity in time series forecasting

Liu, Yong, et al. "Non-stationary transformers: Exploring the stationarity in time series forecasting." Advances in Neural Information Processing Systems 35 (2022): 9881-9893

work page 2022