REVIEW 4 major objections 4 minor 34 references
Hammer: Towards Efficient Hot-Cold Data Identification via Online Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read An adaptive random forest trained online identifies hot and cold data with over 90% accuracy across dynamic workloads.
desk verdict Plausible design, but the 90% accuracy claim is circular because training and evaluation share the same Sketch-Min-plus-threshold labels. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by three coupled components. The first is the adaptive random forest, an online ensemble that grows trees incrementally and can replace underperforming trees when the data distribution shifts, which is the component that is supposed to defeat concept drift. The second is the Sketch-Min counting sketch: D hash functions each map an access address to W counters, and the estimated access count is the minimum over the hash positions, giving a compact approximate counter whose error is bounded by the sketch dimensions. The third is the dynamic threshold tuning algorithm, which sets the hot/cold boundary as a percentile of recent estimated hotness and moves that percentile up or down with slow-tier utilization, thrashing, and CPU load. The label-generation pipeline, consisting of the Sketch-Min count, the percentile threshold, and the evaluation-queue timing, is the load-bearing mechanism because it produces the ground truth that both trains and tests the classifier.
What would settle it
Replay the same workloads but compute ground-truth hot/cold labels with exact per-address access counters and an independently chosen threshold, or a manually curated hot set, then score Hammer against those labels; if accuracy drops well below 90% under concept drift, the reported result is an artifact of using its own Sketch-Min-derived labels as ground truth.
Extended reading notes
Core claim
The central claim the paper argues for is that the hot/cold classification problem can be recast as an online learning problem and solved with an adaptive random forest (ARF) classifier. Each memory access is reduced to a feature vector from data-flow, control-flow, and system information; a finite evaluation queue holds recent accesses; and a Sketch-Min counter estimates per-address access counts that, combined with a dynamically tuned percentile threshold, supply the labels used both for training and for evaluation. The paper shows that on concatenated real-workload traces, ARF reaches 90.33% accuracy and an F1 score of 86.28%, against 71.09% accuracy for the LRU2Q baseline, and that the online model sustains accuracy under concept drift where a batch model decays. The dynamic threshold is adjusted periodically from slow-tier usage, ping-pong (thrashing) behavior, and CPU occupancy, so "hot" is defined relative to current system state rather than a fixed cutoff.
Load-bearing premise
The load-bearing premise is that the Sketch-Min count estimates plus the dynamically tuned percentile threshold define the true hot/cold state, because those same labels are used both to train the classifier and to score its accuracy; if those labels do not match real access heat, the 90% figure measures self-consistency rather than correct identification.
Editorial extensions
If this is right
- Storage systems can drive hot/cold migration from an online classifier instead of per-object recency/frequency lists, avoiding metadata explosion.
- The approach adapts to concept drift without periodic batch retraining, so accuracy need not decay when workload mixes change.
- The dynamic threshold means the definition of "hot" tracks system state, including slow-tier pressure, thrashing, and CPU budget, rather than a fixed count cutoff.
- Applying Hammer to heterogeneous memory or storage tiers could place hot data on fast tiers automatically.
- The Sketch-Min-based labels keep training data cheap enough for continuous online evaluation.
Reading between the lines
- Editorial extension: Because the reported 90% accuracy is measured against labels produced by the very same Sketch-Min estimates and percentile threshold that define Hammer's judgment, an independent evaluation with exact reference counters would test whether the accuracy reflects true access heat rather than self-consistency.
- Editorial extension: If the online classifier transfers across real systems, the same feature stream could also drive cache admission, prefetching, and I/O scheduling decisions, not just tier placement.
- Editorial extension: The Sketch-Min dimensions D and W control the accuracy-memory tradeoff; a production deployment would want a sensitivity study showing how classifier accuracy degrades as the sketch is shrunk, which the paper does not report.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hammer, an online-learning system for identifying hot and cold data in storage systems. Hammer extracts data-flow, control-flow, and system-level features from access traces, uses Sketch-Min counting with a dynamic percentile threshold to assign hot/cold labels, and trains an adaptive random forest online on those labels. The authors report more than 90% hot/cold classification accuracy across AI, big data, graph, and HPC workloads, plus claims of low overhead. The evaluation is based on replayed memory access traces collected with Drmemtrace, and the paper compares an adaptive random forest against LRU2Q, naïve Bayes, and a Hoeffding adaptive tree.
Significance. If the accuracy and overhead claims were properly validated, Hammer would address a real problem: online hot/cold identification with compact metadata and adaptability to concept drift. The combination of an online classifier with Sketch-Min-based estimation is a reasonable direction, and the workload coverage is broad. However, the current evaluation is self-referential: the labels used for training and for accuracy measurement come from Hammer's own Sketch-Min-plus-threshold mechanism, so the reported accuracy may only measure consistency with this internal rule. The paper also contains an unresolved placeholder for the batch-learning comparison and no measured overhead numbers. These issues prevent the significance of the result from being assessed.
major comments (4)
- [§3.3, Fig. 3; §4.2, Table 2] The labels Y_true used for both online training and for the accuracy/F1 numbers in Table 2 are produced by Hammer's own Sketch-Min count estimates and its dynamic percentile threshold (Algorithm 1). The reported 90.33% accuracy therefore measures how well the adaptive random forest reproduces Hammer's internal labeling rule, not whether that rule correctly identifies actual hot/cold data. The paper needs an independent ground truth, for example exact access counts over a fixed window, an externally fixed and motivated definition of hot/cold, or an end-to-end tiering or migration benefit. Without such validation, the headline claim is not supported.
- [§4.2, Table 2] The paper states that paired t-tests showed statistically significant improvements with p-values below 0.05, but no p-values, confidence intervals, standard deviations, or per-fold results are reported. In addition, the accuracy and F1 computation for the LRU2Q baseline is never defined; it is unclear how a cache policy such as LRU2Q is converted into per-item hot/cold classification. The comparison in Table 2 is therefore not verifiable as presented.
- [§4.2] The batch-versus-online comparison that is central to the concept-drift argument is not actually reported; the text says 'The result is shown in xxx', which is a placeholder rather than a result. Additionally, the abstract and introduction claim reduced computational and storage overhead, but no overhead measurements appear anywhere in the evaluation. These claims need to be either quantified or removed.
- [§3.4, Algorithm 1] Line 9 of Algorithm 1 sets p = min{Pmin, p/2}, which can push the threshold percentile below the declared lower bound Pmin; this contradicts the clamping logic on lines 12–13. The update also depends on empirical coefficients α and β whose values are never specified, and on an error-bound computation ϵ that is never defined. Without concrete parameter values and a sensitivity analysis, the dynamic threshold behavior and the resulting labels cannot be reproduced.
minor comments (4)
- [§1 and §2] There are typographical errors such as 'learning-baesd' in the contributions list and a stray closing parenthesis after '[8]' in Section 2.
- [§3.3 and §3.4] Several key parameters are never specified: the Sketch-Min dimensions D and W, the evaluation queue capacity, the instruction sampling rate, and the values of α, β, Pinit, Pmin, and Pmax. These details are needed to reproduce the accuracy and overhead results.
- [§4.1] The abstract mentions both synthetic and real-world datasets, but Section 4 describes only traces collected from real workloads; the use of synthetic data should be clarified or the abstract adjusted.
- [References] Reference formatting is inconsistent, including 'InProceedings' without a space and duplicate entries; the references should be harmonized with the journal or conference style.
Circularity Check
The 90% accuracy claim measures self-consistency: Y_true is produced by Hammer's own Sketch-Min counting and dynamic threshold, then used as both training target and evaluation ground truth.
-
self definitional
[Section 3.3 (Online Classification and Evaluation, Fig. 3); Section 3.4 (Algorithm 1); Section 4.2 (Table 2)]
"Once the queue is full and the feature vector is no longer in the queue, the real label of the data access is obtained through the online evaluation algorithm. This real label is then fed back into the system as training data for the online training of classifiers. ... the actual count value of the element is determined by the minimum of the count values of multiple hash function mapped positions."
The label Y_true that trains the classifier and the ground truth used to compute ARF's 90.33% accuracy in Table 2 are not independent of Hammer itself. Figure 3 defines Y_true as the Sketch-Min count estimate compared with the dynamic percentile threshold from Algorithm 1, and the same Sketch-Min counting and threshold are part of Hammer's own 'online heat judgement' module. Consequently, accuracy is P(Y_pred = Y_true), i.e. how well ARF reproduces Hammer's own Sketch-Min-plus-threshold labeling rule on the feature stream. It does not establish that this label corresponds to actual data heat.
full rationale
The main circular step is the definition and use of Y_true. Section 3.3 states that the 'real label' is obtained through the online evaluation algorithm, which is the Sketch-Min counting method described in the same section and Fig. 3, followed by the dynamic percentile threshold of Section 3.4 and Algorithm 1. This same Y_true is fed back for online training and is the target for the accuracy and F1 metrics in Table 2. Therefore the reported 90% accuracy demonstrates that the adaptive random forest can reproduce Hammer's internal labeling rule, not that the rule correctly identifies hot and cold data. If Sketch-Min count estimates are biased or the alpha/beta coefficients make the threshold arbitrary, both the training signal and the evaluation metric are corrupted in the same direction, so the claimed advantage over LRU2Q and the concept-drift adaptation conclusion are not independently established. The comparison with batch learning is also left incomplete ('The result is shown in xxx.'), and the overhead reduction is asserted without measured numbers. These missing pieces affect completeness rather than circularity, but they corroborate that the only quantitative support for the headline claim is the self-consistent label-reproduction accuracy. No load-bearing self-citation or imported uniqueness theorem appears in the paper, so the circularity is localized to the ground-truth construction rather than to the citation chain.
Assumptions & free parameters
free parameters (6)
- alpha (slow bandwidth impact factor) =
not reported
- beta (ping-pong event impact factor) =
not reported
- Pinit, Pmin, Pmax (percentile bounds) =
not reported
- Sketch-Min dimensions D and W =
not reported
- Evaluation queue capacity =
not reported
- Instruction sampling rate =
10%
assumptions (3)
- domain assumption Labels from Sketch-Min estimated counts plus a dynamic threshold are valid ground truth for hot/cold status.
- domain assumption 10% random sampling of memory access instructions is representative of full access behavior.
- domain assumption Concatenating traces from different applications preserves realistic concept drift.
Cite this review
Pith. "Pith review of Hammer: Towards Efficient Hot-Cold Data Identification via Online Learning." pith.science (2026). https://pith.science/paper/NKGFBNWV
@misc{pith2026241114759,
author = {Pith},
title = {Pith review of: Hammer: Towards Efficient Hot-Cold Data Identification via Online Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/NKGFBNWV}},
note = {Machine review of arXiv:2411.14759}
}
read the original abstract
Efficient management of storage resources in big data and cloud computing environments requires accurate identification of data's "cold" and "hot" states. Traditional methods, such as rule-based algorithms and early AI techniques, often struggle with dynamic workloads, leading to low accuracy, poor adaptability, and high operational overhead. To address these issues, we propose a novel solution based on online learning strategies. Our approach dynamically adapts to changing data access patterns, achieving higher accuracy and lower operational costs. Rigorous testing with both synthetic and real-world datasets demonstrates a significant improvement, achieving a 90% accuracy rate in hot-cold classification. Additionally, the computational and storage overheads are considerably reduced.
Figures
Reference graph
Works this paper leans on
-
[18]
Data heat prediction in storage systems using behavior specific prediction models
Wei Zhang, Jian Li, and Zhonghua Wang. Data heat prediction in storage systems using behavior specific prediction models. IEEE Transactions on Storage, 16(3):1–14, 2020
work page 2020
-
[19]
Predicting frequent-infrequent access of objects in tiered cloud storage services
Zhonghua Wang, Jian Li, and Wei Zhang. Predicting frequent-infrequent access of objects in tiered cloud storage services. Journal of Parallel and Distributed Computing, 118:123–134, 2018
work page 2018
-
[1]
Optimizing data storage in cloud computing: techniques and best practices
Anil Kumar Yadav Yanamala. Optimizing data storage in cloud computing: techniques and best practices. International Journal of Advanced Engineering Technologies and Innovations, 1(3):476–513, 2024
work page 2024
-
[2]
Hdfs optimization strategy based on hierarchical storage of hot and cold data
Yuxin Guan, Zhiqiang Ma, and Leixiao Li. Hdfs optimization strategy based on hierarchical storage of hot and cold data. Procedia CIRP, 83:415–418, 2019
work page 2019
-
[4]
Energy efficiency and renewable energy integration in data centres
Eduard Oró, Victor Depoorter, Albert Garcia, and Jaume Salom. Energy efficiency and renewable energy integration in data centres. strategies and modelling review. Renewable and Sustainable Energy Reviews, 42:429– 445, 2015
work page 2015
-
[5]
A Shastri Pothukuchi, Lakshmi Vasuda Kota, and Vinay Mallikarjunaradhya. a critical analysis of the challenges and opportunities to optimize storage costs for big data in the cloud. Asian J. Multidiscip. Res. Rev. ISSN , 25(1):132–144, 2021
work page 2021
-
[6]
Vasiliki Pouli, Stella Kafetzoglou, Eirini Eleni Tsiropoulou, Aggeliki Dimitriou, and Symeon Papavassiliou. Personalized multimedia content retrieval through relevance feedback techniques for enhanced user experience. In 2015 13th International Conference on Telecommunications (ConTEL), pages 1–8. IEEE, 2015
work page 2015
-
[7]
Chi-bd: A fuzzy rule-based classification system for big data classification problems
Mikel Elkano, Mikel Galar, Jose Sanz, and Humberto Bustince. Chi-bd: A fuzzy rule-based classification system for big data classification problems. Fuzzy Sets and Systems, 348:75–101, 2018
work page 2018
Show all 34 references
-
[8]
Adaptive learning for dynamic environments: A comparative approach
Joana Costa, Catarina Silva, Mário Antunes, and Bernardete Ribeiro. Adaptive learning for dynamic environments: A comparative approach. Engineering Applications of Artificial Intelligence, 65:336–345, 2017
2017
-
[9]
Predicting frequent-infrequent access of objects in tiered cloud storage services
Flávio AA Motta, Glauber Dias Gonçalves, Heder S Bernardino, Saulo Moraes Villela, and Alex B Vieira. Predicting frequent-infrequent access of objects in tiered cloud storage services. 2024
2024
-
[10]
Data heat prediction in storage systems using behavior specific prediction models
Lu Pang, Anis Alazzawe, Krishna Kant, and Jeremy Swift. Data heat prediction in storage systems using behavior specific prediction models. In 2019 IEEE 38th International Performance Computing and Communications Conference (IPCCC), pages 1–8. IEEE, 2019
2019
-
[11]
A novel automated cloud storage tiering system through hot-cold data classification
Ying-Feng Hsu, Ryo Irie, Shuuichirou Murata, and Morito Matsuoka. A novel automated cloud storage tiering system through hot-cold data classification. In 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pages 492–499. IEEE, 2018
2018
-
[12]
An efficient hot-cold data separation garbage collection algorithm based on logical interval in nand flash-based consumer electronics
Xiaoming Liu, Wei Zhang, and Jian Li. An efficient hot-cold data separation garbage collection algorithm based on logical interval in nand flash-based consumer electronics. IEEE Transactions on Consumer Electronics, 67(2):123–130, 2021
2021
-
[13]
Streaming machine learning for supporting data prefetching in modern data storage systems
Zhonghua Wang, Jian Li, and Wei Zhang. Streaming machine learning for supporting data prefetching in modern data storage systems. IEEE Transactions on Knowledge and Data Engineering, 32(5):891–904, 2020
2020
-
[14]
A novel hot-cold data identification mechanism based on multidimensional data
Jian Li, Xiaoming Liu, and Wei Zhang. A novel hot-cold data identification mechanism based on multidimensional data. Journal of Data Science, 19(3):215–232, 2021
2021
-
[15]
A hierarchical storage mechanism for hot and cold data based on temperature model
Zhonghua Wang, Jian Li, and Wei Zhang. A hierarchical storage mechanism for hot and cold data based on temperature model. Journal of Cloud Computing, 9(1):1–15, 2020
2020
-
[16]
A survey on ai for storage
Lina Zhang, Zhonghua Wang, and Jian Li. A survey on ai for storage. ACM Computing Surveys, 54(3):1–35, 2022
2022
-
[17]
Keep hot or go cold: A randomized online migration algorithm for cost optimization in staas clouds
Jian Li, Zhonghua Wang, and Wei Zhang. Keep hot or go cold: A randomized online migration algorithm for cost optimization in staas clouds. IEEE Transactions on Cloud Computing, 7(4):1122–1135, 2019
2019
-
[20]
Online learning: A comprehensive survey
Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: A comprehensive survey. Neurocomput- ing, 459:249–289, 2021
2021
-
[21]
Streaming machine learning for supporting data prefetching in modern data storage systems
Edson Ramiro Lucas Filho, Lun Yang, Kebo Fu, and Herodotos Herodotou. Streaming machine learning for supporting data prefetching in modern data storage systems. In Proceedings of the First Workshop on AI for Systems, pages 7–12, 2023. 9
2023
-
[22]
Modified lru policies for improving second-level cache behavior
Wayne A Wong and J-L Baer. Modified lru policies for improving second-level cache behavior. InProceedings Sixth International Symposium on High-Performance Computer Architecture. HPCA-6 (Cat. No. PR00550), pages 49–60. IEEE, 2000
2000
-
[23]
Practical lfu implementation for web caching
George Karakostas and D Serpanos. Practical lfu implementation for web caching. Technical Report TR-622-00, 2000
2000
-
[24]
Hybrid caching strategy for streaming media files
Jussara M Almeida, Derek L Eager, and Mary K Vernon. Hybrid caching strategy for streaming media files. In Multimedia Computing and Networking 2001, volume 4312, pages 200–212. SPIE, 2000
2001
-
[25]
Self-learning hot data prediction: Where echo state network meets nand flash memories
Wei Zhang, Jian Li, and Zhonghua Wang. Self-learning hot data prediction: Where echo state network meets nand flash memories. IEEE Transactions on Neural Networks and Learning Systems, 33(2):456–468, 2022
2022
-
[26]
A reinforcement learning framework for online data migration in hierarchical storage systems
David Vengerov. A reinforcement learning framework for online data migration in hierarchical storage systems. The Journal of Supercomputing, 43(1):1–19, 2008
2008
-
[27]
Learning under concept drift: A review
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12):2346–2363, 2018
2018
-
[28]
A survey on concept drift adaptation
João Gama, Indr˙e Žliobait˙e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1–37, 2014
2014
-
[29]
Characterizing concept drift
Geoffrey I Webb, Roy Hyde, Hong Cao, Hai Long Nguyen, and Francois Petitjean. Characterizing concept drift. Data Mining and Knowledge Discovery, 30(4):964–994, 2016
2016
-
[30]
Towards cloud storage tier optimization with rule-based classification
Akif Quddus Khan, Nikolay Nikolov, Mihhail Matskin, Radu Prodan, Christoph Bussler, Dumitru Roman, and Ahmet Soylu. Towards cloud storage tier optimization with rule-based classification. In European Conference on Service-Oriented and Cloud Computing, pages 205–216. Springer, 2023
2023
-
[31]
Real-time anomaly detection in data centers for log-based predictive maintenance using an evolving fuzzy-rule-based approach
Leticia Decker, Daniel Leite, Luca Giommi, and Daniele Bonacorsi. Real-time anomaly detection in data centers for log-based predictive maintenance using an evolving fuzzy-rule-based approach. In 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pages 1–8. IEEE, 2020
2020
-
[32]
Reinforcement learning-assisted management for convertible ssds
Qian Wei, Yi Li, Zhiping Jia, Mengying Zhao, Zhaoyan Shen, and Bingzhe Li. Reinforcement learning-assisted management for convertible ssds. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE, 2023
2023
-
[33]
Sibyl: Adaptive and extensible data placement in hybrid storage systems using online reinforcement learning
Gagandeep Singh, Rakesh Nadig, Jisung Park, Rahul Bera, Nastaran Hajinazar, David Novo, Juan Gómez-Luna, Sander Stuijk, Henk Corporaal, and Onur Mutlu. Sibyl: Adaptive and extensible data placement in hybrid storage systems using online reinforcement learning. In Proceedings o...
2022
-
[34]
Parallel algorithms and architectures for rule-based systems
Anoop Gupta, Charles Forgy, Allen Newell, and Robert Wedig. Parallel algorithms and architectures for rule-based systems. ACM SIGARCH Computer Architecture News, 14(2):28–37, 1986
1986
-
[35]
Micro-armed bandit: lightweight & reusable reinforcement learning for microarchitecture decision-making
Gerasimos Gerogiannis and Josep Torrellas. Micro-armed bandit: lightweight & reusable reinforcement learning for microarchitecture decision-making. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, pages 698–713, 2023. 10
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.