REVIEW 54 references
Detecting Flow Gaps in Data Streams
T0 review · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read GapFilter uses sequence-number-only matching and a civilian-suspect bucket split to detect major per-flow sequence gaps, beating a Cuckoo-filter baseline in F1, speed, and memory.
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
GapFilter keeps a small table of buckets. Instead of storing the 13-byte flow ID, it stores only a few recent sequence numbers per bucket. When a new item arrives, the algorithm finds the stored number closest to its sequence; if the difference is between 5 and 30, it reports a major gap. Because normal flows are common and big gaps are rare, the closest number is usually the same flow's previous number. A second, accuracy-oriented version splits each bucket into civilian and suspect regions, so flows that once had a gap are watched more carefully and are not pushed out by large normal flows. SIMD instructions make the lookup and rearrangement constant time.
Tests on four internet traces with synthetic packet loss show higher F1 scores than a Cuckoo-filter Straw-man at equal memory. The authors claim their accuracy-oriented version needs about 1/32 of the memory to reach the same accuracy, and their speed version is faster. The paper also gives a theoretical recall bound, but that bound assumes a flow in the bucket is always detected, which side-steps the hardest part of the matching problem.
Extended reading notes
Core claim
The paper's central claim is that GapFilter is the first solution for monitoring the variation of value per key, and that it achieves high accuracy with minimal memory: 'GapFilter-AO requires, on average, 1/32 of the memory to match the accuracy of the Straw-man solution. GapFilter-SO operates at a speed 3 times faster than the Straw-man solution' (abstract and Section 1.2). The theoretical counterpart is 'GapFilter secures high accuracy with minimal memory usage' via Theorem 1's recall bound.
Load-bearing premise
The load-bearing premise is the Section 5 assumption that a major gap is reported whenever the flow's record is still present in its bucket: 'the major gap's being reported indicates that the flow is contained in the existing items in the ith bucket.' This lets the proof ignore the algorithm's actual matching step, where the stored sequence number closest to the incoming one may belong to a different flow with a similar randomized range, causing false matches and missed gaps. The entire memory advantage comes from dropping flow IDs and trusting this nearest-sequence match, so without a bound on matching errors the theorem does not establish the headline accuracy claim.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Assumptions & free parameters
free parameters (4)
- w (cells per bucket) =
8
- lf (fingerprint length) =
8 bits
- suspect:civilian ratio (s:c) =
3:5
- T1 and T2 thresholds =
T1=5, T2=30
assumptions (6)
- standard math Flows are mapped uniformly at random to buckets by the hash function.
- domain assumption Flow sizes follow a Zipf distribution with 1 < alpha <= 3.
- domain assumption All flows share the same gap ratio beta.
- ad hoc to paper If a flow is resident in its bucket, every major gap in it is reported.
- domain assumption Randomized sequence numbers keep concurrently active flows' ranges in a bucket sufficiently separated for nearest-neighbor matching to be correct.
- domain assumption The synthetic item-loss model (consecutive with probability b^j, single with probability p) is representative of real network loss patterns.
Cite this review
Pith. "Pith review of Detecting Flow Gaps in Data Streams." pith.science (2026). https://pith.science/paper/2XZHRLWF
@misc{pith2026250513945,
author = {Pith},
title = {Pith review of: Detecting Flow Gaps in Data Streams},
year = {2026},
howpublished = {\url{https://pith.science/paper/2XZHRLWF}},
note = {Machine review of arXiv:2505.13945}
}
read the original abstract
Data stream monitoring is a crucial task which has a wide range of applications. The majority of existing research in this area can be broadly classified into two types, monitoring value sum and monitoring value cardinality. In this paper, we define a third type, monitoring value variation, which can help us detect flow gaps in data streams. To realize this function, we propose GapFilter, leveraging the idea of Sketch for achieving speed and accuracy. To the best of our knowledge, this is the first work to detect flow gaps in data streams. Two key ideas of our work are the similarity absorption technique and the civilian-suspect mechanism. The similarity absorption technique helps in reducing memory usage and enhancing speed, while the civilian-suspect mechanism further boosts accuracy by organically integrating broad monitoring of overall flows with meticulous monitoring of suspicious flows.We have developed two versions of GapFilter. Speed-Oriented GapFilter (GapFilter-SO) emphasizes speed while maintaining satisfactory accuracy. Accuracy-Oriented GapFilter (GapFilter-AO) prioritizes accuracy while ensuring considerable speed. We provide a theoretical proof demonstrating that GapFilter secures high accuracy with minimal memory usage. Further, extensive experiments were conducted to assess the accuracy and speed of our algorithms. The results reveal that GapFilter-AO requires, on average, 1/32 of the memory to match the accuracy of the Straw-man solution. GapFilter-SO operates at a speed 3 times faster than the Straw-man solution. All associated source code has been open-sourced and is available on GitHub.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
https://github.com/ GapFilter/GapFilter
The source codes of ours and other related algorithms . https://github.com/ GapFilter/GapFilter
-
[2]
Jingyue Pang, Datong Liu, Haitao Liao, Yu Peng, and Xiyuan Peng. Anomaly detection based on data stream monitoring and prediction with improved gauss- ian process regression algorithm. In 2014 International Conference on Prognostics and Health Management, pages 1–7. IEEE, 2014
work page 2014
-
[3]
Spotlight: Detecting anomalies in streaming graphs
Dhivya Eswaran, Christos Faloutsos, Sudipto Guha, and Nina Mishra. Spotlight: Detecting anomalies in streaming graphs. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1378–1386, 2018
work page 2018
-
[4]
Bursts- ketch: Finding bursts in data streams
Ruijie Miao, Zheng Zhong, Jiarui Guo, Zikun Li, Tong Yang, and Bin Cui. Bursts- ketch: Finding bursts in data streams. IEEE Transactions on Knowledge and Data Engineering, 2022
work page 2022
-
[5]
Fast memory-efficient anomaly detection in streaming heterogeneous graphs
Emaad Manzoor, Sadegh M Milajerdi, and Leman Akoglu. Fast memory-efficient anomaly detection in streaming heterogeneous graphs. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1035–1044, 2016
work page 2016
-
[6]
{SketchLib}: Enabling efficient sketch-based monitoring on programmable switches
Hun Namkung, Zaoxing Liu, Daehyeok Kim, Vyas Sekar, and Peter Steenkiste. {SketchLib}: Enabling efficient sketch-based monitoring on programmable switches. In 19th USENIX Symposium on Networked Systems Design and Im- plementation (NSDI 22), pages 743–759, 2022
work page 2022
-
[7]
Bitsense: Universal and nearly zero-error optimization for sketch counters with compressive sensing
Rui Ding, Shibo Yang, Xiang Chen, and Qun Huang. Bitsense: Universal and nearly zero-error optimization for sketch counters with compressive sensing. In Proc. of SIGCOMM, 2023
work page 2023
-
[8]
Machine learning based concept drift detection for predictive maintenance
Jan Zenisek, Florian Holzinger, and Michael Affenzeller. Machine learning based concept drift detection for predictive maintenance. Computers & Industrial Engineering, 137:106031, 2019
work page 2019
Show all 54 references
-
[9]
Big data and stream processing platforms for industry 4.0 requirements mapping for a predictive maintenance use case
Radhya Sahal, John G Breslin, and Muhammad Intizar Ali. Big data and stream processing platforms for industry 4.0 requirements mapping for a predictive maintenance use case. Journal of manufacturing systems , 54:138–151, 2020
2020
-
[10]
Data-driven maintenance: combining predictive maintenance and mixed reality-supported remote assis- tance
Josef Wolfartsberger, Jan Zenisek, and Norbert Wild. Data-driven maintenance: combining predictive maintenance and mixed reality-supported remote assis- tance. Procedia Manufacturing, 45:307–312, 2020
2020
-
[11]
An e-customer behavior model with online analytical mining for internet marketing planning
Irene SY Kwan, Joseph Fong, and Hing K Wong. An e-customer behavior model with online analytical mining for internet marketing planning. Decision Support Systems, 41(1):189–204, 2005
2005
-
[12]
Customer behavior analysis using real-time data processing: A case study of digital signage-based online stores
Ganjar Alfian, Muhammad Fazal Ijaz, Muhammad Syafrudin, M Alex Syaekhoni, Norma Latif Fitriyani, and Jongtae Rhee. Customer behavior analysis using real-time data processing: A case study of digital signage-based online stores. Asia Pacific Journal of Marketing and Logistics ,...
2019
-
[13]
An architecture for the real-time data stream monitoring in iot
Mario José Diván and María Laura Sánchez Reynoso. An architecture for the real-time data stream monitoring in iot. Multimedia Big Data Computing for IoT Applications: Concepts, Paradigms and Solutions , pages 59–100, 2020
2020
-
[14]
Reversible sketches: enabling monitoring and analysis over high-speed data streams
Robert Schweller, Zhichun Li, Yan Chen, and etal. Reversible sketches: enabling monitoring and analysis over high-speed data streams. TON, 2007
2007
-
[15]
On- board vehicle data stream monitoring using minefleet and fast resource con- strained monitoring of correlation matrices
Hillol Kargupta, Vasundhara Puttagunta, Martin Klein, and Kakali Sarkar. On- board vehicle data stream monitoring using minefleet and fast resource con- strained monitoring of correlation matrices. New Generation Computing, 25:5–32, 2006. 13 Siyuan Dong, Yuxuan Tian, Wenhan Ma...
2006
-
[16]
An improved data stream summary: the count-min sketch and its applications
Graham Cormode and Shan Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms, 55(1):58–75, 2005
2005
-
[17]
New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice
Cristian Estan and George Varghese. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Transactions on Computer Systems (TOCS), 21(3):270–313, 2003
2003
-
[18]
Finding frequent items in data streams
Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. In International Colloquium on Automata, Languages, and Pro- gramming, pages 693–703. Springer, 2002
2002
-
[19]
Per-flow traffic measurement through randomized counter sharing
Tao Li, Shigang Chen, and Yibei Ling. Per-flow traffic measurement through randomized counter sharing. IEEE/ACM Transactions on Networking, 20(5):1622– 1634, 2012
2012
-
[20]
Sequential dependencies
Lukasz Golab, Howard Karloff, Flip Korn, Avishek Saha, and Divesh Srivastava. Sequential dependencies. Proceedings of the VLDB Endowment , 2(1):574–585, 2009
2009
-
[21]
Effective and complete discovery of order dependencies via set-based axiomatization
Jaroslaw Szlichta, Parke Godfrey, Lukasz Golab, Mehdi Kargar, and Divesh Sri- vastava. Effective and complete discovery of order dependencies via set-based axiomatization. arXiv preprint arXiv:1608.06169, 2016
2016 arXiv
-
[22]
Data auditor: Exploring data quality and semantics using pattern tableaux
Lukasz Golab, Howard Karloff, Flip Korn, and Divesh Srivastava. Data auditor: Exploring data quality and semantics using pattern tableaux. Proceedings of the VLDB Endowment, 3(1-2):1641–1644, 2010
2010
-
[23]
Issues in data stream management
Lukasz Golab and M Tamer Özsu. Issues in data stream management. ACM Sigmod Record, 32(2):5–14, 2003
2003
-
[24]
Update-pattern-aware modeling and process- ing of continuous queries
Lukasz Golab and M Tamer Özsu. Update-pattern-aware modeling and process- ing of continuous queries. In Proceedings of the 2005 ACM SIGMOD international conference on Management of data , pages 658–669, 2005
2005
-
[25]
New techniques for assessing audio and video quality in real-time interactive communications
Jim Mullin, Lucy Smallwood, Anna Watson, and Gillian Wilson. New techniques for assessing audio and video quality in real-time interactive communications. IHM-HCI Tutorial, pages 1–63, 2001
2001
-
[26]
A performance study of ethernet audio video bridging (avb) for industrial real-time communication
Jahanzaib Imtiaz, Jürgen Jasperneite, and Lixue Han. A performance study of ethernet audio video bridging (avb) for industrial real-time communication. In 2009 IEEE Conference on Emerging Technologies & Factory Automation , pages 1–8. IEEE, 2009
2009
-
[27]
A survey on steganography techniques in real time audio signals and evaluation.International Journal of Computer Science Issues (IJCSI) , 9(1):30, 2012
Abdulaleem Z Al-Othmani, Azizah Abdul Manaf, and Akram M Zeki. A survey on steganography techniques in real time audio signals and evaluation.International Journal of Computer Science Issues (IJCSI) , 9(1):30, 2012
2012
-
[28]
New techniques for assess- ing audio and video quality in real-time interactive communication
Anna Watson, J Mullin, L Smallwood, and G Wilson. New techniques for assess- ing audio and video quality in real-time interactive communication. Tutorial at IHM-HCI, Lille, France. http://citeseerx. ist. psu. edu/viewdoc/download , 2001
2001
-
[29]
Evaluating audio and video quality in low-cost multimedia conferencing systems
Anna Watson and Martina Angela Sasse. Evaluating audio and video quality in low-cost multimedia conferencing systems. Interacting with computers, 8(3):255– 275, 1996
1996
-
[30]
Audio and speech quality survey of the opus codec in web real-time commu- nication
Oliver Jokisch, Michael Maruschke, Martin Meszaros, and Viktor Iaroshenko. Audio and speech quality survey of the opus codec in web real-time commu- nication. In Proc. 27th Conference on Electronic Speech Signal Processing (ESSV) , volume 3, pages 254–62, 2016
2016
-
[31]
Gerardo Rubino. Quantifying the quality of audio and video transmissions over the internet: the psqa approach.Communication Networks And Computer Systems: A Tribute to Professor Erol Gelenbe , pages 235–250, 2006
2006
-
[32]
A study of real-time packet video quality using random neural networks
Samir Mohamed and Gerardo Rubino. A study of real-time packet video quality using random neural networks. IEEE transactions on circuits and systems for video technology, 12(12):1071–1083, 2002
2002
-
[33]
Detecting malicious packet dropping in the presence of collisions and channel errors in wireless ad hoc networks
Thaier Hayajneh, Prashant Krishnamurthy, David Tipper, and Taehoon Kim. Detecting malicious packet dropping in the presence of collisions and channel errors in wireless ad hoc networks. In 2009 IEEE International Conference on Communications, pages 1–6. IEEE, 2009
2009
-
[34]
A distributed protocol for detection of packet dropping attack in mobile ad hoc networks
Jaydip Sen, M Girish Chandra, P Balamuralidhar, SG Harihara, and Harish Reddy. A distributed protocol for detection of packet dropping attack in mobile ad hoc networks. In 2007 IEEE International Conference on Telecommunications and Malaysia International Conference on Communi...
2007
-
[35]
Resisting malicious packet drop- ping in wireless ad hoc networks
Mike Just, Evangelos Kranakis, and Tao Wan. Resisting malicious packet drop- ping in wireless ad hoc networks. In Ad-Hoc, Mobile, and Wireless Networks: Second International Conference, ADHOC-NOW2003, Montreal, Canada, October 8-10, 2003. Proceedings 2 , pages 151–163. Springer, 2003
2003
-
[36]
Design an anomaly based fuzzy intrusion detection system for packet dropping attack in mobile ad hoc networks
Alka Chaudhary, VN Tiwari, and Anil Kumar. Design an anomaly based fuzzy intrusion detection system for packet dropping attack in mobile ad hoc networks. In 2014 IEEE International Advance Computing Conference (IACC) , pages 256–261. IEEE, 2014
2014
-
[37]
A novel technique to detect malicious packet dropping attacks in wireless sensor networks
J Sebastian Terence and Geethanjali Purushothaman. A novel technique to detect malicious packet dropping attacks in wireless sensor networks. Journal of Information Processing Systems, 15(1):203–216, 2019
2019
-
[38]
Privacy-preserving and truthful detection of packet dropping attacks in wireless ad hoc networks
Tao Shu and Marwan Krunz. Privacy-preserving and truthful detection of packet dropping attacks in wireless ad hoc networks. IEEE Transactions on mobile computing, 14(4):813–828, 2014
2014
-
[39]
A provenance based mechanism to identify malicious packet dropping adversaries in sensor networks
Salmin Sultana, Elisa Bertino, and Mohamed Shehab. A provenance based mechanism to identify malicious packet dropping adversaries in sensor networks. In 2011 31st International Conference on Distributed Computing Systems Workshops, pages 332–338. IEEE, 2011
2011
-
[40]
An efficient cross-layer approach for malicious packet dropping detection in manets
Leovigildo S’nchez-Casado, Pedro García-Teodoro, et al. An efficient cross-layer approach for malicious packet dropping detection in manets. In 2012 IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communications, pages 231–238. IEEE, 2012
2012
-
[41]
A model of data forwarding in manets for light- weight detection of malicious packet dropping
Leovigildo Sánchez-Casado, Gabriel Maciá-Fernández, Pedro Garcia-Teodoro, and Roberto Magán-Carrión. A model of data forwarding in manets for light- weight detection of malicious packet dropping. Computer Networks, 87:44–58, 2015
2015
-
[42]
A stochastic approach for packet dropping attacks detection in mobile ad hoc networks
Mohammad Rmayti, Rida Khatoun, Youcef Begriche, Lyes Khoukhi, and Do- minique Gaiti. A stochastic approach for packet dropping attacks detection in mobile ad hoc networks. Computer Networks, 121:53–64, 2017
2017
-
[43]
Cross-layer detection of sinking behavior in wireless ad hoc networks using svm and fda
John Felix Charles Joseph, Bu-Sung Lee, Amitabha Das, and Boon-Chong Seet. Cross-layer detection of sinking behavior in wireless ad hoc networks using svm and fda. IEEE Transactions on Dependable and Secure Computing , 8(2):233–245, 2010
2010
-
[44]
Detecting blackhole attack on aodv-based mobile ad hoc networks by dynamic learning method
Satoshi Kurosawa, Hidehisa Nakayama, Nei Kato, Abbas Jamalipour, and Yoshiaki Nemoto. Detecting blackhole attack on aodv-based mobile ad hoc networks by dynamic learning method. Int. J. Netw. Secur., 5(3):338–346, 2007
2007
-
[45]
New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice
Cristian Estan and George Varghese. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Trans. Comput. Syst., 2003
2003
-
[46]
Elastic sketch: adaptive and fast network- wide measurements
Tong Yang, Jie Jiang, Peng Liu, and etal. Elastic sketch: adaptive and fast network- wide measurements. In SIGCOMM, 2018
2018
-
[47]
Heavykeeper: An accurate algorithm for finding top-𝑘 elephant flows
Tong Yang, Haowei Zhang, Jinyang Li, Junzhi Gong, Steve Uhlig, Shigang Chen, and Xiaoming Li. Heavykeeper: An accurate algorithm for finding top-𝑘 elephant flows. IEEE/ACM Transactions on Networking, 27(5):1845–1858, 2019
2019
-
[48]
https://catalog.caida.org/dataset/passive_ 2016_pcap, 2016
Anonymized Internet Traces 2016. https://catalog.caida.org/dataset/passive_ 2016_pcap, 2016
2016
-
[49]
http://mawi.wide.ad.jp/mawi/
MAWI Working Group Traffic Archive. http://mawi.wide.ad.jp/mawi/
-
[50]
https://www.netresec.com/?page= MACCDC
Capture files from Mid-Atlantic CCDC . https://www.netresec.com/?page= MACCDC
-
[51]
Network traffic charac- teristics of data centers in the wild
Theophilus Benson, Aditya Akella, and David A Maltz. Network traffic charac- teristics of data centers in the wild. In𝑃𝑟𝑜𝑐. IMC, pages 267–280, 2010
2010
-
[52]
http://burtleburtle.net/bob/hash/evahash.html
Hash website. http://burtleburtle.net/bob/hash/evahash.html
-
[53]
Cuckoo filter: Practically better than bloom
Bin Fan, Dave G Andersen, Michael Kaminsky, and Michael D Mitzenmacher. Cuckoo filter: Practically better than bloom. In Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies, pages 75–88, 2014
2014
-
[54]
Cuckoo hashing
Rasmus Pagh and Flemming Friche Rodler. Cuckoo hashing. Journal of Algo- rithms, 51(2):122–144, 2004. 14
2004
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.