{"id":"502c500e-5b93-4d3b-9e04-a0d8816a39b1","arxiv_id":"2502.07815","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A proposed hybrid pipeline combining RE2 regex, Aho-Corasick exact matching, and AI named entity recognition reportedly detects PII and PHI with a 91.6 percent F1 score.","lead":"This paper benchmarks regex engines and exact-match algorithms for detecting personal data, then wraps them into a hybrid AI-plus-regex system called CHPDA. It reports that RE2 and Aho-Corasick give the best speed and accuracy, and that combining regex with AI raises the F1 score to 91.6 percent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Table 3 contradicts the headline 100 MB/s throughput claim, undermining the central benchmark conclusion.","rationale":"The reader's rejection is based on missing evaluation methodology and irreproducible benchmark numbers. My stress-test identifies a sharper, internal problem: the paper's own Table 3 contradicts its stated throughput of 100 MB/s by roughly a factor of five. This is not an external criticism about missing error bars; it is a quantitative inconsistency inside the manuscript. The speed claim is a load-bearing part of the central claim because the paper repeatedly contrasts RE2's speed against PCRE and Hyperscan and reports 100 MB/s in Section 4. If Table 3 is the actual processing time, then either the 100 MB/s claim refers to a different, unspecified measurement or the table is not representative; either way the benchmark conclusions as presented are not reliable. I also note that the F1 arithmetic in Table 4 is almost consistent (the reported 91.6 is correct for 94.8/88.7), but Table 5's counts imply different operating points, and the absence of a dataset definition remains a separate handicap. The verdict remains REJECT because the central claim, as evidenced, is contradicted internally; a CONDITIONAL could follow only if the authors release the benchmark code, dataset, pattern set, and clarify the speed definition.","tokens_in":10060,"tokens_out":2874,"duration_ms":27475,"concrete_test":"Recompute throughput for every row of Table 3 by dividing dataset size by reported processing time (e.g., 100 MB / 4.85 s = 20.6 MB/s; 1 GB / 42.92 s = 23.3 MB/s; 10 GB / 530.87 s = 19.7 MB/s) and compare with the Section 4 claim of 100 MB/s. If the values do not match within a stated definition of 'processing speed,' the central speed claim is unsupported by the paper's own data.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim includes a processing speed of roughly 100 MB/s and RE2 at 10–15 ms/MB. Section 4 reports an 'average processing speed of 100 MB/s.' However, Table 3 (Section 5.3) gives end-to-end processing times for regex sets: 100 MB in 4.85 s implies 20.6 MB/s; 1 GB in 42.92 s implies 23.3 MB/s; 10 GB in 530.87 s implies 19.7 MB/s. These are about a factor of five lower than the claimed 100 MB/s. If Table 3 is the actual measured processing time, then the speed pillar of the central claim is internally contradicted. No methodology (e.g., 'peak regex-only throughput' vs. 'folder classification time') is given to reconcile the discrepancy. The F1 claim (91.6%) also rests on Table 4, but Table 5's per-file-size false-positive and missed-match counts imply different precision/recall values than Table 4, and no ground-truth corpus or pattern set is disclosed. The most load-bearing assumption—that Tables 1–5 come from a consistent, controlled benchmark—is therefore not merely unverified but appears false on its face for speed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid PII/PHI detection pipeline, CHPDA, that combines Google RE2 regular-expression scanning, Aho-Corasick exact keyword matching, AI-based named entity recognition, and validation functions such as Luhn's algorithm, followed by a proximity-based confidence-score threshold. The central claims are that RE2 is the best regex engine (10-15 ms/MB, 8-16 MB, 99.5% accuracy), Aho-Corasick is the best exact matcher (8 ms/MB), and the hybrid system attains a 91.6% F1 score with false positives below 3% and an average processing speed around 100 MB/s across file formats up to 1 TB. The paper presents benchmark tables and device measurements to support these claims.","tokens_in":10240,"tokens_out":7031,"duration_ms":63455,"significance":"If the headline results were reproducible, CHPDA would be a practically useful architecture for data-loss-prevention tooling, and the comparison of RE2, PCRE, Hyperscan, and exact-match algorithms is a relevant engineering question. The manuscript deserves credit for presenting the algorithm as pseudocode in Section 3.6.1, giving explicit example walkthroughs, and candidly listing limitations such as limited multilingual support and the need for pattern updates. However, the evidence supplied is not sufficient to establish the claims: there is no dataset, no pattern list, no software or hardware versioning, no repeated trials, and the numbers in the paper's own tables are mutually inconsistent. The significance is therefore conditional until a controlled, reproducible evaluation is provided.","major_comments":[{"comment":"The headline speed claim of an 'average processing speed of 100 MB/s' (Section 4) is contradicted by Table 3. The reported end-to-end times for the 100-pattern regex set imply 100 MB / 4.85 s = 20.6 MB/s, 1 GB / 42.92 s = 23.3 MB/s, and 10 GB / 530.87 s = 19.7 MB/s, roughly a factor of five lower. No methodology is given to reconcile this discrepancy, so the speed pillar of the central claim is internally inconsistent.","section":"Section 4 and Table 3"},{"comment":"The F1 claim is not supported by the paper's own detection table. Using the standard definitions and Table 5 (100 MB row: total matches 102, false positives 2, missed 2), true positives = 100 and actual positives = 102, giving precision = recall = 98.0% and F1 ≈ 98.0%, not the 94.8% / 88.7% / 91.6% reported in Table 4. The 500, 1000, and 5100 MB rows similarly yield precision and recall near 98-99% under this reading. If 'Total Patterns' is instead the ground truth, the 'Missed Matches' column is contradictory. In either case, the paper does not define what total patterns or total matches mean relative to a ground-truth corpus, so the reported F1 value is not reproducible.","section":"Section 5.4, Tables 4 and 5"},{"comment":"The evaluation lacks an experimental protocol. No ground-truth corpus is described or cited, no regex pattern list is given, no software versions or hardware specifications beyond coarse classes are reported, and every table reports single-point numbers without repetitions, confidence intervals, or statistical tests. 'Accuracy' is never formally defined, and the claimed 'over 95% accuracy' and '<3% false positive rate' cannot be checked. Without this information, the benchmarks in Tables 1-5 are not reproducible and cannot support the comparative conclusions.","section":"Section 4 and Section 5"},{"comment":"The memory-efficiency claim for RE2 is difficult to reconcile with the system memory measurements. The abstract and Table 1 report RE2 memory usage of 8-16 MB, while Table 7 reports pipeline memory usage of 115-190 MB for regex sets of 100-200 patterns. If the former is library-only and the latter is the full CHPDA process, that distinction is never stated; if they are meant to be comparable, they contradict. The same holds for the CPU-usage percentages in Table 6, for which no measurement methodology is given.","section":"Abstract, Table 1, Section 5.4.2, Table 7"},{"comment":"The confidence-scoring example validates an SSN-like pattern '123-45-6789' with Luhn's algorithm, but Luhn's algorithm is a checksum for credit-card numbers, not for SSNs, and the example asserts without demonstration that the pattern passes. The algorithm's free parameters (alpha, Dmax, and Validation_Score in the proximity formula, plus threshold T in Section 3.7) are set by hand with no calibration or sensitivity analysis, leaving the relationship between the scoring mechanism and the reported accuracy unexplained.","section":"Section 3.6.2"}],"minor_comments":[{"comment":"The abstract contains a typographical spacing error ('91. 6%'), and the text refers to 'Table 1.3' and 'Table 1.4' in Sections 5.1 and 5.2, which should be Table 1 and Table 2, respectively.","section":"Abstract and Sections 5.1-5.2"},{"comment":"Figures 4 and 5 are referenced and captioned, but no actual plot images are included in the submitted text, so the claimed linear scaling trend and the match-count trend cannot be inspected.","section":"Figures 4 and 5"},{"comment":"Several references are incomplete or inconsistently formatted; for example, reference [20] is only 'Xiang Wang. Introduction to Hyperscan' with no venue or year, and some citations in the introduction lack closing brackets or page ranges.","section":"References"},{"comment":"The paper would benefit from a data-availability statement; no code, datasets, benchmark scripts, or raw measurement logs are provided, which further limits reproducibility.","section":"Overall"}],"recommendation":"reject","confidential_remarks":"The core benchmark claims are not merely missing supporting detail; they are contradicted by the paper's own tables, and the absence of any dataset or protocol means a revision would require a completely new empirical evaluation rather than targeted corrections. I therefore recommend reject rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an engineering report with a plausible pipeline and a description of a hybrid detection approach, but the central numeric claims don't survive contact with the paper's own tables. The abstract's 100 MB/s and the memory figures are inconsistent with the timings in Table 3 and the memory in Table 7. The F1 claim in Table 4 also doesn't match the false-positive and missed-match counts in Table 5. These are not minor typos; they undermine the main results.\n\nWhat's good: The explanation of why RE2 with its SET mechanism is a reasonable choice for multi-pattern regex matching is accurate and clear. The Aho-Corasick primer is correct, and the decision to combine regex with NER and validation functions is a sensible engineering direction. The proximity-confidence scoring idea, while simple, is a concrete way to weight keyword context and validation, and it's the one piece that could be described as a small contribution. The paper also correctly notes that Hyperscan is fast but resource-hungry, and PCRE's backtracking is a problem for untrusted input.\n\nWhere it falls down: There is no dataset, no pattern list, no model checkpoint, no code, no hardware details beyond vague classes, and no error bars. Section 4's 'average processing speed of 100 MB/s' is directly contradicted by Table 3, which gives 4.85 seconds for 100 MB (about 20 MB/s) and 530 seconds for 10 GB (about 19 MB/s). The RAM claim in the abstract (8-16 MB for RE2) is a different quantity from the end-to-end memory in Table 7 (115-190 MB), and the paper never reconciles the two. Similarly, Table 4 reports 94.8% precision and 88.7% recall for AI+Regex, but Table 5's counts at 100 MB (102 matches, 2 false positives, 2 missed) imply precision and recall around 98%. Those numbers are not consistent with a single benchmark run. The paper also cites 'Table 1.3' and 'Table 1.4' when it means Tables 1 and 2, and there are several broken sentences. These problems are exactly the kind that make a referee suspect the results are assembled rather than measured.\n\nWho it's for: A practitioner who wants a qualitative sketch of a hybrid PII detection pipeline could get something from the architecture description. But as a research submission, it's not reproducible and its headline numbers are unreliable. It would need a complete evaluation package and a reconciliation of the tables to be worth serious review. As submitted, I'd desk-reject it with a clear list of what's missing.","headline":"Central benchmark claims contradict the paper's own tables; the architecture is plausible but the evidence is not.","tokens_in":10858,"tokens_out":3806,"would_cite":false,"duration_ms":35009,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a hybrid pipeline combining Google RE2 regex matching, Aho-Corasick exact search, AI named entity recognition, and validation functions detects PII and PHI with 94–95% accuracy, a 91.6% F1 score, false positives…","keywords":["PII detection","PHI detection","Google RE2","Aho-Corasick algorithm","named entity recognition","hybrid AI and regex","data loss prevention","confidence scoring"],"falsifier":"Re-run CHPDA on a publicly available labeled corpus of documents containing PII and PHI with the same pattern sets and measure F1 and false-positive rate; if the F1 is substantially below 91.6% or false positives exceed 3%, the central claim is not reproducible.","tokens_in":9785,"feed_emoji":"🛡️","tokens_out":6125,"duration_ms":51246,"temperature":0.7,"pith_summary":"This paper sets out to show that sensitive data detection is best done by a hybrid: regular expressions for well-defined formats, an exact-match automaton for keywords, and AI named entity recognition for context, all coordinated by a confidence-scoring step. The authors claim their Context-Aware Hybrid Pattern Detection Algorithm (CHPDA) reaches about 94–95% accuracy on PII and PHI, an F1 score of 91.6%, false positives below 3%, and processing speeds near 100 MB/s on files up to 1 TB. A sympathetic reader would care because real-time data-loss prevention and compliance with GDPR, HIPAA, and CCPA depend on fast, low-false-positive detection in unstructured enterprise data. The paper also argues that Google RE2 is the best regex engine for this purpose and that Aho-Corasick is the best exact match algorithm, based on its own benchmarks.","feed_headline":"Hybrid regex and AI detector scores 91.6% F1 on PII","feed_subtitle":"CHPDA pairs RE2 and Aho-Corasick with NER to cut false positives below 3 percent at 100 MB/s.","key_machinery":"The load-bearing mechanism is the CHPDA pipeline: Google RE2's SET mechanism compiles many regex patterns into one deterministic finite automaton that scans text in a single linear pass; Aho-Corasick builds a trie with failure links to match a dictionary of context keywords simultaneously; a proximity scoring formula assigns a confidence $C_{\\text{total}} = \\text{Proximity\\_Score}(d) + \\text{Validation\\_Score}$ based on the distance $d$ between a keyword and a detected pattern, with $\\text{Proximity\\_Score}(d) = \\max(0, \\alpha(D_{\\max} - d))$; and a user-set threshold $T$ keeps only matches with $C \\geq T$. Validation functions such as Luhn's algorithm for credit card numbers provide the validation score, which is what lowers false positives.","core_discovery":"The central claim is that combining rule-based and AI-based detection in one pipeline outperforms either approach alone: regex alone gives precision 92.5% but recall 75.3%, AI alone gives recall 89.2% but precision 84.7%, while the AI + Regex hybrid reaches precision 94.8% and recall 88.7%, yielding an F1 of 91.6% across dataset sizes from 100 MB to 5 GB. The paper further claims this hybrid maintains a stable confidence score and a false-positive rate below 3% at processing speeds around 100 MB/s on ordinary hardware.","pith_inferences":["The proximity-scoring mechanism is portable: the same $C_{\\text{total}}$ formula could be applied to any context-sensitive detection task, such as finding credentials in logs or sensitive clauses in contracts, not just PII/PHI.","Extending the NER stage with multilingual models is a natural next step: the paper identifies missing multilingual support as a limitation, and modern multilingual NER would likely close that gap without changing the pipeline.","The reported linear scaling could be stress-tested by adding exponentially larger pattern sets; the paper only tests up to 172 patterns, so the claim of linearity is an extrapolation beyond the measured range."],"forward_implications":["If the benchmark numbers hold, a single commodity server can scan a 1 TB corpus for PII/PHI in about three hours, making real-time DLP feasible for mid-size organizations.","The threshold $T$ gives operators a tunable precision-recall dial: raising it cuts false positives, lowering it catches more edge-case patterns.","The reported linear scaling of regex processing time with dataset size and pattern count means capacity planning can be a simple multiplicative estimate.","Because RE2 is preferred over Hyperscan for hardware compatibility, the pipeline can run on ARM-based lightweight agents and shared Windows environments, not just dedicated servers."],"supporting_citations":[{"why":"Source for the claim that Google RE2 is a principled, linear-time regex engine; underpins the choice of RE2 as the regex backbone.","marker":"[17]"},{"why":"Original Aho-Corasick string matching algorithm; the paper builds its exact-match stage on this.","marker":"[21]"},{"why":"Benchmark source for Hyperscan's speed and hardware constraints, used as the comparison that RE2 beats on portability.","marker":"[16]"},{"why":"Prior hybrid showing regex as a pre-filter before an AI model, which the paper extends with validation scoring.","marker":"[14]"},{"why":"Prior work combining regex with deep learning for SSN detection, supporting the hybrid approach's feasibility.","marker":"[15]"},{"why":"spaCy NER as one of the AI models the hybrid stage can use, cited for strong recall in entity detection.","marker":"[12]"},{"why":"BERT as a deep-learning NER baseline, cited for recall in detecting complex entities.","marker":"[11]"}],"fun_headline_variants":["AI + regex beats either alone: 91.6% F1 on PII","Hybrid AI+regex detector: 91.6% F1 on PII","Combine regex and NER to score 91.6% F1 on sensitive data","At 100 MB/s, hybrid AI+regex hits 91.6% F1 on PII","Fast PII detection: hybrid AI+regex hits 91.6% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation numbers assume a controlled, representative test corpus with a fixed ground-truth set of PII/PHI labels, a published list of regex patterns, and repeated runs; the paper does not provide these details, so the reported 94–95% accuracy and 91.6% F1 could be artifacts of an ad hoc test.","fun_headline_variants_meta":{"raw":{"variants":["AI + regex beats either alone: 91.6% F1 on PII","Hybrid AI+regex detector: 91.6% F1 on PII","Combine regex and NER to score 91.6% F1 on sensitive data","At 100 MB/s, hybrid AI+regex hits 91.6% F1 on PII","Fast PII detection: hybrid AI+regex hits 91.6% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00126,"raw_usage":{"total_tokens":5143,"prompt_tokens":908,"completion_tokens":4235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":4120}},"tokens_in":524,"tokens_out":4235,"duration_ms":25230,"temperature":1.0,"reasoning_tokens":4120,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:55:52.879996+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run CHPDA on a publicly available labeled corpus of documents containing PII and PHI with the same pattern sets and measure F1 and false-positive rate; if the F1 is substantially below 91.6% or false positives exceed 3%, the central claim is not reproducible.","supporting_citations":[{"cited_title":"RE2: a principled approach to regular expression matching","cited_arxiv_id":null,"evidence_quote":"Source for the claim that Google RE2 is a principled, linear-time regex engine; underpins the choice of RE2 as the regex backbone."},{"cited_title":"Efficient string matching: an aid to bibliographic searchCommunications of the ACM, vol","cited_arxiv_id":null,"evidence_quote":"Original Aho-Corasick string matching algorithm; the paper builds its exact-match stage on this."},{"cited_title":"Hyperscan: A fast multi-pattern regex matcher for modern {CPUs}","cited_arxiv_id":null,"evidence_quote":"Benchmark source for Hyperscan's speed and hardware constraints, used as the comparison that RE2 beats on portability."},{"cited_title":"Combining Regular Expressions and Machine Learning for SQL Injection Detection in Urban Computing","cited_arxiv_id":null,"evidence_quote":"Prior hybrid showing regex as a pre-filter before an AI model, which the paper extends with validation scoring."},{"cited_title":"Analyzing the Efficacy of Microsoft Presidio in Identifying Social Security Numbers in Unstructured Text","cited_arxiv_id":null,"evidence_quote":"Prior work combining regex with deep learning for SSN detection, supporting the hybrid approach's feasibility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"spaCy NER as one of the AI models the hybrid stage can use, cited for strong recall in entity detection."},{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"BERT as a deep-learning NER baseline, cited for recall in detecting complex entities."}],"review_version":1}