{"id":"153a35c7-708f-45bc-b07a-498ba32f503d","arxiv_id":"2412.09057","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PhishIntel combines local blacklists, a result cache, and a reference-based detector in a two-tier queue to lower response latency for real-world phishing URL screening.","lead":"PhishIntel is a phishing detection system that sorts incoming URLs into fast (blacklist and cache) and slow (crawler and AI analysis) paths. A smart generalist would read it to see a practical blueprint for deploying content-based phishing detectors in real email systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Latency comparison may count 'pending' as a response: on cache/blacklist misses the fast-slow system returns a placeholder immediately, so the measured speedup may not reflect time-to-final-verdict and the zero-day detection claim lacks end-to-end accuracy data.","rationale":"The paper's central contribution is an engineering pattern: use a blacklist/cache fast path and defer RBPD analysis to a slow path. This is reasonable, and the two deployed applications show genuine systems effort, which I credit. My stress-test focuses on the evaluation in §3.3, because both parts of the headline claim depend on it. The fast-slow comparison only 'significantly reduces latency' if 'response' means the same thing in both arms. Section 2.3 explicitly says misses return 'pending' immediately. If the measured fast-slow latency is the time to that pending response, the comparison is structurally biased and the result says nothing about how quickly users obtain a final verdict on a zero-day URL. The paper also does not report the hit rate, so the reader's identified weakest assumption stands, but I think the pending-response ambiguity is more load-bearing because it can create the appearance of a speedup even when no final verdict is faster. Separately, the claim that RBPD detection capability is retained for zero-day URLs is asserted, not measured: no end-to-end precision/recall, no error bars, and no held-out zero-day set. The proposed concrete test would settle both issues with one re-run plus a held-out evaluation. Since these are addressable measurement gaps rather than flaws in the proposed architecture, the conditional verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":5893,"tokens_out":5088,"duration_ms":50861,"concrete_test":"Re-run the §3.3 evaluation on the same 2k URL set with logging that records for each URL: (i) fast-path hit or miss, (ii) time from submission to the FTW's first response (including 'pending'), and (iii) time from submission to the final verdict returned by the STW/cache update. Report the hit rate and compare average final-verdict latency of PhishIntel against the sequential pipeline separately for the miss-only subset. Also run a held-out set of freshly reported phishing URLs not present in PhishTank, Google Web Risk, or the cache, and report precision/recall and final-verdict latency. If the miss-only final-verdict latency is not reduced, or zero-day accuracy is not measured, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §3.3 is that the fast-slow task architecture significantly reduces average response latency compared to a sequential pipeline while retaining RBPD zero-day detection capability. The load-bearing condition is that both architectures are compared on the same notion of 'response'. Section 2.3 states that when a URL misses both the local blacklist and the result cache, 'the FTW returns a pending result' and queues the URL for the STW. The latency evaluation, however, never defines whether the fast-slow response time is the time to this pending status or the time to the final verdict from the STW. If it is the former, the comparison is invalid: for every miss, the fast-slow system 'responds' immediately with an inconclusive placeholder, while the sequential pipeline waits for a final decision. The measured reduction would then be tautological and would not imply that users get a faster phishing verdict. It also would not support the claim of retaining RBPD detection capability for zero-day URLs, since the URLs that require RBPD analysis are exactly the misses; no accuracy, hit rate, or miss latency is reported. The reader's hit-rate concern is real, but the pending-response issue is more fundamental: even with high hit rate, what is being reduced is the time to some response, not necessarily the time to a correct final answer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PhishIntel is a phishing detection system that combines a fast task worker (local blacklist and result cache) with a slow task worker (online blacklist verification, webpage crawling, and a reference-based phishing detector, KnowPhish). URLs that miss both the blacklist and cache are returned as 'pending' and queued for slow analysis; the authors claim this fast-slow architecture reduces response latency while retaining zero-day detection capability. The paper demonstrates two downstream applications—a phishing intelligence platform and an Outlook email plugin—and reports a performance evaluation comparing the fast-slow architecture with a sequential pipeline on 1,000 benign and 1,000 phishing URLs.","tokens_in":6196,"tokens_out":4436,"duration_ms":46404,"significance":"If the central claims were properly supported, the paper would make a modest but practical contribution: applying a standard cache-plus-queue pattern to reference-based phishing detection, with concrete deployment components and two working applications. The manuscript is transparent that the RBPD (KnowPhish) is prior published work, and no model parameters are fit here, so there is no definitional circularity in the evaluation. The use of named components (PhishTank, Google Web Risk, Redis, Celery, Playwright) makes the system description concrete and partially reproducible. However, the quantitative evidence for the main latency claim is currently invalid as presented, and the zero-day retention claim is not empirically evaluated; these issues are load-bearing for the paper's contribution.","major_comments":[{"comment":"The central latency comparison is invalid as reported because the two architectures are not measured to the same endpoint. Section 2.3 states that when a URL misses both the local blacklist and the result cache, the FTW returns a 'pending' result and the URL is queued for the STW; Section 3.1 confirms that such URLs are presented to the user as 'In Queue.' If the fast-slow response time in Fig. 3(a) is the time to issue this pending status, then for every miss the fast-slow system returns an inconclusive placeholder immediately, while the sequential pipeline waits for the crawler and RBPD to produce a final verdict. The measured reduction would then be tautological and would not establish that users receive faster phishing decisions. The paper must report the time to a final verdict for the fast-slow system (including STW processing and cache write-back), conditioned on cache/blacklist hit or miss, and compare both architectures on this common endpoint.","section":"§3.3, Fig. 3(a), with §2.3"},{"comment":"The claim that the fast-slow architecture 'retains the robust detection capabilities of RBPDs for zero-day phishing threats' is not supported by any accuracy measurement. The evaluation only reports response latency and the blacklist-vs-RBPD proportion of phishing reports; it does not report detection accuracy of the full system, cache/blacklist hit rate, or agreement between pending-then-final results and ground truth. Zero-day URLs are exactly the URLs that miss the blacklist and cache, so an end-to-end accuracy result conditional on that miss set is needed before the retention claim can be accepted.","section":"§3.3"},{"comment":"The statement that the fast-slow architecture 'significantly reduces system latency' is not supported statistically. The evaluation uses a single sample of 1,000 benign and 1,000 phishing URLs with no reported variance, confidence interval, or significance test, and Fig. 3(a) appears to contain one measurement per condition. Repeated trials or bootstrap intervals are needed to support the word 'significantly.'","section":"§3.3"}],"minor_comments":[{"comment":"The axis labels and numeric values in Fig. 3 are difficult to read from the text; please report the exact mean response times, the number of trials, and the blacklist/RBPD percentages in the caption or body so the figure is self-contained.","section":"§3.3, Fig. 3"},{"comment":"The term 'pending result' is ambiguous: clarify whether the API returns a provisional status to the caller and how the Outlook plugin handles a pending URL (e.g., whether it waits, defers, or notifies the user later).","section":"§2.3"},{"comment":"The phishing intelligence platform screenshot shows 21 URLs with some labeled 'In Queue'; please state how the platform updates these entries once the STW completes, since dashboards that never resolve pending URLs would limit practical utility.","section":"§3.1"},{"comment":"The system uses both a local PhishTank blacklist and Google Web Risk as an online blacklist; please clarify whether the local blacklist is a subset of the online blacklist and how the synchronization period affects the reported results.","section":"§2.5"},{"comment":"The user-feedback mechanism is described only briefly; please specify how false reports are validated before being inserted into the cache, since unvalidated feedback could poison the cache for all future queries.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the pending-response issue is not merely a missing hit-rate statistic but a mismatch in the outcome being measured. For a four-page companion paper, the system demonstration is within scope, but the latency comparison and the zero-day retention claim must be reworked before publication. If the authors can report time-to-final-verdict and conditional hit-rate/accuracy, the paper could become acceptable as a systems demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a well-put-together systems paper that applies a standard two-tier caching pattern to reference-based phishing detection, and it gives you two working applications. The architecture is sensible. But the evaluation does not actually support the headline claims that it 'retains robust detection capabilities' and 'significantly reduces latency' in a meaningful way.\n\nThe genuinely new part is applying the fast-slow task queue to RBPDs and building the two demos. The system description is clear: FTW checks local blacklist and Redis cache, STW uses a Celery queue, Google Web Risk, a Playwright crawler, and KnowPhish. That is a real integration effort. The paper is honest that this is a demonstration, and it does not oversell the novelty in the text.\n\nThe soft spot is Section 3.3. The comparison between fast-slow and sequential reports average response time, but never defines what counts as a response. Section 2.3 says that on a blacklist/cache miss, the FTW returns a pending result and queues the URL. If the measured fast-slow response time is the time to that pending status, then the speedup is partly tautology: you are comparing a placeholder to a final verdict. For the URLs that actually require RBPD analysis—the misses—the user still waits for the STW. The paper reports no end-to-end accuracy, no hit rate for blacklist or cache, and no error bars. The blacklist filtering figure (8.95% vs 1.05%) is also ambiguous without a clear denominator. These gaps do not sink the architecture, but they do sink the latency and zero-day claims as stated.\n\nI would send this to review with the expectation of a major revision: define response time precisely, report cache/blacklist hit rates, and include an end-to-end accuracy check on a labeled sample. The stress-test concern about pending-response is legit; the authors should address it directly. For a 4-page companion, the engineering is solid, but the evidence is too thin to call it deployment-ready.","headline":"A clean engineering demonstration of a fast-slow architecture for RBPDs, but the latency comparison may count 'pending' as a response and the zero-day claim is unmeasured.","tokens_in":6667,"tokens_out":1905,"would_cite":false,"duration_ms":19009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PhishIntel claims a fast-slow task split makes reference-based phishing detection fast enough to deploy, cutting average response time while still analyzing novel URLs.","keywords":["phishing detection","reference-based phishing detection","fast-slow task architecture","URL blacklist","result cache","zero-day phishing","email security","web security"],"falsifier":"Take a batch of URLs drawn from a real enterprise email stream, record for each URL whether it hit the local blacklist or result cache, and compute the average response time split by hit versus miss; if the hit rate is near zero, the average latency will be close to the sequential pipeline's latency, which would falsify the claim that the architecture significantly reduces response time for that workload.","tokens_in":5643,"feed_emoji":"🛡️","tokens_out":7232,"duration_ms":69934,"temperature":0.7,"pith_summary":"Reference-based phishing detectors tell phishing pages from legitimate ones by checking whether a page's apparent brand matches its real domain, but they are too slow for live email scanning because they must crawl and analyze each page. PhishIntel's proposal is to put those detectors behind a fast-slow task architecture: known URLs are answered instantly from a local blacklist and result cache, while everything else waits in a queue for online blacklist checks, crawling, and full reference-based analysis, after which the answer is cached. The paper reports that this design cuts average response time relative to a sequential pipeline and that blacklist filtering removes a substantial share of traffic. The two demonstrations, a URL intelligence dashboard and an email plugin, are meant to show the system fits into real workflows rather than remaining a benchmark.","feed_headline":"Fast-slow split cuts phishing detection time","feed_subtitle":"Known URLs get instant verdicts from blacklist and cache; novel URLs still get full page analysis.","key_machinery":"The load-bearing mechanism is the fast-slow task architecture, in which a Fast Task Worker (local blacklist plus result cache) answers known URLs with runtime-efficient matching operations, and a Slow Task Worker (online blacklist checker, webpage crawler, and reference-based detector) handles everything else through a queue. The result cache is the hinge: it lets a single slow analysis serve many future requests, so average latency is dominated by cheap cache and blacklist hits while novel URLs still pay the full crawl-and-analyze cost. The system is deployed with multiple fast and slow workers, and user feedback can correct cached results.","core_discovery":"On the paper's own terms, the central discovery is that the bottleneck for deploying accurate phishing detectors is response latency, not accuracy, and that a two-tier processing design can remove that bottleneck without giving up zero-day coverage. The fast task worker checks incoming URLs against a local blacklist and a result cache; URLs that match return a verdict immediately, while URLs that miss both are handed to a slow task worker that queries an online blacklist, crawls the webpage, and runs a reference-based phishing detector. Once the slow worker produces a result, it is written back to the cache so subsequent identical URLs skip the expensive path. The paper's evaluation, on 1,000 benign and 1,000 phishing URLs, shows the fast-slow architecture has lower average response time than a sequential pipeline, and its blacklist layer filters out a substantial portion of phishing reports.","pith_inferences":["The reported average-latency advantage depends on the mix of incoming URLs: if real enterprise traffic is dominated by one-off, never-seen URLs, the fast path rarely triggers and end-to-end latency approaches that of the sequential pipeline; publishing the cache and blacklist hit rate on a realistic traffic sample would settle this.","Caching creates a freshness risk the paper does not address: a page that changes from benign to phishing after its first analysis will keep returning the old cached verdict, so a cache-expiry or scheduled re-analysis policy would be a natural extension.","The architecture is detector-agnostic, so swapping the reference-based detector for a faster or cheaper model would reduce slow-path cost and raise throughput without redesigning the queue structure.","The same task-splitting idea could apply to other slow content-based detectors, not just phishing URL classification, whenever a cheap lookup can answer most requests and an expensive analysis is reserved for the remainder."],"forward_implications":["URLs on the local blacklist or already present in the result cache receive near-instant verdicts, which makes live email scanning feasible for users who would otherwise wait tens of seconds.","Novel URLs still flow to the slow task worker, so the system does not trade zero-day phishing detection for speed; the reference-based detector remains on the critical path only for URLs no cache or blacklist can resolve.","Because every slow-path verdict is cached, repeated sightings of the same phishing URL become progressively cheaper, which matches the pattern seen when a phishing campaign sends many identical links.","The same backend can power different front ends, such as a URL-analysis dashboard and an email plugin, because both only need to submit URL lists to the fast and slow workers."],"supporting_citations":[{"why":"Supplies the reference-based detector used as the paper's slow-path analysis engine and the basis for the stated analysis-time cost.","marker":"[4]"},{"why":"Supplies the empirical basis for the claim that a state-of-the-art reference-based detector can spend about 10 seconds analyzing page content.","marker":"[8]"},{"why":"Represents the visual reference-based detection approach that motivates keeping full page analysis in the slow path.","marker":"[5]"},{"why":"Represents the deep-vision reference-based detection line that the system is designed to host without changing its detection logic.","marker":"[7]"},{"why":"Shows a recent extension of reference-based detectors that does not require a pre-defined reference list, indicating the class of detectors the architecture can wrap.","marker":"[6]"}],"fun_headline_variants":["Fast-slow split speeds phishing detection","PhishIntel: Low-latency phishing via task split","Instant verdicts for known URLs, deep checks for novel ones","Two-tier detection cuts phishing response time","PhishIntel: Fast known, slow novel, zero-day safe"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The latency advantage rests on the assumption that most incoming URLs will be found in the local blacklist or the result cache; any URL that misses both pays the full web-crawl and reference-based analysis cost, making the system no faster than the sequential pipeline it is compared with.","fun_headline_variants_meta":{"raw":{"variants":["Fast-slow split speeds phishing detection","PhishIntel: Low-latency phishing via task split","Instant verdicts for known URLs, deep checks for novel ones","Two-tier detection cuts phishing response time","PhishIntel: Fast known, slow novel, zero-day safe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1438,"prompt_tokens":881,"completion_tokens":557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":497,"tokens_out":557,"duration_ms":5506,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:06.196600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a batch of URLs drawn from a real enterprise email stream, record for each URL whether it hit the local blacklist or result cache, and compute the average response time split by hit versus miss; if the hit rate is near zero, the average latency will be close to the sequential pipeline's latency, which would falsify the claim that the architecture significantly reduces response time for that workload.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reference-based detector used as the paper's slow-path analysis engine and the basis for the stated analysis-time cost."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the empirical basis for the claim that a state-of-the-art reference-based detector can spend about 10 seconds analyzing page content."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the visual reference-based detection approach that motivates keeping full page analysis in the slow path."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the deep-vision reference-based detection line that the system is designed to host without changing its detection logic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows a recent extension of reference-based detectors that does not require a pre-defined reference list, indicating the class of detectors the architecture can wrap."}],"review_version":1}