Pith. sign in

REVIEW 4 major objections 4 minor 12 references

Client-Side Zero-Shot LLM Inference for Comprehensive In-Browser URL Analysis

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully in-browser LLM pipeline classifies malicious URLs at 92% accuracy while keeping page data private.

desk verdict A plausible client-side triage architecture with a thin, cloaking-vulnerable evaluation; worth refereeing as major revision. read the letter →

arxiv 2506.03656 v1 pith:IJ4RHHZG submitted 2025-06-04 cs.CR

Avihay Cohen This is my paper
classification cs.CR
keywords client-sideLLMinferencezero-shotclassificationin-browsersandboxphishingdetectionmaliciousURLJavaScriptstaticanalysisdynamicbehaviormonitoringprivacy-preservingsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that a comprehensive malicious-URL analysis pipeline can run entirely in the user's browser using a small quantized LLM, with no page content sent to a cloud service. The system statically parses the page's JavaScript into AST features, executes the page in a sandboxed iframe while logging DOM mutations, API calls, and network requests, then feeds a structured summary of this evidence to a zero-shot local LLM that returns a verdict and a plain-language explanation. On a balanced 200-URL test set, the best in-browser model, a 4-bit quantized 8B LLaMA model, reaches 92% accuracy and 94% recall, close to a cloud reference model at 93% accuracy and 97% recall, and even the 2B model outperforms a classical Random Forest baseline on the malicious class. If this holds, users could get on-device, explainable phishing and malware detection without training data, cloud APIs, or sending sensitive page contents off the machine.

What carries the argument

The engine of the system is the evidence-to-prompt bridge. A prompt builder compresses three observation channels—static JavaScript features parsed with Acorn, dynamic behavior captured by monkey-patched fetch, XHR, eval, and DOM hooks inside a sandboxed iframe, and extracted visible text—into a structured JSON-instructed prompt; a zero-shot LLM running through an in-browser WebGPU inference engine then answers with a risk verdict, confidence, and explanation. The sandbox's stealth is what makes the dynamic channel faithful to a real user's browsing context, and the LLM's pre-trained knowledge is what allows a small model to fuse those channels without fine-tuning.

What would settle it

Re-run known cloaking pages twice: once under the full instrumented sandbox and once in a clean, uninstrumented browser with the same user agent and IP; if a meaningful share of them show different content, different outgoing requests, or delay their payload until after user interaction, the recorded evidence is not what a real visitor sees and the saved-page accuracy would not hold for live attacks.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that zero-shot reasoning by a compact, quantized LLM running inside the browser can serve as the decision engine for web threat analysis. Given rich evidence from three sources—static AST summaries of JavaScript, a logged runtime trace from an instrumented iframe, and the page's visible text—an 8B model classifies a held-out set of 100 malicious and 100 benign saved pages with 92% accuracy and 94% recall, within a few points of a cloud reference model, without any task-specific training. The paper concludes that client-side inference is a feasible and effective solution for URL analysis, and that the main remaining gap between local and cloud models is a matter of model knowledge and size, not of approach.

Load-bearing premise

The system's accuracy depends on the analyzed page not being able to tell that it is running inside a watched, instrumented iframe, so the page behaves exactly as it would for a real visitor.

Editorial extensions

If this is right

  • A compact 8B model running locally reaches 92% accuracy and 94% recall on the test set, so client-side inference is a realistic alternative to cloud APIs for URL threat detection.
  • Because the classifier is zero-shot, the system can be pointed at new phishing campaigns without collecting labeled data or retraining.
  • Page content, code traces, and network behavior never leave the device, so sensitive or logged-in pages can be scanned without third-party exposure.
  • Each user's browser acts as its own scanner, removing a fixed cloud IP range that attackers can fingerprint and avoid.
  • The LLM output includes an explanation and risk breakdown, so users and analysts see why a page was flagged, not just a score.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sandbox's stealth assumption holds, the natural deployment is a browser extension with a small always-on model that vets every navigation locally; the paper sketches this use case but does not implement or measure it.
  • A direct testable extension is to measure how often malicious pages can mislead the LLM by embedding prompt-like text or decoy content; the paper argues this risk is low because untrusted text is not treated as instructions, but no adversarial-content experiment is reported.
  • The paper's privacy and cost advantage could be quantified by comparing one-time download and on-device inference of a quantized model against the cumulative API cost of sending each page to a cloud model over months of browsing.
  • Because the evaluation uses saved copies of transient phishing pages, the reported transfer to live attacks is untested; re-running the pipeline on live, freshly registered phishing URLs would tighten that connection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a client-side web URL analysis system that combines static JavaScript AST analysis, a dynamic sandbox executing the page in an iframe, and zero-shot inference from a compact LLM running in the browser via WebLLM. The system produces a malicious/benign verdict together with a textual explanation. The authors evaluate four local models (LLaMA-3.1 8B, LLaMA-3.2 3B, Phi-3 3.8B, Gemma 2B) on a self-curated set of 200 URLs (100 malicious, 100 benign) whose page copies were saved and hosted locally, and they compare against a Random Forest baseline and a cloud GPT reference. The best local model reaches 92% accuracy and 94% recall, close to the reported cloud reference, and the paper argues that this demonstrates the feasibility and privacy benefits of fully client-side LLM-based threat analysis.

Significance. The paper addresses a timely and practically important problem: privacy-preserving, client-side detection of malicious webpages. The idea of having a compact LLM reason over combined static and dynamic evidence is attractive, and the detailed architecture description with concrete prompt templates and instrumentation examples makes the approach reproducible in principle. The explicit privacy argument and the comparison of several model sizes also provide useful engineering guidance. If the detection rates are robust against live adversarial pages, the work would meaningfully support the case for on-device security analysis. However, the significance is contingent on the evaluation being representative of real-world adversarial conditions; as it stands, the experimental evidence is suggestive rather than conclusive.

major comments (4)
  1. [Sections 5.1 and 5.3, Table 1] The central effectiveness claim is tested only on 200 self-curated URLs whose page copies were saved and hosted locally. Because the dynamic sandbox loads these saved copies in an iframe, the evaluation cannot exercise client-side cloaking or environment-detection checks, which the paper itself identifies as a core threat in Section 2.2 (citing CrawlPhish). The abstract's claim that the system is an 'effective solution' that works while 'eliminating the need to send potentially sensitive data to cloud services' therefore rests on the unverified assumption that the sandboxed iframe reproduces a real user's browsing context. Please provide either live-URL experiments, a systematic measurement of how many test pages attempt iframe or environment-detection checks, or a revised claim explicitly limited to pages that do not employ such anti-analysis techniques.
  2. [Section 5.1, Table 1] No confidence intervals, significance tests, or cross-validation are reported for the binary classification results. With a 200-sample test set, the 1-2 percentage point differences between LLaMA-3.1 8B, Phi-3, and the GPT cloud reference are within sampling error, and the 7-point advantage over the Random Forest baseline is not shown to be statistically significant. Please provide bootstrap confidence intervals or evaluate on a larger, independently constructed benchmark. Additionally, the Random Forest baseline is trained on a separate 1000+1000 dataset using largely lexical features and only a few page-level counts, so the comparison does not isolate the contribution of LLM reasoning; a strong feature-based classifier using the same static and dynamic signals would be a more informative baseline.
  3. [Section 5.1, Section 6.1] Because the malicious test pages are saved copies of public phishing and malware pages, they may have appeared in the pretraining corpora of LLaMA, Phi-3, and Gemma. The reported zero-shot accuracy could therefore reflect memorization of known pages rather than generalization from the provided static and dynamic evidence. The authors should demonstrate robustness with a temporal split using pages created after the models' knowledge cutoffs, or with synthetic/honeypot pages not present on the public web, to support the zero-shot generalization claim.
  4. [Sections 3 and 3.1] The claimed resistance to anti-analysis techniques is not experimentally supported. The sandbox is an iframe, so any page can check `window.top !== window.self` or `window.parent !== window`; the instrumentation replaces native functions such as fetch, XMLHttpRequest, appendChild, and eval, and pages can detect such replacement using `Function.prototype.toString` or by comparing saved native references. The paper asserts that the sandbox 'is designed to be resistant to detection' and that the hooks are 'stealthy', but no experiment or adversarial analysis is presented. I recommend adding a dedicated detectability evaluation, or removing the evasion-resistance claims from the abstract and introduction and clearly stating that the current evidence applies only to non-anti-analysis pages.
minor comments (4)
  1. [Section 6.1, Table 1] The model name is inconsistent: Section 6.1 refers to 'GPT-3.5 cloud reference' while Table 1 and the abstract use 'GPT-o3'. Please standardize the name and clarify which OpenAI model was actually used.
  2. [Section 3 and Section 6.2] Figure numbering is confusing: Section 3 refers to 'Figure 3' for the system architecture, but the caption reads 'Figure 1', and the examples in Section 6.2 are labeled 'Figure 2' and 'Figure 3' while the captions use the heading 'Figure: LLM Verdict...'. Please renumber the figures consistently.
  3. [Section 5.2, Section 6.2] The explanation-quality evaluation is described only as 'we took a sample' and reports that 90% of explanations were 'correct and sufficient'. The paper should state the sample size, the selection procedure, the rating rubric, and, ideally, inter-rater reliability; as written, this quantitative claim is not verifiable.
  4. [Section 4.5] The 'simple risk scoring' mentioned in Section 4.5 assigns weights to evidence items, but the paper does not specify how these weights are chosen or calibrated. Please state the weighting scheme and whether the risk scoring affects the binary accuracy numbers reported in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported detection accuracy is grounded in external labels and independent baselines, not in self-referential construction.

full rationale

The paper's central claim is empirical: a compact in-browser LLM, given static and dynamic context, classifies malicious versus benign URLs zero-shot. Section 5.1 describes a 200-URL dataset with malicious labels cross-verified via VirusTotal and PhishTank, and Section 5.3 describes the procedure; Table 1 compares the system against a separately trained Random Forest and a cloud GPT-o3 reference. No parameter is fitted to the test set and then reported as a prediction, and no equation defines the claimed result in terms of its inputs. The self-references in the paper — abstract 'client-side framework [1]', Section 6.2 'our new platform', and Section 7.5 'as JavaSith [11] did for extensions' — point to the author's own products or prior work, but none of these citations supplies the accuracy numbers; those numbers come from the described experiment against external ground truth. The sandbox-detectability concern raised by the reviewer is a validity or transfer threat: if a page detects the iframe, live performance may differ from saved-copy evaluation, but it does not make the measured results equivalent to the paper's assumptions by construction. Accordingly, the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no mathematical free parameters; the main hand-chosen values are the analysis wait time and the undisclosed risk weights. It also relies on three domain assumptions about sandbox invisibility, LLM latent knowledge, and dataset representativeness, none of which are proven. No new entities are invented.

free parameters (2)
  • dynamic analysis wait time = 4 seconds (3-5 seconds in Section 3)
    The sandbox stops collecting evidence after a short wait. The paper notes in Section 7.2 that pages which defer malicious behavior beyond the analysis window may slip through, so this hand-chosen value directly affects recall.
  • evidence risk weights = not disclosed
    Section 4.5 states each piece of evidence is assigned a weight for the final risk score, but the weights are not specified. They are chosen by the author and cannot be checked.
assumptions (3)
  • domain assumption A real browser iframe with monkey-patched API wrappers is indistinguishable from a normal user session to the analyzed page.
    Section 3 and Section 7.2; if false, dynamic findings do not reflect real user-visible behavior and the central detection claim weakens.
  • domain assumption Small instruction-tuned LLMs have enough pretrained security knowledge to perform zero-shot malicious/benign classification from structured evidence.
    Section 4.4 relies on the model's latent knowledge rather than task-specific training; if false, the 92% accuracy would not transfer beyond the specific prompts and models tested.
  • domain assumption The 200-URL test set (100 malicious, cross-verified with VirusTotal/PhishTank; 100 benign known sites) is representative and correctly labeled.
    Section 5.1; the curated distribution and offline hosting likely make detection easier than live web traffic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Client-Side Zero-Shot LLM Inference for Comprehensive In-Browser URL Analysis." pith.science (2026). https://pith.science/paper/IJ4RHHZG

@misc{pith2026250603656,
  author       = {Pith},
  title        = {Pith review of: Client-Side Zero-Shot LLM Inference for Comprehensive In-Browser URL Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJ4RHHZG}},
  note         = {Machine review of arXiv:2506.03656}
}
read the original abstract

Malicious websites and phishing URLs pose an ever-increasing cybersecurity risk, with phishing attacks growing by 40% in a single year. Traditional detection approaches rely on machine learning classifiers or rule-based scanners operating in the cloud, but these face significant challenges in generalization, privacy, and evasion by sophisticated threats. In this paper, we propose a novel client-side framework for comprehensive URL analysis that leverages zero-shot inference by a local large language model (LLM) running entirely in-browser. Our system uses a compact LLM (e.g., 3B/8B parameters) via WebLLM to perform reasoning over rich context collected from the target webpage, including static code analysis (JavaScript abstract syntax trees, structure, and code patterns), dynamic sandbox execution results (DOM changes, API calls, and network requests),and visible content. We detail the architecture and methodology of the system, which combines a real browser sandbox (using iframes) resistant to common anti-analysis techniques, with an LLM-based analyzer that assesses potential vulnerabilities and malicious behaviors without any task-specific training (zero-shot). The LLM aggregates evidence from multiple sources (code, execution trace, page content) to classify the URL as benign or malicious and to provide an explanation of the threats or security issues identified. We evaluate our approach on a diverse set of benign and malicious URLs, demonstrating that even a compact client-side model can achieve high detection accuracy and insightful explanations comparable to cloud-based solutions, while operating privately on end-user devices. The results show that client-side LLM inference is a feasible and effective solution to web threat analysis, eliminating the need to send potentially sensitive data to cloud services.

Figures

Figures reproduced from arXiv: 2506.03656 by the authors.

Figure 1
Figure 1. System Architecture. A high-level diagram showing the flow: The user’s browser triggers analysis for a URL. The Static Analysis Engine fetches and parses scripts to AST, extracting code features. The Dynamic Sandbox loads the URL in an iframe, with hooks monitoring DOM changes and API calls. Collected data (code features, runtime events, visible text) are sent to the Prompt Builder, which constructs a detailed promp… view at source ↗
Figure 1
Figure 1. Comparison of Detection Performance Metrics across Different [PITH_FULL_IMAGE:figures/full_fig_p032_1.png] view at source ↗
Figure 2
Figure 2. Example of LLM verdict for a real phishing page. [PITH_FULL_IMAGE:figures/full_fig_p034_2.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Example of LLM score for a real phishing page. [PITH_FULL_IMAGE:figures/full_fig_p034_3.png]
Figure 4
Figure 4. Figure 4: Example of LLM analysis for github.com. This shows the LLM can articulate why something is safe, which is use￾ful to avoid user panic when something is flagged incorrectly. It basically validated the absence of red flags and the consistency of domain identity. We did e…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

  1. [1]

    URL Analysis Platform: https://url.security

    work page

  2. [2]

    Browser Security Platform : https://browsersecurity.ai

    work page

  3. [3]

    Kaspersky. (2024). Kaspersky reports phishing attacks grow by 40% in 2023 . Press release. [Online]. Available: https://www.kaspersky.com/about/press-releases/2024_ kaspersky-reports-phishing-attacks-grow-by-40-percent-in-2023

    work page 2024

  4. [4]

    K., & Gupta, B

    Jain, A. K., & Gupta, B. B. (2019). A machine learning based approach for phishing detection using hyperlinks information. Journal of Ambient Intelligence and Humanized Computing, 10(5), 2015–2028

    work page 2019

  5. [5]

    Le, H., Pham, Q., Sahoo, D., & Hoi, S. C. (2018). URLNet: Learn- ing a URL representation with deep learning for malicious URL detec- 45 tion. arXiv preprint arXiv:1802.03162. https://arxiv.org/abs/1802. 03162

    arXiv 2018

  6. [6]

    Cova, M., Kruegel, C., & Vigna, G. (2010). Detection and analysis of drive-by-download attacks and malicious JavaScript code. In Proceedings of the 19th International World Wide Web Conference (WWW)

    work page 2010

  7. [7]

    Curtsinger, C., Livshits, B., Zorn, B., & Seifert, C. (2011). ZOZZLE: Fast and precise in-browser JavaScript malware detection. In USENIX Security Symposium

    work page 2011

  8. [8]

    C., Wardman, B., Sarker, S., et al

    Zhang, P., Oest, A., Cho, H., Sun, Z., Johnson, R. C., Wardman, B., Sarker, S., et al. (2021). CrawlPhish: Large-scale analysis of client-side cloaking techniques in phishing. In 42nd IEEE Symposium on Security and Privacy (SP)

    work page 2021

Show all 12 references
  1. [9]

    S., Reid, M., Matsuo, Y., & Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in Neural In- formation Processing Systems, 35

    2022

  2. [11]

    Cohen, A. (2025). JavaSith: A client-side framework for analyzing po- tentially malicious extensions in browsers, VS Code, and NPM packages. arXiv preprint arXiv:2505.21263

    2025 arXiv

  3. [12]

    Caballar, R. D. (2024). What is Google Gemma? IBM Technology Blog, Nov 8, 2024. [Online]. Available: https://www.ibm.com/think/ topics/google-gemma

    2024

  4. [13]

    F., Qin, Y., Zhou, X., Lai, R., Jin, H., Dong, Y., Hou, B., Yu, M., Zhai, Y., Agarwal, S., Cao, H., Feng, S., & Chen, T

    Ruan, C. F., Qin, Y., Zhou, X., Lai, R., Jin, H., Dong, Y., Hou, B., Yu, M., Zhai, Y., Agarwal, S., Cao, H., Feng, S., & Chen, T. (2024). We- bLLM: A High-Performance In-Browser LLM Inference Engine. arXiv preprint arXiv:2412.15803. [Online]. Available: https://arxiv.org/ abs/...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.