Pith. sign in

REVIEW 3 major objections 5 minor 35 references

A local multi-agent firewall can stop sensitive data from reaching LLMs without changing apps, with F1 up to 94.93%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A local multi-agent firewall intercepts LLM web and API traffic and detects sensitive data with hybrid detectors, reaching up to 94.93% F1 on a PII benchmark.

T0 review reviewed 2026-07-10 challenge →

load-bearing objection Solid open dual-sensor DLP mesh for LLM web+API traffic; the 94.93% F1 is real under a matched protocol but should not be read as open-domain leakage proof. the 3 major comments →

arxiv 2607.08282 v1 pith:F4X4BBU6 submitted 2026-07-09 cs.CR cs.AIcs.MA

Multi-Agent Firewall Architecture for Privacy Protection of Sensitive Data in Interactions with Language Models

Hugo Garc\'ia Cuesta , Pablo Mateo Torrej\'on , Alfonso S\'anchez-Maci\'an This is my paper
classification cs.CR cs.AIcs.MA
keywords Large Language ModelsData Leakage PreventionPrompt SecurityNamed Entity RecognitionPrivacy-Preserving DetectionLocal InferenceMulti-Agent FirewallBrowser Extension and Proxy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

People and organizations use language models for work and personal tasks, but sensitive details in prompts can leak through web chat and API calls. This paper presents an open-source, privacy-first firewall that sits in front of those interactions. A browser extension watches web UIs and a transparent proxy covers HTTP and WebSocket API traffic, so no application code has to change. Captured content is analyzed by a multi-agent pipeline that starts with fast deterministic and entity detectors and escalates only when needed to language-model semantic checks, with optional code-similarity matching against private repositories. Sensitive values can be redacted before any external model sees them. The design is meant to run locally or on a shared internal server so organizations can trade latency, cost, and detection depth. On a PII benchmark with the best local configuration, the system reaches an F1 of 94.93%.

Core claim

A dual-interception, local-first multi-agent firewall can prevent sensitive data leakage in both web and programmatic LLM interactions without application modifications or external scanning vendors, using hybrid deterministic, NER, code-similarity, and conditional LLM detection that reaches F1 scores up to 94.93% on optimal local configurations.

What carries the argument

The multi-agent security pipeline: a configurable directed acyclic graph that runs parallel first-layer detectors (regex/keyword/checksum, zero-shot NER, fuzzy code similarity) and conditionally escalates to LLM semantic analysis after sanitizing already-found values into typed redaction tokens.

Load-bearing premise

That strong scores on a fixed 500-case US PII text split, with flexible label matching and a peak result from a model fine-tuned on the same dataset family, while skipping multimodal, code, and anonymization tests, stand in for real-world leakage prevention.

What would settle it

Run the same pipeline on a held-out, multi-locale, multi-format corpus that includes images, PDFs, and proprietary code snippets, without fine-tuning on that corpus, and check whether F1 stays near 95% under realistic latency and false-positive constraints.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Minos Verdict Mesh, an open-source local-first multi-agent firewall for sensitive-data leakage prevention in LLM interactions. It combines dual interception (Chromium browser extension with platform adapters plus a mitmproxy-based MiTM proxy for HTTP(S) and WSS) with a LangGraph DAG pipeline that hybridizes deterministic detectors (regex, keywords, checksums), zero-shot NER (GLiNER), optional fuzzy code similarity against Git corpora, and conditional or forced LLM semantic analysis, plus a sanitization-first redaction workflow and configurable risk/policy enforcement. Deployment modes (standalone vs centralized), multimodal file handling (Tesseract then VLM), and extensibility surfaces are described in detail. Evaluation on a fixed 500-case US Nemotron-PII test split reports ablations across 11 configurations (Table 3), with peak F1 94.93% for a fine-tuned Gemma-3 4B LLM-only setup (T10), and per-source TP/FP breakdowns (Table 4).

Significance. If the systems claims hold, the work is a useful practical contribution to LLM data-loss prevention: dual-channel transparent interception without application changes, local-first analysis that avoids the cloud-inspection trust shift, and a configurable hybrid pipeline that lets operators trade latency, cost, and depth. Strengths include a concrete open-source artifact (repository and commit hash given), externalized pipeline/detection JSON, ablation across conditional vs forced LLM and NER thresholds, and explicit fail-closed sensor behavior. These are genuine engineering assets for organizations that cannot ban third-party LLMs but also cannot accept unrestricted prompt exfiltration. The significance is primarily systems and deployment-oriented rather than a new detection algorithm.

major comments (3)
  1. [Abstract; §5.1–5.2; Table 3 (T10)] Abstract and §5 present F1 “up to 94.93% on optimal configurations” as the main empirical support for the framework. Table 3 shows that peak is T10 (LLM-only, forced, gemma3-ft fine-tuned on the Nemotron-PII training split), not the hybrid dual-interception system under realistic conditional routing. §5.1 further excludes multimodal OCR/VLM, code similarity, and anonymization fidelity. The headline number therefore measures specialized local PII tagging on a matched benchmark, not end-to-end leakage prevention of the architecture advertised in the abstract and §1. Reframe the abstract around the hybrid/conditional operating points (e.g., T2–T3 or T6–T7) and clearly label T10 as a fine-tuned upper bound, or add OOD/held-out-taxonomy results.
  2. [§5.1; Table 3; Table 4] Scoring uses flexible entity equivalence (exact, substring, and bidirectional aliases such as FIRST_NAME↔PERSON, IPV4↔IP) against Nemotron-PII labels (§5.1). Combined with min_block_level=low (any detection can drive block), this can inflate both F1 and the practical “prevention” interpretation. Report a strict exact-label metric alongside the flexible one, and separate detection quality from policy outcomes (allow/warn/block rates under medium/high thresholds).
  3. [§3; §5.1; Fig. 1] The central systems claim is dual transparent interception plus hybrid pipeline for web and API traffic. Evaluation only exercises the multiagent-firewall package on textual prompts (§5.1). There is no measurement of extension DOM adapters, proxy payload extraction across providers, WSS handling, fail-closed behavior under backend failure, remediation/sanitization fidelity, or false-block rates on clean prompts. Without at least a limited end-to-end or interception-correctness study, the dual-layer architecture remains largely unvalidated relative to the abstract’s scope.
minor comments (5)
  1. [Title page; References] Manuscript date “July 10, 2026” and several 2025–2026 reference access dates look inconsistent with a normal publication timeline; please verify metadata and citation years.
  2. [Fig. 2; §4] Figure 2 flowchart is dense; a short caption walkthrough of the main conditional branches (file path, Findings?, Block?, New/Known) would help readers follow the DAG without reverse-engineering node names.
  3. [Table 1; §2] Table 1 comparison is helpful but “LocalAnalysis / Multimodal” cells are coarse; a footnote clarifying what “Partial” means for prior systems would reduce ambiguity.
  4. [Table 4; §5.3] Phone-number detector shows ~20% FP across runs (Table 4); a brief discussion of whether keyword windows or locale constraints can reduce this would strengthen the deterministic-layer analysis.
  5. [Throughout] Minor wording/typos: “OW ASP” spacing, “themmitmproxy”, “indetection.json”, and occasional missing spaces after periods in the compiled text.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper with standard held-out evaluation, not a derivation that reduces to its inputs by construction.

full rationale

This is an architecture-and-evaluation paper, not a first-principles derivation. The dual-interception design (browser extension + MiTM proxy) and multi-agent DAG pipeline (deterministic/NER/code-similarity with conditional LLM escalation, sanitization-first workflow, risk policy) are presented as engineered artifacts whose behavior is measured, not deduced from self-referential equations. Evaluation uses the external nvidia/Nemotron-PII test split (US locale, fixed seed, 500 cases) under deterministic LLM decoding; T10’s peak F1 of 94.93% comes from fine-tuning Gemma-3-4B on the training split of the same public dataset and scoring the held-out test split—standard supervised ML practice, not a fitted parameter renamed as a prediction of a closely related quantity, nor a definitional identity. Flexible entity equivalence and min_block_level=low are scoring/policy choices, not circular reductions. The sole prior-work citation to de la Riva [15] (same institution, non-overlapping authors) is acknowledged as a foundation that is advanced, not invoked as a uniqueness theorem or load-bearing external fact that forces the present claims. Multimodal, code-similarity and anonymization axes are explicitly excluded from the reported numbers, so the abstract’s “up to 94.93%” is an empirical upper bound under stated conditions rather than a self-fulfilling construction. No self-definitional loops, no ansatz smuggled via self-citation, and no renaming of a known result as a novel derivation appear. Score 0 is therefore the correct, proportionate finding.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Load-bearing content is engineering assumptions and evaluation protocol choices, not new physics entities. Free parameters are thresholds and policy knobs that directly change F1/latency/block decisions. Domain axioms cover the threat model and traffic conventions. Invented entities are the named system and pipeline roles as software constructs, not independently evidenced natural kinds.

free parameters (5)
  • NER confidence threshold
    Swept at 0.3/0.5/0.7/0.85; strongly shifts precision/recall and whether LLM escalation fires (Group 2, Table 3).
  • min_block_level / risk tier weights
    Policy maps field counts to allow/warn/block; evaluation uses the most restrictive low setting, which changes when LLM is invoked and what counts as success for leakage prevention (§4.5, §5.2.1).
  • Flexible entity equivalence rules (aliases/substrings)
    Scoring accepts exact, substring, and bidirectional aliases (e.g., IPV4↔IP, FIRST_NAME↔PERSON), which can raise measured F1 relative to strict label match (§5.1).
  • LLM model choice and fine-tuning on Nemotron-PII train (T10)
    Peak 94.93% F1 depends on a fine-tuned gemma3-ft detector; non-fine-tuned models sit lower (Table 3).
  • Tesseract confidence threshold for VLM escalation
    Controls when expensive multimodal OCR runs; affects cost/latency though not scored in §5 (§4.2).
axioms (5)
  • domain assumption Major LLM providers expose user content in interceptable HTTP(S)/WSS forms with role-tagged messages, so dual sensors can cover dominant usage without app hooks.
    Stated as design premise in §3 and proxy extraction logic §3.3; future gRPC/WebRTC noted as out of current scope.
  • domain assumption Deterministic + zero-shot NER findings redacted before external/local LLM semantic analysis preserve enough context for useful detection without re-exposing raw secrets.
    Sanitization-first workflow §4.4; central to resolving the privacy paradox.
  • ad hoc to paper Nemotron-PII US 500-case sample with the paper’s matching rules is a fair proxy for sensitive-data leakage detection quality in LLM prompts.
    Evaluation design §5.1; multimodal/code/anonymization excluded by choice.
  • domain assumption Fail-closed behavior on backend unavailability is acceptable operational policy.
    Extension and proxy both refuse unchecked traffic (§3.2–3.3).
  • standard math Standard string similarity / regex / GLiNER / instruction-tuned LLM behavior as used in the stack is adequate building material for DLP agents.
    Uses established libraries and models (RapidFuzz, phonenumbers, GLiNER, LiteLLM) without new learning theory (§4.3).
invented entities (2)
  • Minos Verdict Mesh multi-agent firewall pipeline independent evidence
    purpose: Name and structure the configurable DAG of detectors, risk policy, and remediation as a deployable product architecture.
    Introduced as the proposed framework; independent evidence is the open-source implementation and benchmarks, not an external natural phenomenon.
  • Browser BasePlatform adapter registry independent evidence
    purpose: Decouple DOM interception from specific LLM websites so new providers can be added without core rewrites.
    Software abstraction in §3.2; evidence is engineering extensibility, not external measurement.

reviewed 2026-07-10 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Agent Firewall Architecture for Privacy Protection of Sensitive Data in Interactions with Language Models." pith.science (2026). https://pith.science/paper/F4X4BBU6

@misc{pith2026260708282,
  author       = {Pith},
  title        = {Pith review of: Multi-Agent Firewall Architecture for Privacy Protection of Sensitive Data in Interactions with Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4X4BBU6}},
  note         = {Machine review of arXiv:2607.08282}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While Large Language Models (LLMs) have become essential productivity tools, their integration into workflows without adequate safeguards creates significant risks. This paper proposes an open-source, privacy-focused, user-facing firewall designed to secure both web-based and programmatic LLM interactions. The architecture combines a browser extension and a proxy for total traffic interception across both HTTP(S) and WebSocket communications. At its core, a flexible multi-agent pipeline delivers data leakage prevention through a hybrid approach combining deterministic detectors with LLM-driven semantic analysis, proprietary code leakage prevention, and extensible components designed for future security enhancements such as prompt injection evasion. The framework's layered architecture enables deployment across heterogeneous environments, allowing organizations to balance computational cost, detection depth and latency. Evaluation results demonstrate it achieves F1 scores of up to 94.93% on optimal configurations.

Figures

Figures reproduced from arXiv: 2607.08282 by Alfonso S\'anchez-Maci\'an, Hugo Garc\'ia Cuesta, Pablo Mateo Torrej\'on.

Figure 1
Figure 1. Figure 1: Diagram representation of the system’s architecture. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Flowchart of the Detection and Remediation Logic. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Deterministic Detector Internal Flow. Overall, this NER agent provides a basic semantic detection layer that captures context-dependent entities which lack a predictable structural format, serving as a flexible fallback for identifying sensitive information that regular expressions alone fail to resolve. 4.3.3 Heuristic Fuzzy Code Similarity Matching Proprietary source code represents a critical exfiltrati… view at source ↗
Figure 4
Figure 4. Figure 4: Precision vs recall trade-off, with bubble size proportional to latency. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average TP/FP contribution by detector group cross evaluated configurations. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages · 1 internal anchor

  1. [1]

    Alert fatigue in security operations centres: Research challenges and opportunities.ACM Computing Surveys, 2025

    Shan Tariq, Meera Baruwal Chhetri, Surya Nepal, and Cecile Paris. Alert fatigue in security operations centres: Research challenges and opportunities.ACM Computing Surveys, 2025

    work page 2025

  2. [2]

    Siying Hu, Piaohong Wang, Ka I Chan, Yaxing Yao, and Zhicong Lu. I always felt that something was wrong: Understanding compliance risks and mitigation strategies when highly-skilled compliance knowledge workers use large language models.arXiv preprint arXiv:2411.04576, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    European Union. Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation). Official Journal of the European Union, L119, pp. 1–88, 2016

    work page 2016

  4. [4]

    Datadvantage: Data governance and security for unstructured data

    Varonis Systems. Datadvantage: Data governance and security for unstructured data. https://www.varonis. com/, 2005

    work page 2005

  5. [5]

    Symantec data loss prevention: Network discover and cloud storage

    Symantec Corporation. Symantec data loss prevention: Network discover and cloud storage. https://www. broadcom.com/products/cybersecurity, 2007

    work page 2007

  6. [6]

    Owasp 2025 top 10 risk & mitigations for llms and gen ai apps.https://genai.owasp

    OW ASP Foundation. Owasp 2025 top 10 risk & mitigations for llms and gen ai apps.https://genai.owasp. org/llm-top-10/, 2025. Part of the OW ASP GenAI Security Project

    work page 2025

  7. [7]

    Presidio: Data protection and de-identification sdk

    Microsoft. Presidio: Data protection and de-identification sdk. https://github.com/microsoft/presidio,

    work page

  8. [8]

    Open-source software release available athttps://microsoft.github.io/presidio

    work page

  9. [9]

    spacy: Industrial-strength natural language processing in python.https://zenodo.org/record/1212303, 2020

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial-strength natural language processing in python.https://zenodo.org/record/1212303, 2020. Zenodo software release

    work page arXiv 2020

  10. [10]

    Llm guard: Security and privacy for llms

    Protect AI. Llm guard: Security and privacy for llms. https://protectai.github.io/llm-guard/, 2025. Toolkit for LLM security and privacy

    work page 2025

  11. [11]

    Nvidia nemo guardrails

    NVIDIA Corporation. Nvidia nemo guardrails. https://github.com/NVIDIA/NeMo-Guardrails, 2025. Open-source toolkit for LLM safety and security

    work page 2025

  12. [12]

    Lakera guard: The ai security layer

    Lakera. Lakera guard: The ai security layer. https://www.lakera.ai/, 2025. Cloud-based AI security platform

    work page 2025

  13. [13]

    The nightfall ai platform

    Nightfall. The nightfall ai platform. https://www.nightfall.ai/, 2025. AI-powered data leakage prevention platform

    work page 2025

  14. [14]

    Azure ai content safety

    Microsoft Corporation. Azure ai content safety. https://azure.microsoft.com/en-us/products/ ai-services/ai-content-safety, 2025. Accessed: 2025-02-09

    work page 2025

  15. [15]

    Exploiting privacy preserving prompt techniques for online large language model usage

    Youxiang Zhu, Ning Gao, Xiaohui Liang, and Honggang Zhang. Exploiting privacy preserving prompt techniques for online large language model usage. InIEEE Global Communications Conference (GLOBECOM), pages 4304–4309, March 2024. 16 Multi-Agent FW Architecture For Protection Of Sensitive Data In Interactions With LLMs

    work page 2024

  16. [16]

    Sistema intermediario para la detección y clasificación de información sensible en interacciones con llms

    Guillermo Cabello de la Riva. Sistema intermediario para la detección y clasificación de información sensible en interacciones con llms. Master’s thesis, Universidad Carlos III de Madrid, Leganés, Spain, September 2025

    work page 2025

  17. [17]

    Manifest v3

    Google Chrome Developers. Manifest v3. https://developer.chrome.com/docs/extensions/develop/ migrate/what-is-mv3, 2025

    work page 2025

  18. [18]

    mitmproxy: An interactive https proxy

    mitmproxy Project. mitmproxy: An interactive https proxy. https://docs.mitmproxy.org/stable/, 2024. Open-source proxy tool for intercepting and analyzing HTTP/HTTPS traffic

    work page 2024

  19. [19]

    Fastapi framework, high performance, easy to learn, fast to code, ready for production

    Tiangolo. Fastapi framework, high performance, easy to learn, fast to code, ready for production. https: //fastapi.tiangolo.com/, 2024. Python web framework for building APIs

    work page 2024

  20. [20]

    Langgraph: Building language agents as graphs

    LangChain. Langgraph: Building language agents as graphs. https://github.com/langchain-ai/ langgraph, 2024. Library for building stateful, multi-agent applications

    work page 2024

  21. [21]

    pdfplumber

    Jeremy Vine. pdfplumber. https://github.com/jsvine/pdfplumber, 2024. Python library for PDF text and table extraction

    work page 2024

  22. [22]

    Tesseract: Open source optical character recognition engine

    Tesseract OCR Project. Tesseract: Open source optical character recognition engine. https://tesseract-ocr. github.io/, 2025. Free, open-source OCR engine

    work page 2025

  23. [23]

    Litellm: Call 100+ llm apis in openai format

    BerriAI. Litellm: Call 100+ llm apis in openai format. https://github.com/BerriAI/litellm, 2024. Unified interface for language model providers

    work page 2024

  24. [24]

    GLiNER: Generalist model for named entity recognition using bidirectional transformer

    Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. GLiNER: Generalist model for named entity recognition using bidirectional transformer. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vo...

    work page 2024

  25. [25]

    Rapidfuzz: Rapid fuzzy string matching in python and c++ using the levenshtein distance

    Max Bachmann. Rapidfuzz: Rapid fuzzy string matching in python and c++ using the levenshtein distance. https://github.com/rapidfuzz/RapidFuzz, 2024

    work page 2024

  26. [26]

    Introducing openai privacy filter: A state of the art model for masking personally identifiable information in text

    OpenAI. Introducing openai privacy filter: A state of the art model for masking personally identifiable information in text. https://openai.com/index/introducing-openai-privacy-filter/ , 2026. OpenAI’s open- weight PII detection model

    work page 2026

  27. [27]

    Nemotron-pii: Benchmark dataset for pii detection

    NVIDIA Corporation. Nemotron-pii: Benchmark dataset for pii detection. https://huggingface.co/ datasets/nvidia/Nemotron-PII, 2025. Publicly available dataset for evaluating PII detection systems

    work page 2025

  28. [28]

    google/gemma-3-4b-it

    Google DeepMind. google/gemma-3-4b-it. https://huggingface.co/google/gemma-3-4b-it , 2025. 4- billion parameter instruction-tuned Gemma 3 model on HuggingFace Hub

    work page 2025

  29. [29]

    mistralai/ministral-3-8b-instruct-2512

    Mistral AI. mistralai/ministral-3-8b-instruct-2512. https://huggingface.co/mistralai/ Ministral-3-8B-Instruct-2512 , 2025. 8-billion parameter instruction-tuned Ministral 3 model on HuggingFace Hub

    work page 2025

  30. [30]

    meta-llama/llama-3.1-8b-instruct

    Meta. meta-llama/llama-3.1-8b-instruct. https://huggingface.co/meta-llama/Llama-3. 1-8B-Instruct, 2024. 8-billion parameter Llama 3.1 instruction tuned on HuggingFace Hub

    work page 2024

  31. [31]

    Unsloth AI. Unsloth. https://github.com/unslothai/unsloth, 2024. Open-source framework for efficient LLM fine-tuning

    work page 2024

  32. [32]

    google/gemma-4-e4b-it

    Google DeepMind. google/gemma-4-e4b-it. https://huggingface.co/google/gemma-4-E4B-it , 2026. 4-billion parameter efficient instruction tuned Gemma 4 model on HuggingFace Hub

    work page 2026

  33. [33]

    Desktop browser market share worldwide

    Statcounter Global Stats. Desktop browser market share worldwide. https://gs.statcounter.com/ browser-market-share/desktop/worldwide, 2026. Accessed: 2026-03-30

    work page 2026

  34. [34]

    Realtime api

    OpenAI. Realtime api. https://platform.openai.com/docs/guides/realtime, 2026. WebSocket and REST API integration for real-time AI interactions; accessed: 2026-03-30

    work page 2026

  35. [35]

    temperature

    Google Cloud. Vertex ai generative ai live api.https://cloud.google.com/vertex-ai/generative-ai/ docs/live-api/get-started-websocket , 2026. WebSocket Secure integration for Gemini Live API; ac- cessed: 2026-03-30. 17 Multi-Agent FW Architecture For Protection Of Sensitive Data In Interactions With LLMs A Source Code, Extensibility and Dependencies A.1 Re...

    work page 2026

This paper was first reviewed by grok-4.5 on July 10, 2026.