{"id":"5205ef4f-8abe-41a3-bca1-a8423a4a4876","arxiv_id":"2501.12456","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A case study claiming OneShield Privacy Guard achieves 0.95 F1 in multilingual PII detection and saves 300+ review hours, with limited public evidence.","lead":"This paper describes OneShield Privacy Guard, a framework that detects personal information in prompts and outputs for enterprise and open-source LLM deployments. It reports high F1 scores and time savings from two real deployments, but the evaluation details are thin and the headline numbers are hard to verify.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 0.95 F1 and 'up to 12%' outperformance are not supported by Table 2, so the central performance claim rests on an internal inconsistency.","rationale":"The reader's weakest assumption concerns the 300-hour effort-savings estimate, which depends on an unvalidated 15-minute manual review time. That is a valid external-evidence concern, but the more load-bearing problem is internal: the paper's headline 0.95 F1 result for Deployment 1 is contradicted by the paper's own Table 2. The macro-average of the seven reported D1 F1 scores is 0.936, not 0.95, and the 'up to 12%' improvement figure similarly does not match the table's differences of up to 27%. This matters because the abstract's quantitative performance claim is the paper's primary contribution; if the number cannot be derived from the presented evidence, the entire evaluation section is called into question. The effort-savings issue is secondary and would matter even if the F1 numbers were correct. Since the reader's verdict is already REJECT, my finding does not change that verdict; I identify a different, more fundamental weakness. The concrete test is straightforward and could be done with the paper alone or with released artifacts, which are currently absent.","tokens_in":9333,"tokens_out":5499,"duration_ms":50532,"concrete_test":"Compute the macro-F1 and micro-F1 from Table 2 for Deployment 1 using the seven reported per-type F1 values (0.98, 0.96, 0.94, 0.89, 0.91, 0.92, 0.95). If neither aggregation equals 0.95, the abstract's headline claim is unsupported. To rule out a typo in Table 2, re-run OneShield and the two baselines on a fixed multilingual test set with released code and data and regenerate Table 2; the per-type values and the aggregate must reproduce and match the claimed 0.95.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires a well-defined aggregate F1 of 0.95 for Deployment 1 and a comparison showing at most 12% improvement over baselines. The paper's own evidence contradicts both. Table 2 reports Deployment 1 per-PII-type F1 scores of: Person 0.98, Date 0.96, Email 0.94, Phone 0.89, Location 0.91, National ID 0.92, Credit Card 0.95. The macro-average over these seven types is 0.936, and no reasonable weighting of the listed types yields 0.95; the only cell equal to 0.95 is Credit Card, which is not the 'dates, names, and phone numbers' named in the abstract. The comparison column likewise does not support 'up to 12%': differences between the claimed OneShield D1 numbers and StarPII/Presidio in Table 2 range from 0.07 to 0.27, and no aggregate or per-type difference is reported as 12% except one baseline pair (Phone vs. Presidio, 0.12). Additionally, the abstract says 'across 26 languages' while the Deployment 1 narrative first names 13 languages and later says 26, with no per-language evaluation to show the aggregate. Since the paper provides no dataset, no confusion matrices, and no evaluation protocol, the 0.95 headline cannot be checked or reproduced. For the central claim to hold, there would need to be a defined aggregation rule under which the reported per-type scores average to 0.95; no such rule is stated or evident. This is an internal inconsistency, not merely a disagreement with outside consensus.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OneShield Privacy Guard, a framework for detecting and redacting personally identifiable information (PII) in LLM inputs and outputs, and reports on two deployments: an enterprise deployment in 'Data and Model Factory' and an open-source deployment in 'PR Insights'. The enterprise deployment is claimed to achieve a 0.95 F1 score across 26 languages and to outperform StarPII and Presidio by up to 12%, while the open-source deployment is reported to flag 8.25% of 1,256 pull requests and save over 300 hours of manual effort. The paper also compares per-PII-type F1 scores for both deployments against StarPII and Presidio, discusses contextual sensitivity scoring and policy enforcement, and offers lessons learned and future directions.","tokens_in":9693,"tokens_out":4625,"duration_ms":46768,"significance":"If the reported results were substantiated, the paper would be a useful contribution to the understudied area of deployed privacy guardrails for LLMs: it documents contextual entity scoring, policy-driven masking, multilingual coverage, and human-in-the-loop refinement in real operational settings. The paper's qualitative description of the two architectures and the lessons learned has some value as a deployment experience report. However, the central quantitative claims are not reproducible from the presented information, and one headline claim is arithmetically inconsistent with the paper's own reported numbers. As it stands, the paper does not provide a validated comparative evaluation, and its main contribution would need to be reframed as a qualitative case study with substantially corrected claims.","major_comments":[{"comment":"The headline claim of a 0.95 F1 score across 26 languages is internally inconsistent with Table 2. For Deployment 1, Table 2 reports per-type F1 scores of 0.98 (Person), 0.96 (Date), 0.94 (Email Address), 0.89 (Phone Number), 0.91 (Location), 0.92 (National ID), and 0.95 (Credit Card). The macro-average of these seven scores is 0.936, and no stated weighting rule yields 0.95. Moreover, the abstract's examples of 'dates, names, and phone numbers' correspond to 0.96, 0.98, and 0.89, not 0.95; the only 0.95 cell is Credit Card. The same section first mentions 13 languages and later claims 26 languages, with no per-language results reported. The aggregate 0.95 claim should be replaced by a clearly defined aggregation of the reported per-type scores, or supported by a per-language/per-entity evaluation that is actually presented.","section":"Abstract and Deployment 1 Results and Evaluation, Table 2"},{"comment":"The comparative evaluation is not sufficiently specified to be checked or reproduced. The paper does not report the number of documents or prompts used for the F1 computation, the language distribution across the claimed 26 languages, the ground-truth labeling protocol, or the details of the 'human assessments' mentioned in the text. No error bars, confidence intervals, or significance tests accompany any F1 score in Table 2. It is also unclear how StarPII and Presidio were configured, which underlying models or dictionaries were used for these baselines, and how overlapping or nested entities were scored. Without this information, the claimed 'up to 12%' improvement over StarPII and Presidio and the per-type differences in Table 2 are unsupported.","section":"Deployment 1 Results and Evaluation and Table 2"},{"comment":"The '300+ hours saved' claim is arithmetically inconsistent with the data given in the same section. The paper reports that 8.25% of 1,256 pull requests were flagged, which is approximately 104 PRs. At the stated assumption of 15 minutes of manual review per PR, the implied saving is about 26 hours, not over 300 hours. If the 300-hour figure is intended to cover all 1,256 PRs, then the 8.25% flagged rate is not used in the calculation, and the assumption that all PRs would otherwise have been manually reviewed needs independent support. The earlier sentence in 'Challenges in Deployment 2' that states the system 'saved over 300 hours of manual reviews' repeats this claim before any evaluation is presented. This load-bearing result must be corrected or removed.","section":"Deployment 2 Results and Evaluation"}],"minor_comments":[{"comment":"The phrase 'outperforming state-of-the-art tool such as StarPII and Presidio' should be pluralized ('tools') for grammatical correctness.","section":"Abstract"},{"comment":"Many inline citations use abbreviated keys such as '(com 2023)', '(neu 2023)', '(mic 2023)', '(gdp 2024)', and '(ccp 2024)' that are not resolved in the reference list; the reference list itself contains incomplete and inconsistently formatted entries, including patent citations with garbled author names.","section":"References and inline citations"},{"comment":"There are several typos and malformed phrases, including 'Presidio Analyzed' instead of 'Presidio Analyzer', 'Eg Pseuanoanonyms' in the discussion of pseudonyms, and 'effectivenes' in the State of the Art section.","section":"Comparison of Deployments and Discussion"},{"comment":"The repository is referred to only as 'name anonymized'; if this is intended for double-blind review, the placeholder should be disclosed in a footnote, and if not, the repository should be named for reproducibility.","section":"Deployment 2"},{"comment":"Figure 3 is described only as a 'snapshot' of PII detection; the text should explain what is shown and how it relates to the reported quantitative results.","section":"Figure 3"}],"recommendation":"reject","confidential_remarks":"The paper's central quantitative claims are not supported by the evidence presented: the abstract's 0.95 aggregate F1 is not derivable from Table 2, the evaluation protocol is absent, and the 300+ hours saving is arithmetically inconsistent with the reported flagged rate and review-time assumption. These are load-bearing issues that would require new measurements or substantial reframing rather than local edits, so I cannot recommend acceptance at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has real deployment value—two production-grade privacy guardrail case studies with useful context-sensitivity lessons—but its headline quantitative claims are not supported by its own table. Table 2 contradicts the abstract's 0.95 F1 and 'up to 12%' advantage, and no dataset, labeling protocol, or baseline configuration is supplied to reconcile that.\n\nCredit where due. The two deployments are genuinely different environments (enterprise multilingual model factory vs. open-source GitHub PR bot), and the authors describe architecture, policy enforcement, and the human-in-the-loop feedback loop in enough detail to be actionable. The examples distinguishing sensitive from non-sensitive context (office phone vs. personal phone, historical figures vs. fictional placeholders) are the kind of practical nuance that often doesn't make it into research papers. The comparison to StarPII and Presidio, though thin, at least provides an external anchor.\n\nThe problems are load-bearing. The abstract reports a 0.95 F1 across 26 languages and says the system outperforms baselines by up to 12%. Table 2 gives per-type F1 values whose macro average is about 0.94, and the largest margin over Presidio is 0.24, not 0.12. A language-weighted or micro-average might produce 0.95, but the paper never says. There are no per-language numbers, no ground-truth labeling description, no error bars, no significance tests, and no configuration details for StarPII or Presidio. The 300+ hours saved is explicitly an estimate based on an assumed 15-minute review time; the text labels it as such, but the abstract presents it without the qualifier. No code or data is released, so none of these numbers can be independently checked.\n\nThe qualitative description suggests the framework is real and the authors have hands-on experience. But as a research contribution the evidence is too underspecified for the central claims. The discrepancy between the abstract and Table 2 is exactly what a referee would need resolved before publication.\n\nWho this is for: practitioners building LLM privacy guardrails will get more from this than researchers looking for a rigorous comparative evaluation. I would not desk-reject this on sight—the deployment lessons are valuable enough to warrant referee attention—but I would send it back for major revision with a specific request for the evaluation protocol, the aggregation rule that yields 0.95, and the baseline settings. If those cannot be produced, reject.","headline":"Real deployment lessons worth reading, but the headline F1 and margin claims are contradicted by the paper's own Table 2 and cannot be checked without data or protocol.","tokens_in":10174,"tokens_out":4428,"would_cite":false,"duration_ms":45060,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Privacy guard hits 0.95 F1 across 26 languages","keywords":["privacy guardrails","PII detection","large language models","named entity recognition","contextual sensitivity","multilingual","GDPR compliance","open-source triage"],"falsifier":"Audit the evaluation data behind the 0.95 F1 claim: list the number of prompts per language among the roughly 1,200 user prompts. If most of the 26 languages have only a handful of examples, or if the F1 was computed on synthetic or internally curated text rather than an independently annotated sample, the multilingual claim would not survive; likewise, if a replay of the PR Insights deployment logs triage times shows per-review effort near zero because most flagged PRs were auto-closed, the 300-hour saving would evaporate.","tokens_in":9190,"feed_emoji":"🛡️","tokens_out":4185,"duration_ms":38275,"temperature":0.7,"pith_summary":"The paper argues that OneShield Privacy Guard, a framework for detecting personally identifiable information (PII) in text that flows into and out of large language models, works in two very different real settings: an enterprise data and model factory and an open-source GitHub repository. In the enterprise deployment it reports a 0.95 F1 score for detecting names, dates, phone numbers, and other sensitive entities across 26 languages, beating the StarPII and Presidio tools by up to 12 percent. In the open-source deployment it flagged 8.25 percent of 1,256 pull requests for privacy review and is credited with saving over 300 hours of manual triage in three months. The paper's point is that context-aware sensitivity scoring, not pattern matching alone, is what makes one framework portable across these settings.","feed_headline":"Privacy guard hits 0.95 F1 across 26 languages","feed_subtitle":"Two live deployments show context-aware PII detection beating StarPII and Presidio and saving hundreds of review hours.","key_machinery":"The load-bearing mechanism is the three-stage detection pipeline: entity recognition, contextual sensitivity scoring, and policy enforcement. The framework deliberately combines rule-based components with machine-learning classifiers so that ambiguous strings, such as an office phone number versus a personal phone number, are resolved by context. In Deployment 2 the same machinery is wrapped in an automated GitHub bot with a human-in-the-loop feedback channel that iteratively retrains the classifier on cases like historical figures and fictional characters.","core_discovery":"The central claim is that a single privacy-guardrail architecture can be deployed at enterprise scale and in community open-source workflows, and that its edge over existing detectors comes from contextual scoring rather than from larger dictionaries or regex sets. OneShield's detector first recognizes entities, then scores sensitivity by looking at surrounding text and entity relationships, and then applies policy templates for GDPR, CCPA, or PIPEDA to mask, block, or pass the data. The paper reports F1 scores of 0.95 in Deployment 1 and 0.86 in Deployment 2, with per-entity scores above 0.89 for person, date, email, phone, location, national ID, and credit card categories in the enterprise setting. It also reports sub-millisecond detection latency on prompts up to 250 tokens, adding less than 5 percent overhead.","pith_inferences":["Editorial: The 300-plus-hours saved number is an arithmetic estimate built on a 15-minute-per-review assumption; the actual savings could be measured directly by logging triage times before and after deployment.","Editorial: Because the framework's advantage is attributed to context rather than language-specific resources, a natural testable extension is to run the same detector on low-resource languages not in the reported 26; its performance there would separate context-based generalization from tuning to the evaluated languages.","Editorial: The comparison tables omit confidence intervals and dataset sizes per entity type, so the reported up-to-12-percent margin should be read as a point estimate until a shared benchmark exists.","Editorial: The same contextual-scoring idea could apply to other sensitive content classes such as medical records or internal API tokens, where a token's sensitivity depends on where it appears."],"forward_implications":["If the 0.95 and 0.86 F1 results replicate on independent labeled data, organizations can deploy one multilingual guardrail rather than separate detectors per language or jurisdiction.","Context-aware scoring implies policy actions can be modulated: the same entity type can be masked in one context and passed in another, which matters for GDPR-style pseudonymity and public-figure exceptions.","The sub-millisecond latency figures suggest the guardrail can sit inline in interactive LLM applications without a noticeable slowdown.","The PR Insights deployment demonstrates that automated triage plus human feedback can reduce manual review volume to roughly 8.25 percent of incoming pull requests, assuming the flagging threshold is calibrated to the community's tolerance.","Deployment experience points to next steps: multimodal data and self-learning policy adaptation are the stated future directions."],"supporting_citations":[{"why":"Serves as the Presidio Analyzer baseline whose per-entity F1 scores are compared against OneShield in Table 2.","marker":"(mic 2023)"},{"why":"Serves as the StarPII baseline whose per-entity F1 scores are compared against OneShield in Table 2.","marker":"(sta 2023)"},{"why":"Establishes the risk of LLMs memorizing and leaking training data, motivating the need for input and output guardrails.","marker":"(Carlini et al. 2021)"},{"why":"Propile work on probing privacy leakage in LLMs is used to frame prompt-injection and indirect-leakage risks that the guardrail addresses.","marker":"(Kim et al. 2024)"},{"why":"Survey of privacy-preserving prompt engineering used to identify the lack of standardized evaluation benchmarks as a gap.","marker":"(Edemacu and Wu 2024)"},{"why":"Provides the GDPR regulatory requirements that the Privacy Policy Manager templates enforce.","marker":"(gdp 2024)"},{"why":"Provides the CCPA regulatory requirements that the Privacy Policy Manager templates enforce.","marker":"(ccp 2024)"},{"why":"Provides the PIPEDA regulatory requirements that the Privacy Policy Manager templates enforce.","marker":"(pip 2024)"}],"fun_headline_variants":["OneShield privacy guard hits 0.95 F1 in 26 languages","Privacy guard beats StarPII and Presidio by up to 12%","LLM privacy tool saves 300+ hours in 3 months","Sub-millisecond privacy detection for LLM prompts","One privacy framework for enterprise and open source"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline effort saving rests on the assumption that a human would otherwise have spent 15 minutes manually reviewing each of the 1,256 pull requests; the paper gives no measurement of actual review times, so the 300-plus-hours figure is an estimate, not a tracked outcome.","fun_headline_variants_meta":{"raw":{"variants":["OneShield privacy guard hits 0.95 F1 in 26 languages","Privacy guard beats StarPII and Presidio by up to 12%","LLM privacy tool saves 300+ hours in 3 months","Sub-millisecond privacy detection for LLM prompts","One privacy framework for enterprise and open source"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":4051,"prompt_tokens":974,"completion_tokens":3077,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":2989}},"tokens_in":590,"tokens_out":3077,"duration_ms":21457,"temperature":1.0,"reasoning_tokens":2989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:10:22.092168+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the evaluation data behind the 0.95 F1 claim: list the number of prompts per language among the roughly 1,200 user prompts. If most of the 26 languages have only a handful of examples, or if the F1 was computed on synthetic or internally curated text rather than an independently annotated sample, the multilingual claim would not survive; likewise, if a replay of the PR Insights deployment logs triage times shows per-review effort near zero because most flagged PRs were auto-closed, the 300-hour saving would evaporate.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the risk of LLMs memorizing and leaking training data, motivating the need for input and output guardrails."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Propile work on probing privacy leakage in LLMs is used to frame prompt-injection and indirect-leakage risks that the guardrail addresses."}],"review_version":1}